Analysis and Improvement of Optical Flow Methods with a Focus on Robustness

Abstract

In the past decade, optical flow estimation has garnered increased attention from researchers as a fundamental component of computer vision applications such as autonomous driving and video editing. However, previous research has mainly focused on predicting more accurate results in benchmarks, revealing a gap between experimental results and real world scenarios. This thesis explores this field and aims to propose optical flow estimation methods that are robust for real-world applications. Specifically, we identify three main challenges that previous methods still face: • Small Motion Boundary Error: Existing optical flow methods generate blurred motion boundaries for small objects, which usually fuse with the background. • Generalization to Unseen Data: Existing optical flow methods struggle to predict accurate optical flow for unseen data. • Robustness to Both Clean and Adverse Conditions: Existing optical flow methods contain a trade-off in performance between clean and adverse conditions. To address these issues, we conduct experiments and analyze each problem. We observe that the main reason for small motion boundary error is that traditional downsampling methods can weaken edge information. Thus, we propose a novel downsampling module for the encoder of optical flow, which visibly reduces motion boundary errors. Furthermore, we propose a content-aware regularization method that significantly enhances the generalization ability of unseen data. We introduce RobFlow, an efficient training strategy to make optical flow methods robust in both clean and real-world adverse conditions, such as rain and snow. We then propose a robustness evaluation method to help future studies of optical flow in the real world. To further study video frame interpolation (VFI) as an application of optical flow, we observe that optical flow methods fail to deal with large obstructions in scenes, which are commonly encountered in the real world. We propose a large obstruction robustness framework to address this issue and enhance the robustness of VFI methods for large obstructions. Extensive experiments demonstrate that our methods not only improve the robustness for real-world scenarios but also enhance performance in benchmarks. To further explore how optical flow methods affect the VFI method, we plan to discover the relational representation between two frames. Our main idea is to use a masked auto-encoder to learn a better representation which improves both optical flow and video frame interpolation tasks.