Motion (physics)

Video tracking

Video compression picture types

Multiview Video Coding

Telecommunications

Economics

Computer science

Interpolation (computer graphics)

Video processing

Task (project management)

Mathematics

Politics

Management

Flow (mathematics)

Image (mathematics)

Political science

Video post-processing

Video denoising

Operations management

Video quality

Artificial intelligence

Geometry

Representation (politics)

Motion compensation

Optical flow

Metric (unit)

Computer vision

Frame (networking)

International Journal of Computer Vision

Many video enhancement algorithms rely on optical flow to register frames in a video sequence. Precise flow estimation is however intractable; and optical flow itself is often a sub-optimal representation for particular video processing tasks. In this paper, we propose task-oriented flow (TOFlow), a motion representation learned in a self-supervised, task-specific manner. We design a neural network with a trainable motion estimation component and a video processing component, and train them jointly to learn the task-oriented flow. For evaluation, we build Vimeo-90K, a large-scale, high-quality video dataset for low-level video processing. TOFlow outperforms traditional optical flow on standard benchmarks as well as our Vimeo-90K dataset in three video processing tasks: frame interpolation, video denoising/deblocking, and video super-resolution.

Video Enhancement with Task-Oriented Flow