Physics

Task analysis

Multi-task learning

Politics

Human–computer interaction

Machine learning

Language model

Political science

Artificial intelligence

Management

Economics

Set (abstract data type)

Programming language

Natural language processing

Representation (politics)

Scale (ratio)

Computer science

Task (project management)

Quantum mechanics

Much of vision-and-language research focuses on a small but diverse set of independent tasks and supporting datasets often studied in isolation; however, the visually-grounded language understanding skills required for success at these tasks overlap significantly. In this work, we investigate these relationships between vision-and-language tasks by developing a large-scale, multi-task model. Our approach culminates in a single model on 12 datasets from four broad categories of task including visual question answering, caption-based image retrieval, grounding referring expressions, and multimodal verification. Compared to independently trained single-task models, this represents a reduction from approximately 3 billion parameters to 270 million while simultaneously improving performance by 2.05 points on average across tasks. We use our multi-task framework to perform in-depth analysis of the effect of joint training diverse tasks. Further, we show that finetuning task-specific models from our single multi-task model can lead to further improvements, achieving performance at or above the state-of-the-art.

12-in-1: Multi-Task Vision and Language Representation Learning