Deep learning

Sociology

Pure mathematics

Key (lock)

Feature (linguistics)

Modalities

Mathematics

Social science

Politics

Software deployment

Variety (cybernetics)

Software engineering

Machine learning

Philosophy

Workflow

Political science

Embedded system

Artificial intelligence

Data science

Annotation

Representation (politics)

Computer science

Linguistics

Bottleneck

Field (mathematics)

Computer security

Feature learning

Database

IEEE Signal Processing Magazine

Self-supervised representation learning methods aim to provide powerful deep feature learning without the requirement of large annotated datasets, thus alleviating the annotation bottleneck that is one of the main barriers to practical deployment of deep learning today. These methods have advanced rapidly in recent years, with their efficacy approaching and sometimes surpassing fully supervised pre-training alternatives across a variety of data modalities including image, video, sound, text and graphs. This article introduces this vibrant area including key concepts, the four main families of approach and associated state of the art, and how self-supervised methods are applied to diverse modalities of data. We further discuss practical considerations including workflows, representation transferability, and compute cost. Finally, we survey the major open challenges in the field that provide fertile ground for future work.

Self-Supervised Representation Learning: Introduction, advances, and challenges