Cognitive psychology

Neuroscience

Visualization

Visual perception

Modality (human–computer interaction)

Visual learning

Audio visual

Computer science

Psychology

Multimodal learning

Leverage (statistics)

Modalities

Sociology

Artificial intelligence

Deep learning

Perception

Human–computer interaction

Speech recognition

Social science

Multimedia

International Journal of Automation and Computing

Audio-visual learning, aimed at exploiting the relationship between audio and visual modalities, has drawn considerable attention since deep learning started to be used successfully. Researchers tend to leverage these two modalities to improve the performance of previously considered single-modality tasks or address new challenging problems. In this paper, we provide a comprehensive survey of recent audio-visual learning development. We divide the current audio-visual learning tasks into four different subfields: audio-visual separation and localization, audio-visual correspondence learning, audio-visual generation, and audio-visual representation learning. State-of-the-art methods, as well as the remaining challenges of each subfield, are further discussed. Finally, we summarize the commonly used datasets and challenges.

Deep Audio-visual Learning: A Survey