Quantum mechanics

Speech coding

Voltage

Generative model

Transfer of learning

Statistics

Feature learning

Economics

Coding (social sciences)

Computer science

Natural language processing

Predictive coding

Physics

Generative grammar

Mathematics

Spectrogram

Autoregressive model

Econometrics

Artificial intelligence

Transformer

Machine learning

Speech recognition

Learning meaningful and general representations from unannotated speech that are applicable to a wide range of tasks remains challenging. In this paper we propose to use autoregressive predictive coding (APC), a recently proposed self-supervised objective, as a generative pre-training approach for learning meaningful, non-specific, and transferable speech representations. We pre-train APC on large-scale unlabeled data and conduct transfer learning experiments on three speech applications that require different information about speech characteristics to perform well: speech recognition, speech translation, and speaker identification. Extensive experiments show that APC not only outperforms surface features (e.g., log Mel spectrograms) and other popular representation learning methods on all three tasks, but is also effective at reducing downstream labeled data size and model parameters. We also investigate the use of Transformers for modeling APC and find it superior to RNNs.

Generative Pre-Training for Speech with Autoregressive Predictive Coding