speech recognition

speech enhancement

emotion recognition

multimodal sentiment

speaker recognition

speech emotion

IEEE Journal of Selected Topics in Signal Processing

Self-supervised learning (SSL) achieves great success in speech recognition,\nwhile limited exploration has been attempted for other speech processing tasks.\nAs speech signal contains multi-faceted information including speaker identity,\nparalinguistics, spoken content, etc., learning universal representations for\nall speech tasks is challenging. To tackle the problem, we propose a new\npre-trained model, WavLM, to solve full-stack downstream speech tasks. WavLM\njointly learns masked speech prediction and denoising in pre-training. By this\nmeans, WavLM does not only keep the speech content modeling capability by the\nmasked speech prediction, but also improves the potential to non-ASR tasks by\nthe speech denoising. In addition, WavLM employs gated relative position bias\nfor the Transformer structure to better capture the sequence ordering of input\nspeech. We also scale up the training dataset from 60k hours to 94k hours.\nWavLM Large achieves state-of-the-art performance on the SUPERB benchmark, and\nbrings significant improvements for various speech processing tasks on their\nrepresentative benchmarks. The code and pre-trained models are available at\nhttps://aka.ms/wavlm.\n

WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing