Audio-visual speech recognition with background music using single-channel source separation

Grais, Emad Mounir and Topkaya, İbrahim Saygın and Erdoğan, Hakan (2012) Audio-visual speech recognition with background music using single-channel source separation. In: 20th IEEE Conference on Signal Processing and Communications Applications (SIU2012), Mugla, Turkey

PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader

Official URL: http://dx.doi.org/10.1109/SIU.2012.6204436


In this paper, we consider audio-visual speech recognition with background music. The proposed algorithm is an integration of audio-visual speech recognition and single channel source separation (SCSS). We apply the proposed algorithm to recognize spoken speech that is mixed with music signals. First, the SCSS algorithm based on nonnegative matrix factorization (NMF) and spectral masks is used to separate the audio speech signal from the background music in magnitude spectral domain. After speech audio is separated from music, regular audio-visual speech recognition (AVSR) is employed using multi-stream hidden Markov models. Employing two approaches together, we try to improve recognition accuracy by both processing the audio signal with SCSS and supporting the recognition task with visual information. Experimental results show that combining audio-visual speech recognition with source separation gives remarkable improvements in the accuracy of the speech recognition system.

Item Type:Papers in Conference Proceedings
Subjects:T Technology > T Technology (General)
ID Code:19010
Deposited By:Emad Mounir Grais Girgis
Deposited On:16 May 2012 14:27
Last Modified:31 Jul 2019 10:52

Repository Staff Only: item control page