Audio-visual speech recognition with background music using single-channel source separation

Grais, Emad Mounir and Topkaya, İbrahim Saygın and Erdoğan, Hakan (2012) Audio-visual speech recognition with background music using single-channel source separation. In: 20th IEEE Conference on Signal Processing and Communications Applications (SIU2012), Mugla, Turkey

[thumbnail of siu2012.pdf] PDF
siu2012.pdf

Download (236kB)

Abstract

In this paper, we consider audio-visual speech recognition with background music. The proposed algorithm is an integration of audio-visual speech recognition and single channel source separation (SCSS). We apply the proposed algorithm to recognize spoken speech that is mixed with music signals. First, the SCSS algorithm based on nonnegative matrix factorization (NMF) and spectral masks is used to separate the audio speech signal from the background music in magnitude spectral domain. After speech audio is separated from music, regular audio-visual speech recognition (AVSR) is employed using multi-stream hidden Markov models. Employing two approaches together, we try to improve recognition accuracy by both processing the audio signal with SCSS and supporting the recognition task with visual information. Experimental results show that combining audio-visual speech recognition with source separation gives remarkable improvements in the accuracy of the speech recognition system.
Item Type: Papers in Conference Proceedings
Subjects: T Technology > T Technology (General)
Divisions: Faculty of Engineering and Natural Sciences > Academic programs > Electronics
Faculty of Engineering and Natural Sciences
Depositing User: Emad Mounir Grais Girgis
Date Deposited: 16 May 2012 14:27
Last Modified: 26 Apr 2022 09:06
URI: https://research.sabanciuniv.edu/id/eprint/19010

Actions (login required)

View Item
View Item