Multi-modal deception detection from videos

Warning The system is temporarily closed to updates for reporting purpose.

Şen, Mehmet Umut (2020) Multi-modal deception detection from videos. [Thesis]

[thumbnail of 10358083_Sen_Mehmet__Umut.pdf] PDF

Download (1MB)


Hearings of witnesses and defendants play a crucial role when reaching court trial decisions. Given the high-stakes nature of trial outcomes, developing computational models that assist the decision-making process is an important research venue. In this thesis, we address the deception detection in real-life trial videos. Using a dataset consisting of videos collected from concluded public court trials, we explore the use of verbal and non-verbal modalities to build a multimodal deception detection system that aims to classify the defendant in a given video as deceptive or not. Three complementary modalities (visual, acoustic and linguistic) are evaluated separately for the classification of deception. The final classifier is obtained by combining the three modalities via score-level classification, achieving 83.05% accuracy. Multimodal analysis of trial videos involves many challenges. Prior to developing the final deception detection system, we have worked on sub-problems that would be helpful on improving deception detection performance. High volume of background sounds in a video decreases the quality of the speech features, and it results in low speech recognition performance. We developed a neural network based single-channel source separation model to extricate the speech from the mixed sound recording. Word embeddings, is the state-of-art technique in processing of textual data. In addition to evaluating pretrained word embeddings in developing the deception system for English, we have also worked on learning word embeddings for Turkish and used them for categorizing text documents. This work can be applied in future for a deception system in Turkish
Item Type: Thesis
Uncontrolled Keywords: deception detection. -- multi-modal. -- word embeddings. -- document classification. -- speech source separation . -- aldatmaca kestirimi. -- çoklu-modalite. -- kelime temsilleri. -- doküman sınıflandırma.-- konuşma kaynak ayırımı.
Subjects: T Technology > TK Electrical engineering. Electronics Nuclear engineering > TK7800-8360 Electronics
Divisions: Faculty of Engineering and Natural Sciences > Academic programs > Electronics
Faculty of Engineering and Natural Sciences
Depositing User: IC-Cataloging
Date Deposited: 01 Nov 2020 14:42
Last Modified: 26 Apr 2022 10:34

Actions (login required)

View Item
View Item