Multimodal deception detection using real-life trial data

Şen, Mehmet Umut and Perez-Rosas, Veronica and Yanıkoğlu, Berrin and Abouelenien, Mohamed and Burzo, Mihai and Mihalcea, Rada (2020) Multimodal deception detection using real-life trial data. IEEE Transactions on Affective Computing . ISSN 1949-3045 Published Online First http://dx.doi.org/10.1109/TAFFC.2020.3015684

Warning
There is a more recent version of this item available.
Full text not available from this repository. (Request a copy)

Abstract

Hearings of witnesses and defendants play a crucial role when reaching court trial decisions. Given the high-stakes nature of trial outcomes, developing computational models that assist the decision-making process is an important research venue. In this paper, we address the identification of deception in real-life trial data. We use a dataset consisting of videos collected from public court trials. We explore the use of verbal and non-verbal modalities to build a multimodal deception detection system that aims to discriminate between truthful and deceptive statements provided by defendants and witnesses. In particular, three complementary modalities (visual, acoustic and linguistic) are evaluated for the classification of deception at the subject level. The final classifier is obtained by combining the three modalities via score-level classification, achieving 83.05% accuracy in subject-level deceit detection. To place our results in perspective, we present a human deception detection study where we evaluate the human capability of detecting deception using different modalities and compare the results to the developed system. The results show that our system outperforms the average non-expert human capability of identifying deceit.
Item Type: Article
Divisions: Faculty of Engineering and Natural Sciences > Academic programs > Computer Science & Eng.
Faculty of Engineering and Natural Sciences
Depositing User: Berrin Yanıkoğlu
Date Deposited: 03 Sep 2021 16:18
Last Modified: 03 Sep 2021 16:18
URI: https://research.sabanciuniv.edu/id/eprint/42388

Available Versions of this Item

Actions (login required)

View Item
View Item