Multimodal deception detection using real-life trial data

Şen, Mehmet Umut and Perez-Rosas, Veronica and Yanıkoğlu, Berrin and Abouelenien, Mohamed and Burzo, Mihai and Mihalcea, Rada (2022) Multimodal deception detection using real-life trial data. IEEE Transactions on Affective Computing, 13 (1). pp. 306-319. ISSN 1949-3045

This is the latest version of this item.

Full text not available from this repository. (Request a copy)


Hearings of witnesses and defendants play a crucial role when reaching court trial decisions. Given the high-stakes nature of trial outcomes, developing computational models that assist the decision-making process is an important research venue. In this article, we address the identification of deception in real-life trial data. We use a dataset consisting of videos collected from public court trials. We explore the use of verbal and non-verbal modalities to build a multimodal deception detection system that aims to discriminate between truthful and deceptive statements provided by defendants and witnesses. In particular, three complementary modalities (visual, acoustic and linguistic) are evaluated for the classification of deception at the subject level. The final classifier is obtained by combining the three modalities via score-level classification, achieving 83.05 percent accuracy in subject-level deceit detection. To place our results in perspective, we present a human deception detection study where we evaluate the human capability of detecting deception using different modalities and compare the results to the developed system. The results show that our system outperforms the average non-expert human capability of identifying deceit.
Item Type: Article
Uncontrolled Keywords: acoustic; classification; deception detection; linguistic; multimodal; Real-life trial; visual
Divisions: Faculty of Engineering and Natural Sciences
Depositing User: Berrin Yanıkoğlu
Date Deposited: 31 Aug 2022 20:55
Last Modified: 31 Aug 2022 20:55

Available Versions of this Item

Actions (login required)

View Item
View Item