Semantic similarity based evaluation for abstractive news summarization

Beken Fikri, Figen and Oflazer, Kemal and Yanıkoğlu, Berrin (2021) Semantic similarity based evaluation for abstractive news summarization. In: 1st Workshop on Natural Language Generation, Evaluation, and Metrics, GEM 2021, Virtual, Online

Full text not available from this repository. (Request a copy)

Abstract

ROUGE is a widely used evaluation metric in text summarization. However, it is not suitable for the evaluation of abstractive summarization systems as it relies on lexical overlap between the gold standard and the generated summaries. This limitation becomes more apparent for agglutinative languages with very large vocabularies and high type/token ratios. In this paper, we present semantic similarity models for Turkish and apply them as evaluation metrics for an abstractive summarization task. To achieve this, we translated the English STSb dataset into Turkish and presented the first semantic textual similarity dataset for Turkish. We showed that our best similarity models have better alignment with average human judgments compared to ROUGE in both Pearson and Spearman correlations.
Item Type: Papers in Conference Proceedings
Divisions: Faculty of Engineering and Natural Sciences > Academic programs > Computer Science & Eng.
Faculty of Engineering and Natural Sciences
Depositing User: Berrin Yanıkoğlu
Date Deposited: 25 Aug 2022 17:24
Last Modified: 25 Aug 2022 17:24
URI: https://research.sabanciuniv.edu/id/eprint/43986

Actions (login required)

View Item
View Item