Text-based image retrieval system using semantic visual content for re-ranking

Official URL: https://dx.doi.org/10.1016/j.engappai.2025.111770

We address the problem of retrieving relevant images from the web in response to a text query. Text-based image retrieval is a challenging multimodal problem that requires understanding of the user intent from a few keywords and a semantic understanding of the images to predict relevance. In this paper, we present the hybrid search approach of the Turkcell-Yaani Search Engine, which was the leading search engine for queries in the Turkish language in 2017–2022. With over 1 million monthly users and processing millions of requests per month, its hybrid image search approach demonstrates the practical relevance and effectiveness of integrating text and visual information for enhanced image retrieval. The initial stage of the system focuses on retrieving images by comparing the query text with textual information associated with previously crawled and indexed images. The second stage aims to address the limitations of relying solely on text-based information by incorporating the post-processing steps based on semantic analysis of retrieved images. This stage, which is the main contribution of this paper, involves clustering, filtering and reordering the retrieved images using their embedded representations that are obtained from pretrained deep neural networks. The experimental results show that incorporating visual content improves the performance of the text-based retrieval engine significantly.