Context-aware confidence sets for fine-grained product recognition

Baz, İpek and Yörük, Erdem and Çetin, Müjdat (2019) Context-aware confidence sets for fine-grained product recognition. IEEE Access, 7 . pp. 76376-76393. ISSN 2169-3536

[thumbnail of Listed in DOAJ as an open access journal] PDF (Listed in DOAJ as an open access journal)

Download (36MB)


We present a new approach for fine-grained classification of retail products, which learns and exploits statistical context information about likely product arrangements on shelves, incorporates visual hierarchies across brands, and returns recognition results as “confidence sets” that are guaranteed to contain the true class at a given confidence level. Our system consists of three important components: 1) a nested hierarchy of product classes are automatically constructed based on visual similarities, 2) a confidence set predictor is trained based on class posteriors by using coarse-to-fine binary classifiers to discriminate each nested cluster of the hierarchy from the remainder of classes and a Bayesian network (BN) model that encodes the joint distribution of classifier scores with the fine-level class variable, and 3) n hidden Markov model (HMM) is trained with nested hidden states from the class hierarchy to model spatial transition across the nodes of product class hierarchy and resolve errors in the context-free confidence set results. Novel aspects of the proposed method include 1) combining confidence sets and context information via a HMM, 2) applying this concept to fine grained recognition of products arranged in retail shelves, and 3) presenting experimental results on four large datasets, collected from actual retail stores. We compare our approach with existing confidence set approaches and state-of-the-art convolutional neural networks classifiers including SENet-154, DenseNet-161, B-CNN, and Inception-Resnet-v2. Our approach performs comparably or better than state-of-the-art deep classifiers and exhibits high accuracy for relatively small confidence set sizes.
Item Type: Article
Uncontrolled Keywords: Confidence sets; context-aware classification; hidden Markov model; fine-grained classification; hierarchical classification
Subjects: T Technology > TK Electrical engineering. Electronics Nuclear engineering
Divisions: Faculty of Engineering and Natural Sciences > Academic programs > Electronics
Faculty of Engineering and Natural Sciences
Depositing User: Müjdat Çetin
Date Deposited: 24 Aug 2019 23:42
Last Modified: 24 Jul 2023 14:33

Actions (login required)

View Item
View Item