Human-in-the-loop optimization of perceived realism of multi-modal haptic rendering under conflicting sensory cues

Official URL: https://dx.doi.org/10.1109/TOH.2025.3535416

During haptic rendering, a visual display and a haptic interface are commonly utilized together to elicit multi-sensory perception of a virtual object, through a combination and integration of force-related and movement-related cues. In this study, we explore visual-haptic cue integration during multi-modal haptic rendering under conflicting cues and propose a systematic means to determine the optimal visual scaling for haptic manipulation that maximizes the perceived realism of spring rendering for a given haptic interface. We show that the parameters affecting visual-haptic congruency can be effectively optimized through a qualitative feedback-based human-in-the-loop (HiL) optimization to ensure a consistently high rating of perceived realism. Accordingly, the multi-modal perception of users can be successfully enhanced by solely modulating the visual feedback without altering the haptic feedback, to make virtual environments feel stiffer or more compliant, significantly extending the range of perceived stiffness levels for a haptic interface. We extend our results to a group of individuals to capture the multi-dimensional psychometric field that characterizes the cumulative effect of feedback modalities utilized during sensory cue integration under conflicts. Our results not only provide reliable estimates of just noticeable difference thresholds for stiffness with and without visual scaling but also capture all the prominent features of sensory cue integration, indicating weights that are proportional to the congruency level of manipulated visual signals. Overall, preference-based HiL optimization excels as a systematic and efficient method of studying multi-modal perception under conflicts.