The case for audio-first mixed reality: an AI-enhanced framework

Official URL: https://dx.doi.org/10.1109/AIxVR63409.2025.00032

Current mixed reality (MR) interfaces predominantly rely on visual modalities, creating fundamental limitations in high-stress scenarios and detail-critical applications. This position paper challenges the visual-first paradigm in MR interfaces by proposing a shift toward audio-centric interaction supported by artificial intelligence. We argue that the convergence of three key developments - advances in spatial audio processing, the emergence of powerful language models, and the increasing sophistication of retrieval-augmented generation - creates a unique opportunity to reimagine MR interfaces. We present a conceptual framework for an Audio-Based Situated Analytics system that prioritizes auditory interaction while reserving visual channels for essential tasks. Drawing from research in audio augmented reality, situated analytics, and AI, we explore how this paradigm shift could transform critical applications such as first response scenarios and cultural heritage experiences. This study aims to spark discussion in the MR community about the role of audio interfaces, outline key research challenges, and propose a roadmap for future investigation. By challenging current assumptions about MR interface design, we hope to encourage new directions in research that could lead to more effective, inclusive, and less cognitively demanding MR experiences.