Interactive Augmented Reality System for Learning Phonetics Using Artificial Intelligence

Abstract

The increasing adoption of language learning apps that utilize Augmented Reality (AR) and Artificial Intelligence (AI) for speech recognition has sparked interest in the potential benefits for phonetics education. However, currently available AR apps only focus on teaching letter names and vocabulary, lacking the potential for a more immersive learning experience. To address this limitation, this paper introduces an interactive AR system that integrates AI speech recognition with AR to provide an engaging and interactive learning experience. To showcase the capabilities of the proposed system, we have created a prototype for the Arabic Phonetic Atlas textbook. This prototype enhances reading the /s/ sound page in the Atlas by incorporating a 3D animated model of the speech organs onto the existing 2D image. The dynamic animation of the 3D model reflects the sound description provided in the Atlas. The system also offers real-time feedback on users’ pronunciation through a customized AI phoneme recognition system. A comprehensive user study was conducted to evaluate the usability and learning impact of the proposed system, involving 83 adult participants aged between 18-40. The assessment approach involved the use of both direct and indirect observations, as well as various surveys to gather both numerical and qualitative information. The findings indicate not only a greater level of understanding compared to conventional methods, but also an improved capability to master specific phonemes quickly and effortlessly. In addition, showing a great potential for the proposed system to be incorporated into the conventional classroom setting as an instructional aid.