Lip-reading: Phoneme-to-text Machine Translation using Large Language Models

Abstract

An effective conversion of recognised phonemes to words has a significant impact on the overall performance of an automatic lip-reading system. In this paper, transformer-based models have been examined for phoneme-to-text translation. The benchmark BBC Lip-reading Sentence 2 (LRS2) data is used in this study, and the T5 model (Text-To-Text Transfer Transformer) and the GPT-2 model are considered the conversion models. A detailed discussion is provided on the analysis procedure, data pre-processing, model implementation, and the metrics employed for model evaluation. The experimental results have demonstrated that the T5 model can achieve a substantial improvement over the GPT-2 model with a superior accuracy and fluency in the phoneme-to-text translation, offering a potentially promising solution for enhancing the precision of automatic translation and speech processing systems. This work contributes to the development of more robust applications in the field of speech recognition and communication technologies.