A large scale multimodal dataset for healthcare domain Ghanaian sign language translation and retrieval based synthesis

Abstract

Despite rapid advancements in AI-driven language processing tools, sign languages across Africa, including Ghanaian Sign Language (GhSL), remain critically underrepresented. Unlike spoken languages, sign languages involve complex hand gestures, facial expressions, and spatial movements, making their computational processing inherently multimodal. Challenges such as the lack of standardization, regional variations, and the absence of large, publicly available datasets have hindered the development of AI-powered sign language recognition and translation systems, particularly in underrepresented communities, leading to lack of accessibility and inclusion for persons with hearing impairment, especially in essential areas like healthcare, where effective communication is vital. Currently, no publicly available Ghanaian Sign Language datasets on healthcare access and interactions exist. This highlights the need for a comprehensive, standardized dataset to support robust research and practical applications. To address this, we introduce SignTalk-Gh, the first curated domain-specific Ghanaian Sign Language dataset for healthcare, designed to support future development of recognition and translation systems, with demonstrated suitability for retrieval-based text-to-sign applications. This dataset is specifically designed to aid translation research, capturing doctor-patient conversations in Ghanaian healthcare settings. This paper outlines the methodology behind the dataset’s construction, including data collection, annotation, validation, and evaluation processes. We also discuss its potential applications in AI-driven healthcare accessibility and its role in advancing research on African sign languages. Future directions include expanding the dataset to other domains, such as education, general conversations, and, among others, subsequently developing neural machine translation models for GhSL interpretation in medical environments and other domains