Using Deep Learning Models for Multimodal Sentence Level Sentiment Analysis of Sign Language

Abstract

Deaf and hard-of-hearing individuals communicate with signs such as hand signals, gestures, facial expressions, and body movements. This medium of communication is called sign language, which is a non-verbal, visual means of communication. However, some non-deaf and non-hard-of-hearing individuals do not understand sign language; those who understand it use it to communicate with deaf and hard-of-hearing individuals. Some active users of social media are deaf and hard-of-hearing individuals; therefore, it is necessary to develop technological tools that will guarantee effective communication between deaf & hard-of-hearing individuals and non-deaf & non-hard-of-hearing individuals, especially across various social media platforms. Sentiment analysis of sign language is one such technological tool that helps to communicate the polarity expressed in sign language. A multimodal approach to sentiment analysis of sign language is the focus of this study, which uses a multimodal sign language dataset to train two Deep Learning models. The dataset consists of video clips of sentence-level sign language and textual equivalents. The dataset trains a deep convolutional neural network model called VGG16 for visual modality. The other Deep Learning model, which the dataset trains, is Bidirectional Encoder Representation from Transformer, BERT for textual modality. The results of the performance metrics showed that the multimodal approach performed better than the single-modality text-based approach.