Hybrid Approach for Automatic Stemming of Chichewa Words Using Clustering with 4-grams and Affix Stripping

Abstract

Stemming refers to transforming different word forms into a standard root form. Research shows that stemming is crucial for improving the performance of tasks such as information retrieval and text classification in Natural Language Processing. For low-resource languages like Chichewa, the obvious choice for automatic stemming is the N-gram stemming technique. However, N-gram stemming algorithms lack contextual information and may not consider the broader linguistic context of words. They require a large and balanced dataset to stem out-of-vocabulary words effectively. Unfortunately, it is difficult for languages that exhibit rich morphology, like Chichewa, to have a dataset representative of all the words since the words are formed by combining morphemes, which can form an uncountable number of possible word forms. This paper proposes stemming Chichewa words by stripping affixes of similar words in a cluster and then extracting the most common character 4-grams of the resultant words in the cluster. The clustering of the words is done using a pre-trained Fasttext word embedding model, which clusters words by considering linguistic and context information and is capable of constructing vector representations of out-of-vocabulary (OOV) words, taking them into account. We evaluated the efficacy of our algorithm in comparison to the affix parsing method. The empirical findings revealed that our approach surpasses the affix parsing approach, registering an F1-score of 62% and an accuracy of 70%.