Comparative evaluation of lexicon-based and transformer-based sentiment analysis tools

Abstract

Sentiment analysis is a cornerstone of social media–based public opinion monitoring, yet the optimal approach for pandemic-related discourse remains contested. This study conducts a comparative evaluation of two dominant paradigms: lexicon-based sentiment analysis and transformer-based deep learning models. The proliferation of user-generated content on social media platforms during global crises has catalyzed the use of sentiment analysis as a tool for understanding public perceptions and behaviours. However, the choice of sentiment analysis tools significantly influences the accuracy, interpretability, and applicability of results. This study conducts a comparative evaluation of three widely used sentiment analysis tools—VADER (Valence Aware Dictionary for Sentiment Reasoning), TextBlob, Orange Data Mining’s integrated sentiment classifier and transformer-based deep learning (BERT) —in analysing COVID-19–related Twitter data. Using a dataset of approximately 57,000 tweets collected during multiple pandemic phases, the tools’ performance in terms of classification accuracy, precision, recall, F1-score, computational efficiency, and alignment are assessed with human-labelled ground truth data. The findings reveal that while VADER outperforms in capturing nuanced sentiment in short, informal texts, TextBlob demonstrates strengths in polarity scoring but suffers from over-generalization, and Orange’s classifier provides strong baseline performance with enhanced usability for non-programmers. The study offers practical guidelines for selecting sentiment analysis tools in pandemic-related social media research, balancing accuracy with accessibility for multidisciplinary research teams.