Multimodal multitask deep learning for chili pepper disease diagnosis and severity estimation using images and agrometeorological data

Abstract

Climate change is increasing the frequency of extreme weather events, profoundly altering the epidemiology of crop diseases, and threatening food security in vulnerable tropical agroecosystems. Although deep learning has revolutionized automated plant disease detection, prevailing models rely predominantly on unimodal visual analysis, often failing to capture the complex environmental interactions that govern pathogen development under field conditions. To overcome this limitation, we introduce a unified Multi-Task Learning (MTL) framework that simultaneously diagnoses disease type, localizes symptoms via pixel-level segmentation, and quantifies severity with high precision. Focusing on chili peppers ( Capsicum annuum ), we employ a curated dataset of 536 high-resolution field images (221 anthracnose, 196 TYLCV, 119 healthy); 527 images with complete climate linkage were retained and partitioned into training ( \(N = 315\) ), validation ( \(N = 106\) ), and independent test ( \(N = 106\) ) sets. To bolster symptom localization, 152 auxiliary anthracnose images were integrated exclusively for the segmentation branch, while a lesion-aware augmentation pipeline expanded the core multimodal training set to roughly 2,662 samples. The proposed end-to-end architecture is built upon a shared MobileNetV2 encoder. This backbone concurrently feeds a U-Net-style decoder, optimized using a novel Focal Tversky-based Hybrid Loss to address severe class imbalance caused by minute lesions, and a spatially aware multimodal fusion module. Within this module, global visual embeddings are concatenated with spatial summaries from the decoder and normalized climatic variables (temperature, humidity, rainfall, solar radiation, wind speed, and evapotranspiration) to inform subsequent dense layers. On the independent test set, the unified model achieved a classification accuracy of 99.06% (F1macro = 98.87%, MCC = 0.986), a leaf segmentation Dice of 0.844, and a disease-lesion Dice of 0.615, and a severity mean absolute error of 0.009 (global \(R^2 = 0.91\) ). Specifically, an ablation study revealed that incorporating climatic variables and spatial fusion reduced severity estimation error by 25.6% compared to a baseline only for vision, confirming the vital role of environmental context. Integration of Explainable AI (SHAP) provided biological validation, autonomously identifying low solar radiation as a key driver of fungal infection and high radiation as a promoter of viral vectors. By effectively merging computer vision with agrometeorology in a lightweight architecture capable of real-time inference (92.5 ms), this research delivers a scalable, interpretable, and climate-resilient tool for next-generation precision agriculture.