Optimized Extraction of Handwritten Characters in Images: A Hybrid Approach Based on CNN-ViT and Hough Transform

Abstract

Automatic recognition of handwritten characters remains a major challenge in image processing, mainly due to the variability of human handwriting and the inherent distortions of handwritten media. In this study, we propose a novel hybrid architecture called Hough_ConViT, which combines ResNet50, Inception-V3, and a Vision Transformer (ViT) model for feature extraction. The methodological framework includes a preliminary skew-correction step using the Hough Transform, followed by robust data augmentation involving rotation and normalization operations. The dataset was built by merging the <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$A \_Z$</tex> Handwritten Data collection with additional handwritten images gathered online, resulting in a complete and balanced corpus. The model was trained in the PyTorch environment and executed on Google Colab, incorporating an early-stopping mechanism to prevent overfitting. Experimental results demonstrate high performance, with an overall accuracy of 99.76%, a Character Error Rate (CER) of 0.24%, and a macro F1-score of 96.35%. These results, which significantly outperform traditional methods, confirm the robustness and reliability of the proposed approach for accurate recognition of French handwritten characters.