An Enhanced Filter-based approach for Feature Selection

Abstract

Feature selection is a critical preprocessing step in machine learning that enhances model performance, reduces computational complexity, and improves interpretability by identifying the most informative input features. Among the common approaches—filter, wrapper, and embedded methods— filter-based methods are widely favored for their simplicity, speed, and scalability to high-dimensional datasets. However, they face key limitation including reliance on arbitrary predefined feature threshold, assumption of independence from the learning algorithm, and absence of performance-driven feedback. These limitations often resulting in the selection of suboptimal feature subsets. This paper proposes an enhanced filter-based feature selection approach that addresses these limitations by incorporating a scorer method via machine learning and an early-stopping grid search strategy on ranked features. Instead of relying on fixed thresholds, the proposed approach systematically evaluates varying feature subset sizes up to matching criteria to identify subsets of features that maximizes the model’s accuracy. Experiments conducted on three datasets demonstrate the effectiveness of the proposed approach. Three common statistical ranking methods are utilized including χ<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup>, mutual information, and Variance Threshold. The proposed approach achieved significant feature reduction, eliminating up to 98% of features, while consistently outperforming baseline filter methods in terms of classification accuracy. Notably, the classification accuracy improved by 26.09%, 8.64%, and 5.28% on the three datasets, respectively, compared to the baseline results obtained using all features.