Impact of Internal Validation Protocols on Predictive Maintenance Performance in Biomedical Equipment

Abstract

Predictive maintenance (PdM) is a strategic enabler of healthcare digitalization, yet its deployment remains constrained by methodological weaknesses in model evaluation. Biomedical maintenance data, structured around equipment life cycles and repeated interventions, violate the independence and stationarity assumptions of conventional random cross-validation. This work presents an empirical analysis of internal validation protocol design using a real-world, multi-hospital dataset comprising 3403 maintenance interventions. Three classification models (logistic regression, random forest, histogram-based gradient boosting) are evaluated under four validation schemes: random K-fold, equipment-grouped K-fold, temporal holdout, and roll-forward validation. The results reveal a consistent decrease in apparent predictive performance as validation constraints are progressively strengthened. Random cross-validation overestimates AUROC by approximately 0.03–0.06 compared with temporally constrained protocols. Under deployment-aligned temporal validation, model performance stabilizes at an AUROC of approximately 0.83–0.84. Equipment-grouped and temporal validation effectively mitigate structural bias and yield more stable and interpretable models. These findings highlight the critical role of validation protocol choice in the credible assessment of predictive maintenance models and provide practical guidance for the deployment of PdM systems based on real-world data in resource-limited healthcare environments. The analysis is limited to public hospitals within a single national context and relies on a class-balanced experimental subset, which may affect the direct transferability of absolute performance estimates to other healthcare systems or operational settings.