Human-supervised, large language model-based clinical decision support aligned to national newborn protocols in Kenya: a pragmatic, early-stage evaluation

Abstract

Abstract Introduction Timely, protocol-adherent clinical decisions are crucial for reducing neonatal mortality in low-resource settings. Translating extensive national guidelines into bedside practice remains challenging. Objective We developed and evaluated AIFYA, a human-supervised, large language model (LLM)-based clinical decision support system (CDSS) aligned with Kenya’s national newborn care protocols. Methods This prospective, mixed-methods, early-stage evaluation, guided by the DECIDE-AI framework, embedded AIFYA into routine workflows at two public health facilities (Level 5 and Level 4) in Bungoma County, Kenya, from September 2024 to June 2025. Primary outcomes were: (1) adoption, measured by cumulative neonatal cases managed; (2) training reach, assessed by credentialed healthcare workers (HCWs); and (3) guideline and citation concordance, evaluated through blinded review of 118 AI-generated recommendations by two neonatologists, with adjudication by a third. Secondary outcomes included protocol adherence and triage-to-decision time Results A total of 50 HCWs were trained, and 550 neonatal cases were managed over 10 months. Among surveyed HCWs (n = 33), 76% were female (mean age 32.1 years). Expert review found 75% of recommendations were correct and 15% partially correct, with strong interrater reliability (weighted Cohen’s kappa 0.85; 95% CI 0.79–0.91). Citation accuracy was 96%. In 40 complex dosing scenarios, 75% of outputs were rated correct. The median triage-to-decision time was 23 minutes (IQR 18–31). Implementation was supported by an offline-first architecture and a facility-based coaching model, sustaining engagement despite staff turnover. Conclusion A human-supervised AI CDSS, directly and transparently anchored to national clinical guidelines, can be successfully implemented in routine, low-resource neonatal care settings. The system demonstrated high user adoption and strong expert-rated concordance. The high citation accuracy is a critical feature that builds clinical trust, ensuring safety and enabling auditable AI. These findings provide a robust foundation for progression to controlled, multi-site trials to evaluate clinical effectiveness.