conference paper
Standard approaches to learner modelling in introductory computer science often focus strictly on predicting whether a student will correctly solve subsequent exercises. This study investigates the incorporation of detailed process data and expert-refined misconception signals derived from programming traces in the CodeWorkout platform. Comparing correctness-only models with process-aware and misconception-based models demonstrated clear performance advantages. The process-aware system achieved the highest macro F1 score, whereas the model integrating misconception signals delivered the best discrimination and calibration, as measured by AUC and Brier scores. A human-in-the-loop evaluation showed that educational experts considered 90.0% of the identified misconception signals to be both plausible and actionable for teaching. Furthermore, instructors preferred misconception-aware summaries over standard reports in 90.0% of evaluations, rating them superior in credibility, clarity, usefulness, and actionable teaching value.
Automated educational tools frequently alert teachers only to whether a student passed or failed a task, omitting the reasons behind their difficulties. By identifying recurring misconceptions and process patterns directly from student code traces, automated systems can provide actionable diagnostic insights. This allows computer science instructors to deliver timely, targeted interventions tailored to the specific conceptual hurdles a novice programmer faces.
This research is applicable to educational technology providers and computer science instructors seeking more interpretable student analytics in automated tutoring platforms. Because the models and feedback summaries have already been validated with real course traces and assessed by human instructors, the approach is applied and tested. It could be integrated into commercial programming learning environments to offer diagnostics rather than basic pass-fail tracking.
AI-generated from the published abstract. Always read the original work before citing.
Knowledge tracing in introductory programming is commonly framed as predicting whether a student will eventually solve the next problem. While useful, this approach reduces rich programming activity to correctness labels and offers limited support for instructional interpretation. This study examines whether learner modelling in CS1 can be improved by moving from correctness histories to expert-refined misconception signals derived from programming traces. Using CodeWorkout trace data, we compared three models on a held-out later-course split: a correctness-only model, a process-aware model using compilation, execution, and revision features, and a misconception-signal model that added expert-refined recurrent code-pattern signals. Because eventual correctness was highly imbalanced, we evaluated performance using macro F1, AUC, and Brier score rather than relying on accuracy alone. The process-aware model achieved the strongest macro F1, while the misconception-signal model achieved the best AUC and Brier score, indicating stronger discrimination and calibration than correctness-only histories. To examine pedagogical value, we conducted a human-in-the-loop evaluation. Experts judged 90.0% of candidate signals plausible and 90.0% actionable for teaching, and misconception-aware summaries were preferred in 90.0% of paired instructor evaluations. These summaries were also rated higher for credibility, usefulness, clarity, and actionability. The study shows that richer learner representations can move CS1 knowledge tracing beyond predicting outcomes alone towards explaining the nature of student struggle in ways instructors can interpret and use.
This page summarises published work. The authoritative version sits with the publisher.
DOI: 10.1145/3830800.3830801
Is something wrong with this record? Report it or request removal.
Discussion
Have you built on this work, tried to replicate it, or seen it applied in practice? Share what you know. Verified researchers and MARATTO™ domain experts can open a discussion, and any member can reply. Contributions are reviewed before they appear.
No discussion yet. Open the first thread.
New to MARATTO™? Create a free account.
