BeGrading: large language models for enhanced feedback in programming education

Abstract

Abstract In recent years, large language models (LLMs) have gained significant traction across various domains, including education. This paper explores the application of LLMs in grading programming assignments. By leveraging data collected from existing programming assignments and their corresponding grades, we aim to develop a robust LLM-based grading system. We also incorporate augmented data representing various grading scenarios to enhance the model’s performance and ensure comprehensive coverage across all grading levels. Our approach involves training the LLM on this combined dataset to enable accurate and consistent evaluation of programming assignments. The proposed model, BeGrading, aims to reduce the grading burden on educators and provide timely and objective feedback to students. Compared to the Codestral model, our proposed model demonstrates an absolute difference rate of 19%, equivalent to $$\pm 0.95$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mo>±</mml:mo> <mml:mn>0.95</mml:mn> </mml:mrow> </mml:math> out of 5. This is acceptable for using a small, fine-tuned model with optimized data. Additionally, the Codestral model compared to the dataset optimized score shows a difference of 15% equivalent to a margin of $$\pm 0.75$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mo>±</mml:mo> <mml:mn>0.75</mml:mn> </mml:mrow> </mml:math> out of 5. Preliminary results demonstrate the potential of LLMs to perform grading tasks with a high degree of reliability, opening avenues for further research and practical applications in automated education systems.