Automating Software Test Case Generation with Pre-trained Large Language Models

Abstract

the rise of Large Language Models (LLMs) has significantly impacted software engineering based on their capability to comprehend and create natural language. Traditional test-case automation techniques, whether they depend only on textual descriptions or exclusively on focal methods often fail to express expected behavior, capture edge conditions, and expose faults. Such limitations make them illsuited for Test-Driven Development (TDD). To overcome these challenges, we present an approach that combines text and methods for test case generation. Using prompt engineering techniques, we evaluate two LLMs: Meta-LLaMA-3-8B (base) and Mistral-7B-Instruct on a dataset of $\mathbf{1 3 6, 0 6 0}$ entries, resulting in the generation of 7,606- unit test cases from opensource projects to assess whether instruction tuning or an increase in parameter count has a greater impact on test case generation quality. Results show that instruction tuning outperforms a larger base model, especially in zero-shot settings, while few-shot prompting reduces syntax errors by $\boldsymbol{\sim} \mathbf{9 0 \%}$ for both.