ECJ-LLM: From Extraction to Judgment; Reinventing Automated Review Annotation

Abstract

This paper introduces a novel multi-agent debate framework for interest-based data annotation, leveraging Large Language Models (LLMs) to facilitate structured, collaborative annotation. The framework orchestrates three specialized LLM agents—an aspect extractor, a critic, and a judge—who engage in a systematic debate: the extractor identifies relevant aspects, the critic evaluates and challenges these aspects, and the judge synthesizes their input to assign final, high-quality interest-level labels. This debate-driven process not only enhances annotation fidelity and context but also allows for flexible customization of models used in each role and the interest to be detected. To ensure transparency and quality, the framework incorporates an evaluation suite with metrics such as precision, recall, F1-score, and confusion matrices. Empirical results on a gold-standard hotel review dataset demonstrate that the framework outperforms single-agent methods in annotation quality. A customizable annotation tool, developed as a demonstration of the framework’s practical utility, further showcases its flexibility and extensibility for a range of annotation tasks.