Accurate or Artificial?

Abstract

Peer review stands as the foundation of academic publishing, shaping the quality and credibility of ‎scholarly work. However, concerns remain about its consistency and reliability. The emergence of ‎large language models (LLMs) presents a transformative opportunity in this regard. This study ‎examines the quality of peer review feedback generated by a general AI tool, ChatGPT, and a ‎specialized one, SCiNiTO, compared to human reviewers. Four research articles, each previously ‎reviewed by human experts with different conclusions, were uploaded to the selected LLMs for ‎evaluation. The review reports of the four research articles were analyzed for the final decision ‎reached and the score of certain quality dimensions based on a developed rubric. The analysis showed that ‎both AI tools gave almost similar decisions, but they were mostly different from the decisions given ‎by human reviewers. In addition, AI tools showed superior quality performance. Eventually, ‎the study findings uncover AI's ‎promise and possible implications toward enhancing academic evaluation and communication.‎