A Retrieval-Augmented Large Language Model Framework for Trait-Level Automated Essay Scoring

Abstract

Automatic Essay Grading (AEG) aims to automatically evaluate student essays using natural language processing (NLP) and artificial intelligence (AI). Recent advances in transformer-based models and large language models (LLMs), including BERT and GPT, have significantly improved automated essay scoring by capturing rich semantic and contextual information. This paper proposes a Retrieval-Augmented Large Language Model (RAG-LLM) framework for trait-level automated essay scoring and evaluates its performance against LSTM-, BERT-, and GPT-based approaches. The proposed framework evaluates multiple writing traits, including Content, Organization, Word Choice, Sentence Fluency, Conventions, Prompt Adherence, Language, Narrativity, Style, and Voice. Experiments conducted on the ASAP dataset demonstrate that the proposed RAG-LLM framework achieves the highest agreement with human raters, outperforming fine-tuned BERT multi-head models and GPT-based prompting approaches across most essay prompts and evaluation traits. Comprehensive prompt-wise, trait-wise, and ablation analysis further demonstrate the robustness of the proposed framework and provide insights into prompt-specific performance variation and the contribution of key architectural components. The results indicate that integrating semantic retrieval with large language model reasoning provides an effective and scalable solution for trait-level automated essay scoring.