AAAI Publications, 2018 AAAI Spring Symposium Series

Font Size: 
Trustworthy Automated Essay Scoring without Explicit Construct Validity
Patti West-Smith, Stephanie Butler, Elijah Mayfield

Last modified: 2018-03-15

Abstract


Automated essay scoring (AES) is a broadly used application of machine learning, with a long history of real-world use that impacts high-stakes decision-making for students. However, defensibility arguments in this space have typically been rooted in hand-crafted features and psychometrics research, which are a poor fit for recent advances in AI research and more formative classroom use of the technology. This paper proposes a framework for evaluating automated essay scoring models trained with more modern algorithms, used in a classroom setting; that framework is then applied to evaluate an existing product, Turnitin Revision Assistant.

Keywords


automated essay scoring; data annotation; construct validity

Full Text: PDF