The Right Threshold Value: What Is the Right Threshold of Cosine Measure When Using Latent Semantic Analysis for Evaluating Student Answers?

May 2003

Proceedings of the Sixteenth International Florida Artificial Intelligence Research Society Conference (FLAIRS 2003)

Volume

Proceedings of the Sixteenth International Florida Artificial Intelligence Research Society Conference (FLAIRS 2003)

All Papers

Auto Tutor is an intelligent tutoring system that holds conversations with learners in natural language. Auto Tutor uses Latent Semantic Analysis (LSA) to match sentences the student generates in response to essay type questions to a set of sentences (expectations) that would appear in a complete and correct response or which reflect common but incorrect understandings of the material (bads). The correctness of student contributions is decided using a threshold value of the LSA cosine between the student answer and the expectations. Our results indicate that the best agreement between LSA matches and the evaluations of subject matter experts can be obtained if the cosine threshold is allowed to be a function of the lengths of both student answer and the expectation being considered.