AAAI Publications, 2011 AAAI Fall Symposium Series

Font Size: 
Evaluating Questions in Context
Lee Becker, Martha S. Palmer, Sarel van Vuuren, Wayne H. Ward

Last modified: 2011-11-03

Abstract


We present an evaluation methodology and a system for ranking questions within the context of a multimodal tutorial dialogue. Such a framework has applications for automatic question selection and generation in intelligent tutoring systems. To create this ranking system we manually author candidate questions for specific points in a dialogue and have raters assign scores to these questions. To explore the role of question type in scoring, we annotate dialogue turns with labels from the DISCUSS dialogue move taxonomy. Questions are ranked using a SVM-regression model trained with features extracted from the dialogue context, the candidate question, and the human ratings. Evaluation shows that our system’s rankings correlate with human judgments in question ranking.

Keywords


Question Generation, Question Ranking, Tutorial Dialogue)

Full Text: PDF