Evaluating Comparative Evaluation Models

Retrieval Algorithms: Empirical Evaluations

In initially developing a model of case-based comparative evaluation, we have followed a plan that involved an early empirical evaluation by. a noted domain expert. We describe the empirical evaluation and the lessons learned after we introduce the program and its knowledge sources and work through an extended example. TRUTH-TELLER, a program for testing a case-based comparative evaluation model in practical ethics, compares cases presenting ethical dilemmas about whether to tell the truth. Its comparisons list ethically relevant similarities and dit~erences (i.e., reasons for telling or not telling the truth which apply to both cases, and reasons which apply more strongly in one case than another or which apply only to one case). In generating context-sensitive comparisons, the program reasons about reasons which may invoke ethical principles or selfish considerations. In a preliminary evaluation, a professional ethicist scored the program’s output for randomly-selected pairs of cases.