A Method for Evaluating Elicitation Schemes for Probabilities

We present an objective approach for evaluating probability elicitation methods in probabilistic models. Our method draws on ideas from research on learning Bayesian networks: if we assume that the expert’s knowledge is manifested essentially as a database of records that have been collected in the course of the expert’s experience, and if this database of records is available to us, then the structure and parameters of the expert’s beliefs could be reliably constructed using techniques for Bayesian learning from data. This learned model could, in turn, be compared to elicited models to judge the effectiveness of the elicitation process. We describe a general procedure by which it is possible to capture the data corresponding to the expert’s beliefs, and we present a simple experiment in which we utilize this technique to compare three methods for eliciting discrete probabilities: (1) direct numerical assessment, (2) the probability wheel, and (3) the scaled probability bar. We show that for our domain, the scaled probability bar is the most effective tool for probability elicitation.