AAAI Publications, Thirty-First AAAI Conference on Artificial Intelligence

Font Size: 
Proper Proxy Scoring Rules
Jens Witkowski, Pavel Atanasov, Lyle H. Ungar, Andreas Krause

Last modified: 2017-02-10

Abstract


Proper scoring rules can be used to incentivize a forecaster to truthfully report her private beliefs about the probabilities of future events and to evaluate the relative accuracy of forecasters. While standard scoring rules can score forecasts only once the associated events have been resolved, many applications would benefit from instant access to proper scores. In forecast aggregation, for example, it is known that using weighted averages, where more weight is put on more accurate forecasters, outperforms simple averaging of forecasts. We introduce proxy scoring rules, which generalize proper scoring rules and, given access to an appropriate proxy, allow for immediate scoring of probabilistic forecasts. In particular, we suggest a proxy-scoring generalization of the popular quadratic scoring rule, and characterize its incentive and accuracy evaluation properties theoretically. Moreover, we thoroughly evaluate it experimentally using data from a large real world geopolitical forecasting tournament, and show that it is competitive with proper scoring rules when the number of questions is small.

Keywords


Information Elicitation; Forecasting; Proper Scoring Rules; Peer Prediction; Mechanism Design; Incentive Schemes

Full Text: PDF