AAAI

Association for the Advancement of Artificial Intelligence

Home > Proceedings / Proceedings of the AAAI Conference on Artificial Intelligence, 36 > No. 7: AAAI-22 Technical Tracks 7

When AI Difficulty Is Easy: The Explanatory Power of Predicting IRT Difficulty

Authors

Fernando Martínez-Plumed

European Commission, Joint Research Centre Universitat Politècnica de València


David Castellano

Valencian Research Institute for Artificial Intelligence (VRAIN), Universidad Politécnica de Valencia


Carlos Monserrat-Aranda

Valencian Research Institute for Artificial Intelligence (VRAIN), Universitat Politècnica de València


José Hernández-Orallo

Valencian Research Institute for Artificial Intelligence (VRAIN), Universitat Politècnica de València Leverhulme Centre for the Future of Intelligence, University of Cambridge


Proceedings:

No. 7: AAAI-22 Technical Tracks 7

Volume

Issue:

Proceedings of the AAAI Conference on Artificial Intelligence, 36

Track:

AAAI Technical Track on Machine Learning II

Downloads:

Download PDF

Abstract:

One of challenges of artificial intelligence as a whole is robustness. Many issues such as adversarial examples, out of distribution performance, Clever Hans phenomena, and the wider areas of AI evaluation and explainable AI, have to do with the following question: Did the system fail because it is a hard instance or because something else? In this paper we address this question with a generic method for estimating IRT-based instance difficulty for a wide range of AI domains covering several areas, from supervised feature-based classification to automated reasoning. We show how to estimate difficulty systematically using off-the-shelf machine learning regression models. We illustrate the usefulness of this estimation for a range of applications.

DOI:

10.1609/aaai.v36i7.20739


AAAI

Proceedings of the AAAI Conference on Artificial Intelligence, 36