Benchmarks, Test Beds, Controlled Experimentation, and the Design of Agent Architectures

Authors

  • Steve Hanks
  • Martha E. Pollack
  • Paul R. Cohen

DOI:

https://doi.org/10.1609/aimag.v14i4.1066

Abstract

The methodological underpinnings of AI are slowly changing. Benchmarks, test beds, and controlled experimentation are becoming more common. Although we are optimistic that this change can solidify the science of AI, we also recognize a set of difficult issues concerning the appropriate use of this methodology. We discuss these issues as they relate to research on agent design. We survey existing test beds for agents and argue for appropriate caution in their use. We end with a debate on the proper role of experimental methodology in the design and validation of planning agents.

Downloads

Published

1993-03-15

How to Cite

Hanks, S., Pollack, M. E., & Cohen, P. R. (1993). Benchmarks, Test Beds, Controlled Experimentation, and the Design of Agent Architectures. AI Magazine, 14(4), 17. https://doi.org/10.1609/aimag.v14i4.1066

Issue

Section

Articles