Reproducibility Checklist

Unless specified otherwise, please answer “yes” to each question if the relevant information is described either in the paper itself or in a technical appendix with an explicit reference from the main paper. If you wish to explain an answer further, please do so in a section titled “Reproducibility Checklist” at the end of the technical appendix.

This paper:

  • clearly states what claims are being investigated (yes/partial/no)
  • explains how the results substantiate the claims (yes/partial/no)
  • explicitly identifies limitations or technical assumptions  (yes/partial/no)
  • includes a conceptual outline and/or pseudocode description of AI methods introduced (yes/partial/no/NA)

Does this paper make theoretical contributions? (yes/no) 

If yes, please complete the list below. 

  • All assumptions and restrictions are stated clearly and formally. (yes/partial/no)
  • All novel claims are stated formally (e.g., in theorem statements). (yes/partial/no)
  • Proofs of all novel claims are included. (yes/partial/no)
  • Proof sketches or intuitions are given for complex and/or novel results. (yes/partial/no)
  • Appropriate citations to theoretical tools used are given. (yes/partial/no)

Does this paper rely on one or more data sets? (yes/no) 

If yes, please complete the list below. 

  • All novel datasets introduced in this paper are included in a data appendix (yes/partial/no/NA)
  • All novel datasets introduced in this paper will be made publicly available upon publication of the paper with a license that allows free usage for research purposes (yes/partial/no/NA)
  • All datasets drawn from the existing literature (potentially including authors’ own previously published work) are accompanied by appropriate citations (yes/no/NA)
  • All datasets drawn from the existing literature (potentially including authors’ own previously published work) are publicly available (yes/partial/no/NA)
  • All datasets that are not publicly available are described in detail (yes/partial/no/NA)

Does this paper include computational experiments? (yes/no) 

If yes, please complete the list below.

  • All source code required for conducting experiments is included in a code appendix (yes/partial/no). 
  • All source code required for conducting experiments will be made publicly available upon publication of the paper with a license that allows free usage for research purposes (yes/partial/no)
  • If an algorithm depends on randomness, then the method used for setting seeds is described in a way sufficient to allow replication of results. (yes/partial/no/NA)
  • This paper specifies the computing infrastructure used for running experiments (hardware and software), including GPU/CPU models; amount of memory; operating system; names and versions of relevant software libraries and frameworks. (yes/partial/no)
  • This paper formally describes evaluation metrics used and explains the motivation for choosing these metrics (yes/partial/no)
  • This paper states the number of algorithm runs used to compute each reported result (yes/no)
  • Analysis of experiments goes beyond single-dimensional summaries of performance (e.g., average; median) to include measures of variation, confidence, or other distributional information (yes/no)
  • This paper lists all final (hyper-)parameters used for each model/algorithm in the paper’s experiments (yes/partial/no/NA)
  • This paper states the number and range of values tried per (hyper-)parameter during development of the paper, along with the criterion used for selecting the final parameter setting. (yes/partial/no/NA)

This site is protected by copyright and trademark laws under US and International law. All rights reserved. Copyright © 1995–2021 AAAI