• Skip to main content
  • Skip to primary sidebar
AAAI

AAAI

Association for the Advancement of Artificial Intelligence

    • AAAI

      AAAI

      Association for the Advancement of Artificial Intelligence

  • About AAAIAbout AAAI
    • AAAI Officers and Committees
    • AAAI Staff
    • Bylaws of AAAI
    • AAAI Awards
      • Fellows Program
      • Classic Paper Award
      • Dissertation Award
      • Distinguished Service Award
      • Allen Newell Award
      • Outstanding Paper Award
      • Award for Artificial Intelligence for the Benefit of Humanity
      • Feigenbaum Prize
      • Patrick Henry Winston Outstanding Educator Award
      • Engelmore Award
      • AAAI ISEF Awards
      • Senior Member Status
      • Conference Awards
    • AAAI Resources
    • AAAI Mailing Lists
    • Past AAAI Presidential Addresses
    • Presidential Panel on Long-Term AI Futures
    • Past AAAI Policy Reports
      • A Report to ARPA on Twenty-First Century Intelligent Systems
      • The Role of Intelligent Systems in the National Information Infrastructure
    • AAAI Logos
    • News
  • aaai-icon_ethics-diversity-line-yellowEthics & Diversity
  • Conference talk bubbleConferences & Symposia
    • AAAI Conference
    • AIES AAAI/ACM
    • AIIDE
    • IAAI
    • ICWSM
    • HCOMP
    • Spring Symposia
    • Summer Symposia
    • Fall Symposia
    • Code of Conduct for Conferences and Events
  • PublicationsPublications
    • AAAI Press
    • AI Magazine
    • Conference Proceedings
    • AAAI Publication Policies & Guidelines
    • Request to Reproduce Copyrighted Materials
  • aaai-icon_ai-magazine-line-yellowAI Magazine
    • Issues and Articles
    • Author Guidelines
    • Editorial Focus
  • MembershipMembership
    • Member Login
    • Developing Country List
    • AAAI Chapter Program

  • Career CenterCareer Center
  • aaai-icon_ai-topics-line-yellowAITopics
  • aaai-icon_contact-line-yellowContact

Home / Conferences / AAAI Conference on Artificial Intelligence / AAAI 2023 Conference /

Reproducibility Checklist

January 29, 2023

Reproducibility Checklist

Unless specified otherwise, please answer “yes” to each question if the relevant information is described either in the paper itself or in a technical appendix with an explicit reference from the main paper. If you wish to explain an answer further, please do so in a section titled “Reproducibility Checklist” at the end of the technical appendix.

This paper

  • Includes a conceptual outline and/or pseudocode description of AI methods introduced (yes/partial/no/NA)
  • Clearly delineates statements that are opinions, hypothesis, and speculation from objective facts and results (yes/no)
  • Provides well marked pedagogical references for less-familiare readers to gain background necessary to replicate the paper (yes/no)

Does this paper make theoretical contributions? (yes/no)

If yes, please complete the list below.

  • All assumptions and restrictions are stated clearly and formally. (yes/partial/no)
  • All novel claims are stated formally (e.g., in theorem statements). (yes/partial/no)
  • Proofs of all novel claims are included. (yes/partial/no)
  • Proof sketches or intuitions are given for complex and/or novel results. (yes/partial/no)
  • Appropriate citations to theoretical tools used are given. (yes/partial/no)
  • All theoretical claims are demonstrated empirically to hold. (yes/partial/no/NA)
  • All experimental code used to eliminate or disprove claims is included. (yes/no/NA)

Does this paper rely on one or more datasets? (yes/no)

If yes, please complete the list below.

  • A motivation is given for why the experiments are conducted on the selected datasets (yes/partial/no/NA)
  • All novel datasets introduced in this paper are included in a data appendix. (yes/partial/no/NA)
  • All novel datasets introduced in this paper will be made publicly available upon publication of the paper with a license that allows free usage for research purposes. (yes/partial/no/NA)
  • All datasets drawn from the existing literature (potentially including authors’ own previously published work) are accompanied by appropriate citations. (yes/no/NA)
  • All datasets drawn from the existing literature (potentially including authors’ own previously published work) are publicly available. (yes/partial/no/NA)
  • All datasets that are not publicly available are described in detail, with explanation why publicly available alternatives are not scientifically satisficing. (yes/partial/no/NA)

Does this paper include computational experiments? (yes/no)

If yes, please complete the list below.

  • Any code required for pre-processing data is included in the appendix. (yes/partial/no).
  • All source code required for conducting and analyzing the experiments is included in a code appendix. (yes/partial/no)
  • All source code required for conducting and analyzing the experiments will be made publicly available upon publication of the paper with a license that allows free usage for research purposes. (yes/partial/no)
  • All source code implementing new methods have comments detailing the implementation, with references to the paper where each step comes from (yes/partial/no)
  • If an algorithm depends on randomness, then the method used for setting seeds is described in a way sufficient to allow replication of results. (yes/partial/no/NA)
  • This paper specifies the computing infrastructure used for running experiments (hardware and software), including GPU/CPU models; amount of memory; operating system; names and versions of relevant software libraries and frameworks. (yes/partial/no)
  • This paper formally describes evaluation metrics used and explains the motivation for choosing these metrics. (yes/partial/no)
  • This paper states the number of algorithm runs used to compute each reported result. (yes/no)
  • Analysis of experiments goes beyond single-dimensional summaries of performance (e.g., average; median) to include measures of variation, confidence, or other distributional information. (yes/no)
  • The significance of any improvement or decrease in performance is judged using appropriate statistical tests (e.g., Wilcoxon signed-rank). (yes/partial/no)
  • This paper lists all final (hyper-)parameters used for each model/algorithm in the paper’s experiments. (yes/partial/no/NA)
  • This paper states the number and range of values tried per (hyper-) parameter during development of the paper, along with the criterion used for selecting the final parameter setting. (yes/partial/no/NA)

 

Categories: 2023, aaai

Primary Sidebar

We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept All”, you consent to the use of ALL the cookies. However, you may visit "Cookie Settings" to provide a controlled consent.
Cookie SettingsAccept All
Manage consent

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
CookieDurationDescription
cookielawinfo-checkbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
Functional
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
Performance
Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
Analytics
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
Advertisement
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
Others
Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.
SAVE & ACCEPT