SC has been a leader in tangible progress towards scientific rigor, through its pioneering practice of enhanced reproducibility of accepted papers. The SC20 initiative builds on this success by continuing the practice of using appendices to enhance scientific rigor. SC20 sees the initiative expanded to encompass transparency and reproducibility for the broader objective of enhanced scientific rigor, and streamlined for reduced researcher burden.
Transparency of research results is achieved by papers that, for instance, provide evidence for why one algorithm worked and others do not, or provide evidence of a rigorous criteria having been followed to choose one software architecture over another. In the specific case of AI transparency, ad hoc methods (e.g., to tune an AIs’ “learning rates”—how much an algorithm corrects itself after each mistake) are sometimes used without justification for why one ad hoc method is better than others. Author attention to enhancing the transparency of their science methodologies is a complement to reproducibility details in strengthening the overall rigor of a scientific publication.
The SC20 Transparency and Reproducibility Initiative impacts technical papers and their submission and peer review in the following ways:
- The Artifact Description (AD) Appendix will be auto-generated from author responses to a standard form embedded in the online submission system. It will have greater emphasis on linking to the broader products resulting from research. The AD is required but an indication that no artifacts were created or used is acceptable. Accepted papers with available artifacts will be honored with the ACM badge.
- The Artifact Evaluation (AE) remains optional.
- Transparency of research results. Papers making fundamental, non-performance related research contributions to algorithms, or their application within a software architecture should address in the body of their manuscript the questions of transparency raised. If additional resources need to be included, they can be linked to in the AD appendix.
The SC20 Transparency and Reproducibility Initiative accomplishes its work through the following three tracks/committees:
- Artifact Description/Artifact Evaluation Appendices Track
- Reproducibility Challenge Track
- Initiative Evaluation: Taking the Temperature Track
SC Transparency and Reproducibility Subcommittees
AD/AE Appendices Track
This subcommittee has the following tasks within its charge:
- Manage and/or develop transparency and reproducibility FAQ and other resources as aid to authors during submission. Could include one-on-one assistance with prospective author.
- Develop and publicly release a rubric for reviewing appendices. This release will occur prior to the paper submission deadline.
- Review the appendices of submitted papers according to the rubric. Provide a justified assessment to the PC who will incorporate this recommendation into their final determination.
This subcommittee has the following tasks within its charge:
- Select a paper accepted to the previous conference to be used as source of the Reproducibility Challenge in the Student Cluster Competition (SCC) of the next conference;
- Work with the authors of the selected paper to build the challenge benchmark for the SCC teams;
Continue working until the time of the conference in crafting the rules and requirements for the challenge, and ensure compatibility with the various hardware used by the SCC teams;
- Convene a subgroup to collect publish-quality reports from each student team who participated in the reproducibility challenge of SCC. The reports in this collection will have a citable DOI and be made available through an archive venue.
Transparency and Reproducibility Initiative Evaluation
This subcommittee, new to SC20, is charged with soliciting feedback from the community about the transparency and reproducibility initiative. It has the following tasks within its charge:
- Develop a survey instrument
- Administer the survey
- Compile results into a report that is made publicly available
AD/AE Appendix Form: Its Role in a Paper and Its Review
All manuscripts submitted to the SC20 Technical Papers program must contain an AD Appendix. The AD Appendix describes the significant research products and other evidence needed for greater transparency and rigor of the scientific conclusions of the manuscript. If a manuscript’s scientific findings have no significant supporting research products or require no additional evidence in support of its scientific conclusions, the AD appendix can easily be generated to reflect this fact. That is, the AD form is mandatory but the author makes the determination whether additional evidence is needed for their particular scientific contribution. The AE appendix is optional. The AD/AE form is standard, and embedded in the conference submission system.
Role of AD/AE Appendix during review: Submissions to SC are double-blind reviewed. SC Papers Committee (PC) reviewers will have only the information in the AD/AE form available to them that does not compromise the double-blind protection.
The AD/AE Appendices Committee works independently of the SC PC because supporting materials will, by their nature, frequently reveal the identity of the submitter. That is, the AD/AE Appendices Committee works under a double-open arrangement (authors and committee members are known to each other). The AD/AE Appendices Committee will develop and publicly release a rubric for reviewing appendices. Appendices of submitted papers are reviewed according to the rubric. The AD/AE Appendices Committee results are recommendations to the PC chairs. That is, the committee will provide a justified assessment to the PC chairs who will incorporate this recommendation into their final acceptance determination.
Additional notes to authors:
- The on-line submission form accommodates both Artifact Description (required) and Artifact Evaluation (optional).
- The appendices are intended for supplemental material only. Specifically, the research or experimental methodology is critical to understanding the scientific contribution so must be included in the body of the paper.
- Please contact us with your questions.
History of the SC Reproducibility Initiative
AD Appendices were mandatory for all submissions. AE Appendices were still optional, and both were submitted via a standard form in the conference submission system. Three new Technical Program tracks, with their respective committees and chairs, were introduced in support of the SC Reproducibility Initiative.
SC extended the option of submitting AD Appendices to Workshops and Posters. The CRA Appendix was renamed Artifact Evaluation (AE) Appendix, and limited to four pages. AD Appendices were limited to 2 pages and remained optional (but required for consideration as Best Paper/Best Student Paper, and also Best Poster/Best Student Poster).
SC made the AD Appendix a requirement to be considered for the Best Paper or Best Student Paper awards. SC17 also introduced the Computational Results Analysis (CRA) Appendix. 40% of submitted and 50% of accepted papers included an AD appendix; nine submitted papers (six accepted) included a CRA Appendix.
Authors submitting to the SC16 conference could optionally submit an AD Appendix: nine authors submitted one, three were finalists, and one was selected to become the source for the SC17 Student Cluster Competition Reproducibility Challenge.
The SC steering committee approved the reproducibility initiative. Authors of SC15 papers were invited to submit an AD Appendix after the conference: one paper did so, became the source for the SC16 Student Cluster Competition Reproducibility Challenge and the first SC paper to display an ACM badge.