SC is an unquestioned pioneer in improving scientific rigor of the works published in its conference proceedings. Over the past 5 years, the SC reproducibility initiative has strived to enhance reproducibility, offering appendices through which the community is given additional detail (beyond what the typical paper structure allows) to build upon a research result. This post is intended for authors and the general community, to call your attention to a few new developments. Reproducibility is a work in progress, and responsiveness to community needs is an ongoing priority.
SC20 Transparency and Reproducibility Initiative Chair
Q: The name of the initiative is now the “Transparency and Reproducibility Initiative”, how does that affect authors?
A: The new naming is recognition of the growing prevalence of AI-based innovation in HPC, and the awareness that the established norms in HPC paper structuring often impede key decisions. For instance, research results should include evidence for why one algorithm worked and others do not, or why one software architecture is chosen over another. Also, ad hoc methods (e.g., to tune an AIs’ “learning rates”—how much an algorithm corrects itself after each mistake) are sometimes used without justification for why one ad hoc method is better than others.
The call for participation for SC20 has been edited to address the need for greater transparency in submitted manuscripts. Papers making fundamental, non-performance related research contributions to algorithms, or their application within a software architecture, should address in the body of their manuscript the questions of transparency raised above. If additional resources need to be included, they should be linked to in the AD appendix.
Q: How are the Appendices (AD/AE) reviewed?
A: Since its inception the initiative has striven for appendices that improve scientific rigor. Thus, and because appendix information will frequently reveal the identity of the author, the SC AD/AE Appendices Committee (SC AC) is independent of the SC Papers Committee (SC PC). The SC AC works under a double-open arrangement where authors and committee members are known to each other, and the SC AC team is poised to help authors improve their appendices.
SC20 further clarifies that the results of SC AC review of appendices is in the form of recommendations to the SC PC chairs, and that these recommendations are to be provided to the SC PC at each of the multiple points of SC PC review of submitted manuscripts.
Q: Any other important information to share?
A: We’re excited about a new initiative that we are working on this year to take the temperature of the community. SC has nurtured the reproducibility activity over the last 5 years, and the community has had this long to not only respond as authors, but to benefit from the potential the initiative provides for new discovery. The transparency and reproducibility team is building a community survey that should be coming out soon. We are also looking forward to supporting the student reproducibility challenge as in years past. Consult with the SC20 Transparency and Reproducibility Leadership Team, members listed below, and stay tuned!
SC20 Transparency and Reproducibility Leadership Team:
- Beth Plale, Chair, Indiana University
- Stephen Harrell, Purdue University
- John Linford, ARM
- Line Pouchard, Brookhaven National Lab
- Andrew Younge, Sandia National Lab
SC20 Communications Team