SC20 Transparency and Reproducibility Initiative Discusses Early Findings of Community Survey
The SC20 Transparency and Reproducibility Initiative undertook a survey in August of this year to study perceptions within the community of the measures implemented by SC to make results from the Technical Program and Workshops more transparent and reproducible. SC’s measures for increased reproducibility take the form of additional information collected at the time of submission of papers to the Technical Program about the experimental activity that supports the results in the manuscript. This information is submitted as an appendix to the manuscript and published as an appendix to the final published version in the SC Conference Proceedings.
The study is timely, as the reproducibility initiative has been in effect since SC15, when authors of SC15 papers were invited to submit an Artifacts Description (AD) Appendix after the conference.
While analysis of the data is ongoing, early findings speak to the strength and value of the sustained effort towards transparency and reproducibility at SC.
After six years of the initiative, a reassuring result of the survey is that a full 90% of the community respondents are aware of the issues around reproducibility. Only 15% think the concerns about reproducibility in science are exaggerated, and only 7% think that the concerns about reproducibility in science do not apply to computer and computational science.
An important impact of the SC reproducibility initiative is whether the reproducibility measures result in published papers having greater impact. The results suggest this to be true. Thirty five percent (35%) of the respondents have used the appendices information from papers in the SC Conference Proceedings in their research.
Food for thought for the future of the SC reproducibility effort: 75% of respondents strongly agreed or somewhat agreed with the statement that transparency (and not reproducibility) is the goal of the SC AD/AE appendices. This result is somewhat surprising and suggests that the visibility that has come through the additional appendix information is overwhelmingly seen as a benefit of the SC reproducibility measures.
The survey was open August 17–31, 2020 and had 204 survey respondents. A manuscript is in preparation that discusses the study, analysis, and detailed conclusions. The study was conducted under Indiana University protocol #2005780098, “Assessing Reproducibility Initiative of IEEE/ACM SC Conference,” led by Beth Plale and Line Pouchard.
Contact the Transparency and Reproducibility Committee for more details.
Beth Plale, SC20 Transparency and Reproducibility Initiative Chair
Line Pouchard, SC20 Transparency and Reproducibility Initiative Evaluation Chair