Double-Blind Review Policy

This document aims to help authors, reviewers, and Papers Chairs understand the double-blind review process that the SC Conference Series has adopted. Please contact us with any questions or comments.

Rationale

Papers submitted to the SC Conference will undergo a double-blind review process. In this process, authors do not know who reviews their papers and reviewers do not see author names.

The primary rationale for double-blind review is to mitigate implicit or explicit bias, as there is ample evidence that double-blind policies can reduce such bias. There is also evidence that papers subjected to double-blind review receive more citations than those that undergo single-blind review; thus double-blind review may also be associated with higher-quality papers.

Please see the references at the end of this document for additional discussion of double-blind review. Those of McKinley (2008; updated in 2015) and Snodgrass (2007) are particularly lucid; McKinley reminds us that “Double-blind is not perfect, just better,” while Snodgrass advises “… uncertainty as to the authors’ identity is often sufficient to realize most or all of the benefits of masking.” In other words, the goal is not “perfect” blinding, but rather enough uncertainty about authorship in the process to reduce actual or perceived bias.

Guidance for Authors

If you are an author, you should write your paper so as not to disclose your identity or the identities of your co-authors. The following guidelines are best practices for “blinding” a submission in a way that should not weaken it or the presentation of its ideas. These guidelines are broken up into the major submission and review phases: while writing (before submitting), at submission time, during the revision and response process.

These practices were distilled from McKinley (2015) and Snodgrass (2007).

If you have additional questions or comments, please contact the Papers Chair.

While Writing

• Do not use your name or your co-authors’ names, affiliations, funding sources, or acknowledgments in the heading or body of the document. It is absolutely fine and encouraged to use the name of the machine you are working on and describe it.
• Do not eliminate self-references to your published work that are relevant and essential to a proper review of your paper solely in an attempt to anonymize your submission. Instead, write self-references in the third person. Recall that the goal and spirit of double-blind review is to create uncertainty about authorship, which is sufficient to realize most of its benefits.
• To reference your unpublished work, use anonymous citations. From Snodgrass (2007): “The authors developed … [1]” where the reference [1] appears as, “[1] Anonymous (omitted due to double-blind review).” You will have a chance to explain these references to the non-conflicted Papers Chair or their designee(s); see At Submission Time, below. See the FAQ for more examples.

At Submission Time

• At submission time, you will be asked to declare conflicts of interest you may have with program committee members. You will also have the option to upload a list of conflicts. Reviewers will be asked separately to verify declared conflicts.
• Suppose you feel that there is supplemental material essential to reviewing your submission, in addition to the Artifact Description, but which would also reveal your identity (e.g., an earlier technical report or published software). In this case, you will have the option to upload or link to those materials and include an explanation at the time of your submission. By default, reviewers will not see this material; instead, the non-conflicted Papers Co-Chairs or their designee(s) will review and use their discretion to decide whether to include such materials during the review.
• Because of the double-blind process, there will no longer be a limit on the number of submissions by reviewers. (Track Co-Chairs are subject to limits.) However, there will be a limit of four accepted papers for reviewers.

During the Review Period

• You are not forbidden from disseminating your work via talks or technical reports. However, you should not try to directly or otherwise unduly influence program committee members who may be reviewing your paper.

During the Revision and Response Period

• During the rebuttal period, authors should still assume double-blind review. Therefore, authors should not disclose their identities in their revision and response to the reviewers. However, as with the original submission, authors will have the option of entering identity-revealing information in a separate part of the revision and response form that will, by default, be visible only to a non-conflicted Chairs, or their designee(s) in the case of conflicts.

Guidance for Papers Chairs and Reviewers

The following is a set of guidelines for the Papers Co-Chairs or Track Co-Chairs (hereafter, “Chair”) and reviewers (i.e., Papers Committee members). Generally speaking, the procedures draw inspiration from the three principles suggested by Snodgrass:

“The first is that authors should not be required to go to great lengths to blind their submissions. The second is that comprehensiveness of the review trumps blinding efficacy. The final principle … is that [editors and chairs] retain flexibility and authority in managing the reviewing process.”

Before the Submissions Deadline

• Correctly identifying conflicts of interest (COIs) is one of the most important procedural aspects of double-blind review. Therefore, before the Papers submissions deadline, Chairs and reviewers should log into the review system to verify and upload their conflicts of interest. This process can be time-consuming, so please plan accordingly.
• Reviewers will be given a list of submitting authors from all tracks who have indicated they have a conflict with that reviewer. The authors will be listed separately from any potential co-authors. This will enable the reviewer to point out to the Papers chair(s) any declared conflicts that are potentially spurious.
• During paper bidding, reviewers should let their Chair know if they suspect a conflict with a submission and what they believe is the nature of the conflict.

During the Review Period

• A reviewer may accidentally discover the identities of the authors during the review. (For instance, he or she might be checking references to determine the novelty of the submission and discover a technical report with the same content.) In this case, the reviewer should disclose this discovery to the Papers Chair. Such incidents do not necessarily “violate” the double-blind policy, and the reviewer may continue to review the paper. The spirit of double-blind reviewing is that reviewers should not actively try to discover who the authors of a submission are.
• A reviewer who thinks he or she knows an author’s identity should not reveal their suspicion in their review or during discussions with other reviewers (whether online or in-person).
• SC Papers follows a “double-blind until accept” procedure. That is, author identities remain hidden until the review committee has determined all of the accepted papers. For rejected papers, author identities remain hidden even after rejection.
• Reviewers who feel that knowing an author’s name or affiliations is necessary to review a submission can make their case to the Papers Chair at any time during the review process.
• Reviewers who wish to ask colleagues to help with reviews must clear these requests with the Chair first and take steps to ensure that the colleague understands the double-blind policy. In any case, a reviewer is responsible for representing their reviews fully.

During the Program Committee Meeting

• Chairs should still observe and manage conflicts as they would in a single-blind review. For instance, they should avoid discussing a paper until all of the paper’s conflicted reviewers have left the room.

References

Papers Frequently Asked Questions (Available Winter 2020)

McKinley, Kathryn S., “More on Improving Reviewing Quality with Double-Blind Reviewing, External Review Committees, Author Response, and in Person Program Committee Meetings,” 2015, http://www.cs.utexas.edu/users/mckinley/notes/blind-revised-2015.html.

Snodgrass, Richard T., “Editorial: Single-Versus Double-Blind Reviewing,” 2007, http://history.acm.org/pictures/tods/tods.acm.org/editorial.pdf.

Laband, D.N. and M. J. Piette, “A citation analysis of blinded peer review,” The Journal of the American Medical Association (JAMA): The Second International Congress on Peer Review in Biomedical Publication, 272(2) (1994):147–149, http://www.ncbi.nlm.nih.gov/pubmed/8015128.