Two sentence summary: The ASPLOS’21 PC has taken specific steps to protect the integrity of the review process and significantly reduce the potential impact of collusion on the outcome for submitted papers.
With regard to the review process for submitted papers, the ASPLOS’21 PC chairs designed and implemented a process that we believe significantly reduces the potential that collusion can affect paper outcomes. The following describes the steps we have taken:
1. No bidding for papers.
Reviewers were not able to ask for specific papers, e.g., those of their “friends”. All review assignments, including PC, ERC, and external reviewers, were done by the PC co-chairs. As such, the likelihood of a group of “friends” reviewing a paper is much lower.
2. Every paper was reviewed by 5 or more PC members across both review rounds.
Five or more reviewers makes it difficult for a small number of people to have a disproportionate impact on the paper’s outcome. The large number of PC members involved in the review process allows us to identify unusual behavior, raise concerns about reviews, and have a good understanding of the actual strengths and weaknesses of each paper.
3. The online discussion is blind.
While the reviewers discuss the papers, they don’t know others’ identities beyond reviewer #A, #B, …. Hence, a single reviewer cannot easily assert seniority and silence other voices, or influence them beyond the strength of their arguments.
4. We have repeatedly asked the PC members to watch out for abnormally positive or negative behavior.
PC members have been asked to notify us when they see a reviewer being unreasonably positive or negative about a paper. We have also been tracking when a review abruptly changes in sentiment or score.
5. PC members can dispute an online accept or reject decision.
A PC member can dispute an online decision for a paper they reviewed. This will automatically trigger a discussion at the PC meeting. Hence, if a single PC member thinks the other 4 or more PC members are making an unreasonable or improper decision, the paper will be discussed by the entire PC.
6. All interactions with HotCRP are logged.
All access to a paper, reviews, changes to reviews, and discussion is logged by HotCRP. This record will be preserved, and, if necessary, used to identify improper behavior.
We believe that these measures make it significantly more difficult for systematic collusion to be effective. It is still possible for a “friendly” reviewer to be assigned to a paper and provide a positive score that pushes a marginal paper over the hill. However, it is much more difficult to get 2-3 “friendly” reviewers that push a weak paper over the hill without being detected and challenged.
Signed,
Emery Berger & Christos Kozyrakis (ASPLOS’21 Program Committee co-chairs)
with input from and endorsed by the ASPLOS Steering Committee