{"id":2173,"date":"2020-11-16T13:47:33","date_gmt":"2020-11-16T18:47:33","guid":{"rendered":"https:\/\/asplos-conference.org\/?page_id=2173"},"modified":"2020-11-16T13:47:33","modified_gmt":"2020-11-16T18:47:33","slug":"process-integrity","status":"publish","type":"page","link":"https:\/\/asplos-conference.org\/process-integrity\/","title":{"rendered":"Statement on the Integrity of the ASPLOS\u201921 Paper Selection Process"},"content":{"rendered":"\n

Two sentence summary<\/strong>: The ASPLOS\u201921 PC has taken specific steps to protect the integrity of the review process and significantly reduce the potential impact of collusion on the outcome for submitted papers. <\/p>\n\n\n\n

\n\n\n\n

With regard to the review process for submitted papers, the ASPLOS\u201921 PC chairs designed and implemented a process that we believe significantly reduces the potential that collusion can affect paper outcomes.  The following describes the steps we have taken:<\/p>\n\n\n\n

1.<\/strong> No bidding for papers.<\/strong><\/p>\n\n\n\n

Reviewers were not able to ask for specific papers, e.g., those of their \u201cfriends\u201d. All review assignments, including PC, ERC, and external reviewers, were done by the PC co-chairs.\u00a0 As such, the likelihood of a group of \u201cfriends\u201d reviewing a paper is much lower.<\/p>\n\n\n\n

2. Every paper was reviewed by 5 or more PC members across both review rounds.<\/strong><\/p>\n\n\n\n

Five or more reviewers makes it difficult for a small number of people to have a disproportionate impact on the paper\u2019s outcome. The large number of PC members involved in the review process allows us to identify unusual behavior, raise concerns about reviews, and have a good understanding of the actual strengths and weaknesses of each paper. <\/p>\n\n\n\n

3. The online discussion is blind.<\/strong><\/p>\n\n\n\n

While the reviewers discuss the papers, they don\u2019t know others’ identities beyond reviewer #A, #B, …. Hence, a single reviewer cannot easily assert seniority and silence other voices, or influence them beyond the strength of their arguments.<\/p>\n\n\n\n

4. We have repeatedly asked the PC members to watch out for abnormally positive or negative behavior.<\/strong><\/p>\n\n\n\n

PC members have been asked to notify us when they see a reviewer being unreasonably positive or negative about a paper. We have also been tracking when a review abruptly changes in sentiment or score.<\/p>\n\n\n\n

5. PC members can dispute an online accept or reject decision.<\/strong><\/p>\n\n\n\n

A PC member can dispute an online decision for a paper they reviewed. This will automatically trigger a discussion at the PC meeting. Hence, if a single PC member thinks the other 4 or more PC members are making an unreasonable or improper decision, the paper will be discussed by the entire PC.<\/p>\n\n\n\n

6. All interactions with HotCRP are logged.<\/strong><\/p>\n\n\n\n

All access to a paper, reviews, changes to reviews, and discussion is logged by HotCRP. This record will be preserved, and, if necessary, used to identify improper behavior.<\/p>\n\n\n\n

We believe that these measures make it significantly more difficult for systematic collusion to be effective. It is still possible for a \u201cfriendly\u201d reviewer to be assigned to a paper and provide a positive score that pushes a marginal paper over the hill. However, it is much more difficult to get 2-3 \u201cfriendly\u201d reviewers that push a weak paper over the hill without being detected and challenged.<\/p>\n\n\n\n

Signed,
Emery Berger & Christos Kozyrakis (ASPLOS’21 Program Committee co-chairs)
with input from and endorsed by the ASPLOS Steering Committee<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

Two sentence summary: The ASPLOS\u201921 PC has taken specific steps to protect the integrity of the review process and significantly reduce the potential impact of collusion on the outcome for … <\/p>\n

Continue reading “Statement on the Integrity of the ASPLOS\u201921 Paper Selection Process”<\/span><\/a><\/p>\n","protected":false},"author":12,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":[],"_links":{"self":[{"href":"https:\/\/asplos-conference.org\/wp-json\/wp\/v2\/pages\/2173"}],"collection":[{"href":"https:\/\/asplos-conference.org\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/asplos-conference.org\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/asplos-conference.org\/wp-json\/wp\/v2\/users\/12"}],"replies":[{"embeddable":true,"href":"https:\/\/asplos-conference.org\/wp-json\/wp\/v2\/comments?post=2173"}],"version-history":[{"count":1,"href":"https:\/\/asplos-conference.org\/wp-json\/wp\/v2\/pages\/2173\/revisions"}],"predecessor-version":[{"id":2174,"href":"https:\/\/asplos-conference.org\/wp-json\/wp\/v2\/pages\/2173\/revisions\/2174"}],"wp:attachment":[{"href":"https:\/\/asplos-conference.org\/wp-json\/wp\/v2\/media?parent=2173"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}