Does Peer Reviewing for COVID-19-Related Papers Still Work?

Abstract

In this article, we aimed to analyze whether coronavirus disease 2019 (COVID-19)-associated articles were being subjected to the same standards of peer-review as non-COVID-19 articles. In order to do this, we taught eight PhD students manuscript reviewing skills and analyzed eight papers published in valued journals, five of them on COVID-19. Each selected publication was reviewed by at least two graduate students from a Scientific English class and two scientists in charge of the course at the Faculty of Medicine, Universidad de Chile. Several shortcomings were identified in the revised studies, particularly on those related to COVID-19, which led us to conclude that the emergency imposed by the COVID-19 has endangered the quality of the accepted studies. The COVID-19 pandemic caused by the novel Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) has rapidly spread throughout the world. This virus is killing many people, and taking a massive physical, as well as mental toll, on the lives of all those that have been infected (Yi et al., 2020). The COVID-19 pandemic has severely depressed every country’s economy because governments have been obliged to apply quarantine measures to control the disease. Thus, the impact has not only been on people’s health, but also on their lifestyle and economic situation (Nicola et al., 2020). For these same reasons, many people, including politicians and leaders from different countries, have turned to the scientific community for answers regarding actions that need to be taken to control and treat the disease. Nevertheless, many scientific studies published these days concerning the COVID-19 virus, even those reported by important journals, fall short on experimental evidence to support their conclusions. It is clear that great pressure exists to rapidly know more about this virus and how to stop the pandemic. Notwithstanding, we believe that this is leading editors and reviewers to accept manuscripts that would have never been considered for publication under different circumstances. It is not bad science, it is just not the complete story; the story that the good, high impact journals would normally ask for, when peer reviewing manuscripts for publication. In a Scientific English course for postgraduate students, we taught them how to review a manuscript and gave them several scientific papers on COVID-19 published in prestigious journals. Their task was to elaborate a critique, according to the instructions given by the professors. Additionally, non-COVID-19 articles (but of related subtopics) from similar journals were reviewed by the students as controls. All papers were evaluated by at least two students and reviewed by the two scientists in charge of the course. Considering that: i) journals have received a wealth of manuscripts on COVID-19 and therefore, accelerated the publication reviewing process to allow faster publication, and dissemination of information (pandemic publishing) (Kwon, 2020); ii) the worldwide daily confirmed peak of COVID-19 deaths was at the middle of April 2020 (https://ourworldindata.org/grapher/daily-covid-deaths-region), and started rising in December 2019; and iii) data published in a middle-to-high impact journal (IF > 9; 2019) can cause more damage to the public because it is easier to consider the information as reliable and valid (Kwon, 2020), the papers were selected based on the following criteria: i) paper main topic was on COVID-19; ii) publication date was between January–June 2020; iii) articles were mainly brief reports; only one research article was selected; iv) papers were published by journals with an IF > 9.4 (2019). Additionally, as controls, we selected three papers meeting the same criteria, except they did not cover COVID-19. To perform the critique, we used the criteria described under the sub-item “Critique” in Table 1. These criteria included: structure of the paper, data collection, appropriate methods and controls to gather the evidence, analysis and interpretation of evidence lead the reader to similar conclusions than the authors. TABLE 1. Criteria used to evaluate the work performed by the students. Peer reviewing was performed as suggested by published literature (Benos et al., 2003; McPeek et al., 2009; Lippi, 2018). A paper written by Emmie de Wit and colleagues and published by PNAS in February 2020 (de Wit et al., 2020) was reviewed by all students. In this paper, the authors reported that prophylactic treatment with the antiviral drug Remdesivir prevented clinical manifestations in the lungs of Rhesus macaques infected with MERS-CoV, and provided a clear clinical benefit when the drug was administered post infection. They suggest that Remdesivir could be useful in the treatment of other coronaviruses such as SARS-CoV-2, the virus responsible for COVID-19. Although the article is well backed up, there were a number of important concerns. First, the number of animals in each group of the study was very small (six), which led to a considerable variation in the results observed, making the reported therapeutic effect of Remdesivir questionable. In addition, although the authors had two different types of vehicle control groups (three animals/group), the results were treated as if these two groups were the same. Furthermore, the study lacked a control group without viral inoculation. Additionally, only male animals were used, despite the evidence indicating that adverse effects may vary between male and female animals (Klein, 2012). Finally, important clinical details about the macaques, such as their age—which is known to influence MERS-CoV infection outcome (Garbati et al., 2016)—weight, physical activity, presence of chronic diseases, etc., should have been indicated. Yet another limitation of this publication was the absence of toxicity assays. Measurements of renal clearance, liver or...