Experts discuss how a global pandemic is putting pressure on traditional methods of research dissemination.
A search on PubMed.gov for COVID-19 reveals 93,851 results as of Mid-January 2021.
“To put that in perspective, if I search for EGFR, a major area of cancer research, there are just over 96,000 results in the 40 years since 1980,” said Leora Horn, MD, MSc, Ingram Associate Professor of Cancer Research at Vanderbilt-Ingram Cancer Center and founding member of TERAVOLT (Thoracic Cancers International COVID-19-Collaboration). “This is like nothing we have seen before.”
Data about COVID-19 and Sars-CoV-2 is coming out fast and furious, she said, which has both pros and cons. On the positive side is the incredible global collaboration of people and specialties working together to figure out this disease and how to care for patients. On the negative side,not all of the information may be properly vetted or verified.
Retractionwatch.com currently lists 32 publications related to COVID-19 as retracted, the majority of which were published on preprint servers like medRxiv or SSRN.1 However, two papers published in May 2020 in major medical journals were retracted in early June 2020 when the authors were unable to complete an independent audit of the data source used for the studies.2,3
“As researchers and clinicians, we look for answers in the medical and scientific literature, with their well-established and trusted processes for peer review,” said Neal J. Meropol, MD, vice president, head of medical and scientific affairs, Flatiron Health. “This pandemic has added a great deal of pressure to these processes all across the healthcare universe. Everyone from doctors and nurses to researchers, peer reviewers, and journal editors is acutely aware of the high stakes and the urgency to share knowledge as quickly as possible.”
Pulled Papers
On May 1, The New England Journal of Medicine published a paper by Mehra and colleagues that suggested that underlying cardiovascular disease was linked with increased risk of in-hospital death among patients hospitalized with COVID-19.2 The data from this study were taken from the observational database Surgical Outcomes Collaborative maintained by Surgisphere, which included data from 169 hospitals in Asia, Europe, and North America.
On May 22, another study with data from the Surgical Outcomes Collaborative database was published in The Lancet.3 This study dealt with hydroxychloroquine and chloroquine—drugs steeped in controversy in the United States—as treatments for COVID-19. The study was unable to confirm a benefit of the drugs used alone or in combination with a macrolide on in-hospital outcomes for COVID-19. It included registry data from 671 hospitals on six continents.
On June 2, after a series of scientific questions were raised about the study, Lancet editors published an “expression of concern” about the data in the hydroxychloroquine paper.4 On June 4, three of the four authors pulled the paper. The abstaining author was the founder of Surgisphere, who would not transfer the full dataset on request. Later that day, the authors published a retraction of the New England Journal of Medicine study as well.5
Consequences
Given the urgency of the global pandemic, some high-profile scientific journals have altered normal publication practices to more rapidly disseminate information. For example, The New England Journal of Medicine announced it would “rapidly evaluate submitted manuscripts” and “expedite all editorial steps to make them available as quickly as possible.”6
“With this rapid review, there may be some inherent bias in the people reviewing it, or papers may be coming out without all the questions answered,” Dr. Horn said.
Although these are only two retractions in major journals among the large number of papers published, they are not without consequences. The original publication of the hydroxychloroquine study prompted the World Health Organization to halt a study of the drug. The trial was later resumed but has since been permanently halted.
“High-profile scenarios like these certainly cause soul-searching throughout the research community, as all stakeholders—including researchers, academic institutions, and journals—want to ensure that we are conducting and disseminating the highest quality science,” Dr. Meropol said. “The biggest issue is guarding against any loss of trust—in scientists, in research institutions, in journals, in science itself. We are in desperate need of evidence to understand COVID-19’s effects on patients and to guide prevention and treatment, and the scientific process is the only way to get there,” he said.
Balancing Speed, Integrity
Even with the pressure to publish and the public’s thirst for information, data integrity must be ensured, according to Baktiar Hasan, PhD, a biostatistician with the European Organisation for Research and Treatment of Cancers.
With the advent of the idea of “big data” taken from registry and electronic health records, Dr. Hasan worries that people may forget the importance of clinical trials. Prospective clinical trials are the result of detailed preparation in which everything is planned in advance and there is a high confidence in the integrity of the data, he said.
“Big datasets are not prospective data, where everything is planned,” Dr. Hasan said. “The validation of these data is a huge challenge.” The information in these sets may vary in their definitions and terminology; outcome definitions may be not be the same; and there may be differences depending on the country the data are from, the type of health system, or more.
However, in the case of COVID-19, some published studies are occurring with the use of data taken from “real-world” scenarios and not from clinical trials, let alone randomized clinical trials.
“Real-world evidence can add vital information to the gold standard of randomized clinical trials. The scientific community now has a renewed opportunity—and a responsibility—to align on best practices for sharing early research in a responsible way that engenders trust,” Dr. Meropol said, especially with observational studies occurring in near real time. “At a minimum, such best practices would include transparency about the source of data and analysis methods, as well as the limitations and uncertainty about the findings.”
Integrity of data will always be a key component to making sure studies are accurate, Dr. Horn said. That is the only way to ensure public and community trust.
Dr. Meropol agreed: “Authors of research studies must be able to stand behind their conclusions and the ways in which they reached those conclusions, including the data itself, as well as the study design, analyses, and results. Even authors who might not have direct oversight over the data generation should have adequate visibility to ensure their comfort with the data origin, characteristics, and veracity.”
Balancing the need for speed of publication against the integrity of data is difficult in the pandemic, Dr. Hasan said. However, precision should not be bypassed for speed. “The reviewing process may be shortened to make it more efficient, but not if it sacrifices the quality of the review,” Dr. Hasan said. “That may need to be done by expanding the pool of qualified reviewers so that they can still pick up on any issues with the integrity of the data.”
Dr. Hasan acknowledged that policy and decision makers are looking to the research community to help guide important, life-changing decisions. “If these decisions are being made on half-good data or unreliable data, it could have a huge impact,” he said.
References:
- Retracted coronavirus (COVID-19) papers. Retraction Watch. Accessed August 28, 2020. https://retractionwatch.com/retracted-coronavirus-covid-19-papers/
- Mehra MR, Desai SS, Kuy S, et al. Cardiovascular disease, drug therapy, and mortality in Covid-19. N Engl J Med. 2020;382(25):e102. Retracted in: N Engl J Med. 2020;382:2582.
- Mehra MR, Desai SS, Ruschitzka F, Patel AN. RETRACTED: Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis. Lancet. 2020.
- The Lancet Editors. Expression of concern: Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis. Lancet. 2020.
- Mehra MR, Desai SS, Kuy S, Henry TD, Patel AN. Retraction: Cardiovascular disease, drug therapy, and mortality in Covid-19. N Engl J Med. 2020.
- Rubin EJ, Baden LR, Morrissey S, Campoin EW. Medical journals and the 2019-nCOV outbreak. N Engl J Med. 2020;382:866.
