Peer review of scientific papers submitted for publication is designed to vet quality and merit for inclusion in a scientific journal. However, the latest explosion of scientific papers—evidenced most recently by the torrent of those focused on COVID-191
—has birthed a new kind of more immediate peer review: that of members of the scientific community who do not take published literature at face value.
One recent example occurred in late July, when JAMA Pediatrics issued a notice of retraction of a research letter by Harald Walach, PhD, and colleagues that suggested that children forced to wear face masks were breathing in potentially dangerous levels of exhaled carbon dioxide.2
,3
The retraction stated that the paper was pulled because of issues related to the study’s methodology, conclusions, and other issues.
Within days of this paper’s publication on June 30, comments questioning its validity were posted on the JAMA Pediatrics website, and by July 6 the journal’s editor had posted a comment stating that concerns about the study were being reviewed. A little more than 2 weeks passed between publication of the study and its retraction.
“It is an overall positive that everything is online now,” said Ivan Oransky, editor in chief of Spectrum, distinguished writer in residence at New York University’s Carter Journalism Institute, and co-creator of RetractionWatch.com. “A lot of the systems that publishers use, which are more about making sure they can maintain a certain volume than about quality control, have vulnerabilities.”
Online publication allows people to share text, search for images, and immediately voice concerns. RetractionWatch’s database has recorded more than 29,000 retractions dating back to the 1700-s—about 200 of which are related to lung cancer. According to Mr. Oransky, though, not enough papers are being retracted.
Why Retract?
A 2012 study from the Proceedings of the National Academy of Sciences of the United States of America estimated that, of more than 2,000 biomedical and life-science research articles listed as retracted on May 3, 2012, about two-thirds were attributable to misconduct such as fraud or suspected fraud, duplicate publication, or plagiarism.4
Amid an era marked by frequent accusations that news reports are “fake,” Mr. Oransky clarified some of the types of fraud that might be seen in retracted papers. Fabricated data are entirely made up, whereas falsified data present misleading results that do not accurately reflect a study’s findings.
“Falsification is a lot of what we see,” Mr. Oransky said. “It is fake data in the sense that it isn’t really what happened. There may be a lot of truth in it but not enough truth to make a difference.”
There are scientists out there now who commit their time to trying to spot fraud in scientific papers.
Jennifer A. Byrne, PhD, of the School of Medical Sciences, Faculty of Medicine and Health, The University of Sydney, Australia, has recently started to look into publishing fraud in the form of paper mills.
In 2015, Dr. Byrne discovered several papers about a gene on which she and others had first reported in 1998, but these new papers had “shared and strange irregularities,” including “highly similar language and figures.”5
She came to believe that they had been produced at what she called paper mills, or third-party vendors working for profit to produce papers for doctors on demand.
Since then, Dr. Byrne has devoted more of her time to spotting questionable papers, often by looking for faulty genetic sequences, she said.
“It is pretty simple. We screen the sequences in a paper and ask the question, ‘Are they what the author said or not?’” Byrne said. “It is like if you are claiming to make a lemon pie, but you didn’t use any lemons and instead used pepperoni. You didn’t make a lemon pie. Either you got it wrong or you knew you were actually making a pizza.”
Dr. Byrne and colleagues recently published the results of a computer-aided analysis of more than 11,700 human genetics publications. The analysis identified 712 studies across 78 journals that described at least one wrongly identified nucleotide sequence. These 712 papers had received more than 17,000 citations, including some by human clinical trials.6
Dr. Byrne said that some background in genetics is necessary to do this type of “sleuthing” but that, nevertheless, there are likely hundreds of thousands of people around the world capable of doing this type of work.
Another example of a scientist sleuth is Elizabeth Bik, a microbiologist originally from the Netherlands, who devotes her time to spotting reused or manipulated images from scientific experiments such as western blots.7
Falsified genetic sequences, plagiarism, and manipulated photos are just a couple of examples of what Mr. Oransky called the “many flavors of misconduct.”
Why Do It?
The idea of “publish or perish” is a real concept, Mr. Oransky and Dr. Byrne agreed, and this pressure is likely leading to misconduct. Top medical journals have a limited number of publication slots, and it is no secret that negative studies or studies that duplicate or confirm another study’s findings do not get as much attention as novel findings.
“In order to get ahead, get tenure, or get promoted in academia you need to publish papers. Full stop,” Mr. Oransky said. “That is the incentive, and people respond to incentives.”
Mr. Oransky said that he still does not understand why anyone is surprised that fraud is happening. In fact, he said, he is quite disheartened over how much effort some people put into these schemes instead of focusing on the science.
Dr. Byrne agreed that everybody is under tremendous pressure to publish.
“In some countries or some sectors of the research community, that pressure to publish is not matched by people’s ability to do research, their training, or the availability of research facilities or resources,” Dr. Byrne said. “That is particularly true of medical doctors, where there is often an assumption that if you are a medical doctor you should be doing research. Not all medical doctors have the time or the training.”
Until the system of incentives is fixed, it is difficult to imagine an end to the problem of publishing misconduct, Mr. Oransky said.
What’s the Harm?
The publication of fraudulent papers results in a waste of time and money.
People who commit this fraud may not think about these potential consequences, Mr. Oransky said.
“They are literally wasting dollars, whether it is taxpayer dollars, dollars from private foundations or, frankly, corporate dollars,” he said.
One study8
attempting to estimate the financial costs of the increasing number of scientific article retractions found that papers retracted due to misconduct had accounted for about $58 million in direct funding from the National Institutes of Health between 1992 and 2012.
There is a human cost, as well. First, people are involved in some of the trials tied to misconduct. One estimate found that more than 28,000 patients were enrolled and almost 10,000 patients treated in 180 retracted primary studies from 2000 to 2010. If secondary studies—those that cited the fraudulent primary studies—are included, more than 400,000 patients were enrolled and more than 70,000 patients were treated.9
The second issue is the wasted time and effort of researchers.
“People read these papers, think the results look interesting, and try to duplicate the results,” Dr. Byrne said.
A researcher may take a study back to the lab and try unsuccessfully to duplicate the results. This may lead them to question their abilities or to repeat experiments.
“Real laboratory experiments take time and resources. People can waste 3 to 6 months trying to follow up on something that never actually happened in the first place,” Dr. Byrne said. “Those kinds of wasted efforts are unlikely to be reported or end up in the literature because researchers are unlikely to waste another 3 months of time writing it up.”
Additionally, researchers who follow up on fraudulent research are wasting time that could have been spent on more productive research.
“We also know that some of these papers are cited, and we are still trying to work out what that will mean,” Byrne said. “Those citations can add to the appearance of reliability. I worry that there are PhD students out there who might base their entire thesis on some of these papers. That is frightening.”
Finally, one has to wonder what these publications do for trust in science, Mr. Oransky said.
Is There a Fix?
The best way to cut down on misconduct in publishing may be to create a system with different incentives.
It is possible that more emphasis needs to be put on successfully reproducing results. A Nature survey found that “more than 70% of researchers have tried and failed to reproduce another scientist’s experiments and more than half have failed to reproduce their own experiments.”10
One paper examining incentives in medical research suggested that scientific works be audited for reproducibility by independent bodies, and then researchers ranked according to their ability to keep reproducibility high.11
In the meantime, Mr. Oransky said that there are some players in the publishing community who are acknowledging the problem and trying to correct it. For example, PLOS, an open-access publisher, has hired a team of people to respond to allegations of misconduct.
“They are still way behind, but at least they are working on it,” Mr. Oransky said.
In addition, the publication’s open-access model aids in spotting misconduct. Journals with paywalls, on the other hand, limit the ability to have these “sleuths” review published materials, he said.
Dr. Byrne acknowledged that more and more journals are attempting to screen for image duplication, plagiarism, and other factors that may hint at papers being obtained from paper mills. Prevention may not be enough, though, she said.
“I liken to it cancer. It is good to prevent cancer, but you also can’t ignore the patients that already have it,” she said. “We have to do something about the papers that are already published.”
Byrne emphasized the role that individuals can play in spotting fraudulent papers.
“If you are reading papers about gene function in cancer, just always read those papers with a critical mindset,” she said. “Also, be cautious about basing future research on papers that examine gene function in cancer at the moment.”
Be both cautious and critical, she advised.
“When people look at a published paper, people assume it is OK. It may not be,” Dr. Byrne said. “You can’t always accept literature at face value.”
- 1. Cai X, Fry CV, Wagner CS. International collaboration during the COVID-19 crisis: autumn 2020 developments. Scientometrics. 2021;126:3683-3692.
- 2. Christakis D, Fontanarosa PB. Notice of retraction. Walach H, et al. Experimental assessment of carbon dioxide content in inhaled air with or without face masks in healthy children: a randomized clinical trial. JAMA Pediatr. 2021. [Epub ahead of print].
- 3. Walach H, Weikl R, Prentice J, et al. Experimental assessment of carbon dioxide content in inhaled air with or without face masks in healthy children: a randomized clinical trial. JAMA Pediatr. 2021. [Epub ahead of print].
- 4. Fang FC, Steen RG, Casadevall A. Misconduct accounts for the majority of retracted scientific publications. PNAS. 2012;109(42):17028-17033.
- 5. Byrne J. We need to talk about systemic fraud. Nature. 2019;566(7742):9. https://www.nature.com/articles/d41586-019-00439-9
- 6. Park Y, West RA, Pathmendra P, et al. Human gene function publications that describe wrongly identified nucleotide sequence reagents are unacceptably frequent within the genetics literature. bioRxiv. 2021;doi:10.1101/2021.07.29.453321.
- 7. Shen H. Meet this super-spotter of duplicated images in science papers. 2020;581(7807):132-136. Accessed July 16, 2021. https://www.nature.com/articles/d41586-020-01363-z
- 8. Stern AM, Casadevalli A, Steen RG, Fang FC. Financial costs and personal consequences of research misconduct resulting in retracted publications. eLife. 2014;3:e02956.
- 9. Steen RG. Retractions in the medical literature: how many patients are put at risk by flawed research? J Med Ethics. 2011;37:688-692.
- 10. Baker M. 1,500 scientists lift the lid on reproducibility. Nature. 216;533:452-454.
- 11. Grimes DR, Bauch CT, Ioannidis JPA. Modeling science trustworthiness under publish or perish pressure. R Soc Open Sci. 2018;5:171511.
