Everyone is familiar with the “corrections” columns in newspapers and the errata pages in the backs of books. But those corrigenda are a far cry from identifying the problems created when authors deliberately offer for publication fraudulent results. The 2009 National Academy of Sciences (NAS) report, Strengthening Forensic Science in the United States: A Path Forward, has highlighted the importance of peer review publication as a basis for forensic science.1 And yet, this process alone cannot guarantee absolute reliability.2 Research misconduct and the publication of fraudulent results has become a growing concern.3 Addressing the causes of misconduct,4 spotting faked data, and repairing the damage to the information stream are struggles faced by the scientific and publishing communities.5
Legal information professionals are well-versed in the intricacies of citation analysis—both its virtues and limitations.6 Yet, the realm of scientific publishing does not have a comparably comprehensive tool. The revelations about articles published with fake data or fraudulent results have lead to inconsistent retractions among periodicals and poor linkage between those warnings and the reader’s access points.
In the scientific publishing lexicon there are three levels of caution, resembling Shepard’s signals: “Retraction: What a journal issues when an investigation has shown that an article contains faked data or has been plagiarized. It tells the reader to ignore that article. Expression of Concern: What a journal issues when the editor is concerned that an article contains faked data or has been plagiarized but an investigation has either not begun or has begun but has not reaches a conclusion about that article. Correction: What a journal issues to correct a mistake by substituting correct information or by asking the reader to disregard specified parts of an article (for example, a reference to a retracted article).”7
Warnings and withdrawals can be triggered by a spectrum of reasons stemming from honest mistakes to full blown misrepresentation. Yet, there are no granulated cautions that distinguish the reasons for retraction. The important consideration is that warnings come too late. The disavowed article can be cited hundreds, perhaps thousands of times, before a retraction is posted. Although the source of an error or fraud can be unearthed, the article’s influence will have already radiated throughout the information stream.
One approach might be found in emerging information management tools, e.g., “a single search box that can search a library’s online and physical content including articles, books, journals, newspaper articles, e-books, specialist collections and more.”8 Thus warnings about retracted or withdrawn scholarship could be found and confirmed through multiple avenues and federated searches conducted at one time, so long as there is adequate linkage.9 Frankly, the problem is so complex that even information experts might find it a daunting task.10
This article collects news items and scholarly research concerning quality assurance in published scientific literature as well as specific current awareness tools.11
NEWS ARTICLES
Charges of Fake Research Hit New High, MSNBC, July 10, 2005
“Fabrications Exposed: Allegations of research misconduct reached record highs last year — the Department of Health and Human Services received 274 complaints, which was 50 percent higher than 2003 and the most since 1989 when the federal government established a program to deal with scientific misconduct. Chris Pascal, director of the federal Office of Research Integrity, said its 28 staffers and $7 million annual budget haven’t kept pace with the allegations. The result: Only 23 cases were closed last year. Of those, eight individuals were found guilty of research misconduct. In the past 15 years, the office has confirmed about 185 cases of scientific misconduct. Research suggests this is but a small fraction of all the incidents of fabrication, falsification and plagiarism. In a survey published June 9 in the journal Nature, about 1.5 percent of 3,247 researchers who responded admitted to falsification or plagiarism. (One in three admitted to some type of professional misbehavior.)”
Mistakes in Scientific Studies Surge, Wall St. J., Aug. 10, 2011
“Since 2001, while the number of papers published in research journals has risen 44%, the number retracted has leapt more than 15-fold, data compiled for The Wall Street Journal by Thomson Reuters reveal. Just 22 retraction notices appeared in 2001, but 139 in 2006 and 339 last year. Through seven months of this year, there have been 210, according to Thomson Reuters Web of Science, an index of 11,600 peer-reviewed journals world-wide.”
Publish-or-Perish: Peer Review and the Corruption of Science, The Guardian, Sept. 5, 2011
“Peer review is the process that decides whether your work gets published in an academic journal. It doesn’t work very well any more, mainly as a result of the enormous number of papers that are being published (an estimated 1.3 million papers in 23,750 journals in 2006). There simply aren’t enough competent people to do the job. The overwhelming effect of the huge (and unpaid) effort that is put into reviewing papers is to maintain a status hierarchy of journals. Any paper, however bad, can now get published in a journal that claims to be peer-reviewed.”
PERIODICALS
“Brave New World” of Daubert: True Peer Review, Editorial Peer Review, and Scientific Validity, 70 N.Y.U.L. Rev. 100 (1995)
“This Note seeks to provide a full discussion of the place of true peer review and editorial peer review in the Daubert scientific validity inquiry. As background for this discussion, Part I discusses the Daubert decision as the culmination of a long-standing struggle in evidence law over the proper admissibility standard for proffers of scientific evidence. Part II examines the processes of true peer review and editorial peer review. Finally, Part III seeks to clarify judicial misconceptions about true peer review and editorial peer review and to provide preliminary instruction on how courts should use true peer review as an interpretative tool in the Daubert scientific validity inquiry.”
Distinguishing Published Scholarly Content with Crossmark, 24 Learned Publishing 87 (Apr. 2011)
“The landscape of scholarly communication is complicated by multiple versions of the same documents, available from different sources. What happens if changes need to be made to published documents when scholars tap so many sources? How can important changes be communicated to readers so that they do not rely on outdated or, worse, repudiated research? This article explores the issues and describes the CrossMark service as a possible solution.” See also Discussion on Retractions and Corrections Heats Up, CrossRef Blog, Apr. 14, 2011.
How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data, PLoS ONE 4(5) (2009)
“The frequency with which scientists fabricate and falsify data, or commit other forms of scientific misconduct is a matter of controversy. Many surveys have asked scientists directly whether they have committed or know of a colleague who committed research misconduct, but their results appeared difficult to compare and synthesize. This is the first meta-analysis of these surveys.”
Mentoring and Research Misconduct: An Analysis of Research Mentoring in Closed ORI Cases, 14 Sci. Eng. Ethics 323, 335 (2008)
“We are reporting on how involved the mentor was in promoting responsible research in cases of research misconduct. We reviewed the USPHS [United States Public Health Service] misconduct files of the Office of Research Integrity [ORI]. These files are created by Institutions who prosecute a case of possible research misconduct; ORI has oversight review of these investigations. We explored the role of the mentor in the cases of trainee research misconduct on three specific behaviors that we believe mentors should perform with their trainee: (1) review source data, (2) teach specific research standards and (3) minimize stressful work situations. We found that almost three quarters of the mentors had not reviewed the source data and two thirds had not set standards. These two behaviors are positively correlated. We did not see convincing evidence in the records that mentors were causing stress, but it was apparent in the convicted trainees’ confessions that over 50% experienced some kind of stress. Secondary data, while not created for this research purpose, allows us to look at concrete research behaviors that are otherwise not very researchable. We believe it is important for mentors and institutions to devote more attention to teaching mentors about the process of education and their responsibilities in educating the next generation of scientists. This becomes a critical issue for large research groups who need to determine who is in charge educating, supervising and assuring data integrity.”
Office of Research Integrity: Experience and Authorities, 35 Hofstra L. Rev. 795 (2006)
“The Office of Research Integrity (‘ORI’) is a statutory office created by Congress in 1993, for which authority to respond to allegations of ‘misconduct in science’ originally arose under an earlier federal regulation. Under the 1993 legislation, the key requirements for ORI are: responding to allegations of research misconduct, defining research misconduct, and overseeing inquiries and investigations of research misconduct initiated by research institutions that receive Public Health Service (‘PHS’) funds or apply for such funds. 3 In addition, the statute specifically requires ORI to establish administrative processes and standards for protecting whistleblowers who act in ‘good faith.'”
Peer Review and Publication: Lessons for Lawyers, 36 Stetson L. Rev. 789 (2007)
“The aim here is to understand how the peer-review process works, how good an indicator it is that scientific testimony is ‘reliable’ in the legally relevant sense, and how courts might best use this Daubert factor. So for most of what follows, the focus will be on peer review in the narrow sense – pre-publication peer review.”
Pulp Fiction: Reflections on Scientific Misconduct, 2004 Wis. L. Rev. 975, 1007 (2004)
“Part I of this Comment gives an overview of the occurrence and seriousness of the problem of scientific misconduct. Part II addresses the evolution of the relevant legislative responses. Parts III and IV examine the scope of available civil and criminal actions in misconduct proceedings. The apparently inexorable transformation of the research profession 38 resulting from the need for legal mechanisms in the practice of scientific research is discussed in Part V. Part VI concludes that the legal system needs to begin exercising greater control over the conduct or misconduct of science, since current laws appear ill suited and insufficient.”
Reporting of Article Retractions in Bibliographic Databases and Online Journals, 99 J. Med. Libr. Assoc. 164 (2011)
“Retraction of journal papers containing falsified data plays a central role in ‘correcting’ the scientific record. Of 395 articles retracted between 1982 and 2002, Nath et al. classified 62% as retracted as a result of mistake, 27% because of deliberate falsification or fabrication of data, and 11% without a clear reason for retraction. There is evidence that retracted papers continue to be cited without reference to the retraction: Budd et al. found that 235 retracted articles were cited 2,034 times after retraction notices. Examination of 299 of these citations showed that only 19 referred to the retraction. Similarly, Neale et al.’s analysis of 102 articles from PubMed showed that many of these papers were subsequently cited by other researchers who were unaware that the papers had been affected by scientific misconduct. Limitations in the processes for recording retractions have been noted elsewhere, and the responsibilities of authors, journal editors, and research institutions have been highlighted.”
Research Fraud: Methods for Dealing With an Issue That Negatively Impacts Society’s View of Science, 10 Colum. Sci. & Tech. L. Rev. 61 (2009)
“This article first presents a novel tiered structure based on a simple analysis of two basic components of any instance of scientific misconduct, to provide a method to discriminate between different levels of misconduct. Next, this article looks to a number of authorities and institutions that could educate current and future scientists about the nature and effects of fraud in scientific research. Finally, this article explores options for enforcing and punishing instances of misconduct, including both civil and criminal, particularly mindful of the devastating career implications resulting from any association with an investigation of misconduct. In this vein, First Amendment and due process issues are highlighted and examined. This article concludes that, given the devastating repercussions to even exonerated scientists, a criminal code that takes into account the particular concerns of the scientific community would best provide protection for scientists and promote a more ethical scientific community.”
Research Misconduct, Retraction, and Cleansing the Medical Literature: Lessons from the Poehlman Case, 144 Annals Internal Med. 609 (Apr. 18, 2006)
“Maintaining the integrity of the scientific literature requires governmental institutions that have the authority to investigate and punish guilty scientists and requires that research institutions investigate alleged fraud. It requires journal editors to issue a retraction when they learn that their journal has published a tainted article. It requires research institutions to accept their responsibility to investigate every article published by a scientist who has published even one fraudulent article. Finally, it requires authors to take pains to avoid citing retracted articles and to issue a correction when they inadvertently cite a retracted article.”
Scientific Misconduct in Research, 1 J. Health & Life Sci. L. 63 (2007)
“Allegations of research misconduct, especially in federally-supported research, raise the specter of contentious, difficult confrontations. It is the obligation of the institution at which the research is conducted to ensure that a fair, competent, and thorough review of each allegation occurs. Recently revised Public Health Service research misconduct regulations set forth clear guidance and detailed requirements for institutional policies, inquiries, and investigations. Counsel to research institutions that receive federal support for research must be familiar with these regulations. Failure to prepare for, or respond quickly and compliantly to, allegations of research misconduct may result in injustice to an accused researcher, make definitive determinations impossible, expose an institution to regulatory penalties, and damage the reputation of both researchers and the institution.”
Shepardizing Science: Is an Article Fact or Fiction?, N.Y. L.J., Sept. 27, 2011, at 5
“As far back as 1835, when the New York Sun reported that there was life on the moon,1 false claims have been presented as scientific fact. In modern times, the principal remedy to such misrepresentations is the injunction of ‘peer review and publication’ stressed in the National Academy of Sciences (NAS) report, ‘Strengthening Forensic Science in the United States: A Path Forward.’ However, due to an uptick of dubious scientific data appearing in scholarly journals, another level of validation is required—citation analysis.”
What Is the Future of Peer Review? Why Is There Fraud in Science? Is Plagiarism Out of Control? Why Do Scientists Do Bad Things? Is It All a Case of: “All That Is Necessary for the Triumph of Evil Is that Good Men Do Nothing?”, 3 Vasc. Health Risk Manag. 39 (2007)
“Peer review is an essential component of the process that is universally applied prior to the acceptance of a manuscript, grant or other scholarly work. Most of us willingly accept the responsibilities that come with being a reviewer but how comfortable are we with the process? Peer review is open to abuse but how should it be policed and can it be improved? A bad peer review process can inadvertently ruin an individual’s career, but are there penalties for policing a reviewer who deliberately sabotages a manuscript or grant? Science has received an increasingly tainted name because of recent high profile cases of alleged scientific misconduct. Once considered the results of work stress or a temporary mental health problem, scientific misconduct is increasingly being reported and proved to be a repeat offence. How should scientific misconduct be handled—is it a criminal offence and subject to national or international law? Similarly plagiarism is an ever-increasing concern whether at the level of the student or a university president. Are the existing laws tough enough? These issues, with appropriate examples, are dealt with in this review.”
When Evidence Isn’t: Trials, Drug Companies and the FDA, 15 J.L. & Pol’y 991 (2007)
“Science is not set up like a bank checking system, on the assumption that fraud will be attempted. We cannot have cops, each presumably with a PhD, in every lab. A quicker way of inhibiting free thought and experimentation could not be devised. So we are forced to rely on trust. The editor, also a scientist, is part of this web of trust. Indeed, when there’s an allegation of, say, misconduct on the part of one of our authors at a research university, we editors are singularly helpless. We don’t have the time, resources, forensic expertise, mandate or authority to investigate, adjudicate and punish. All we can do is refer the complaint back to the author’s institution for formal investigation – and trust that the institution will follow through. Thus, the whole system is built on trust. Trust depends on there being someone accountable. This is why Joshua Lederberg, the Nobel Laureate who was president of Rockefeller University, wrote, ‘Above all, the act of publication is an inscription under oath, a testimony.’ That is how I was taught science should operate and assumed it did until, in 1977, I became deputy editor of the NEJM. I soon learned from repeated, bitter personal experience that, when scientists had a great deal at stake, some were prepared, in the name of prestige, to take short cuts, falsify, fabricate, plagiarize, bamboozle, lie, cheat, and throw away their reputations simply to notch up more publications, advance their careers and, of course, make more money. ”
BIBLIOMETRIC GUIDES
“From the user perspective, CrossMark is designed to: 1. Allow the researcher to easily determine whether they are looking at a publisher-maintained copy of a scholarly document. 2. Allow the user to easily ascertain the current status of a scholarly document. 3. Access and use any extra, non-bibliographic metadata that the publisher deems important to the document in question. From the publisher perspective, CrossMark is designed to: 1. Highlight that the scholarly publisher is responsible for both the initial certification1 of a publication, as well as the ongoing stewardship of said certified publication. 2. Enable the publisher to advertise the otherwise invisible steps that they have taken to ensure the trustworthiness of their content.” See also CrossRef Blog.
Errata, Retractions, Partial Retractions, Corrected and Republished Articles, Duplicate Publications, Comments (including Author Replies), Updates, Patient Summaries, and Republished (Reprinted) Articles Policy for MEDLINE (National Library of Medicine)
“The National Library of Medicine(R) (NLM) has a long-standing tradition of providing access to information in the biomedical literature through quality programs and services. One of the ways NLM assists users is to add subsequent notices of and/or linkages between citations for errata, retractions, partial retractions, corrected and republished articles, duplicate publications, comments (including author replies), updated versions of articles, patient summaries and republished (reprinted) articles indexed and available in NLM’s online MEDLINE database. Users who search MEDLINE will be informed if they retrieve a citation for an article that has been corrected by an erratum notice, retracted or partially retracted, corrected and republished, been found to duplicate another article, generated a separately published commenting article, been updated by a subsequent article, if a summary for patients has been published, or has been republished (reprinted) in another journal.”
CURRENT AWARENESS
Handling Misconduct – Case Summaries (Office of Research Integrity (ORI))
“This section contains summaries of closed inquiries and investigations. Institutions are not required to report inquiries to ORI if an investigation is not warranted unless the allegation had been forwarded to the institution by ORI. Closed investigations that found misconduct, or that imposed administrative actions without a misconduct finding, are reported in the first section. The listing of summaries involving misconduct findings is limited to the current year and two previous years because administrative actions are typically imposed for three years. A list of all individuals currently under a PHS Administrative Action is available on the PHS Administrative Action Bulletin Board. ORI does not maintain a listing of all persons against whom a finding of research misconduct has been made. The second section contains summaries of all closed inquiries and investigations that did not result in findings of research misconduct since 1994. These summaries are sanitized to protect the privacy of the individuals involved.”
Reference Point (National Library of Medicine)
“ReferencePoint is a tool to increase awareness of NLM products and services available online and onsite for health sciences library staff (especially those providing reference services). It will also inform health sciences library staff about non-NLM health sciences resources. The blog will serve as a dialogue and learning exchange between NLM staff and staff at other libraries.”
Retraction Watch (eds. Adam Marcus & Ivan Oransky)
“Tracking retractions as a window into the scientific process.”
Scholarly Kitchen (Society for Scholarly Publishing)
“The mission of the Society for Scholarly Publishing is ‘[t]o advance scholarly publishing and communication, and the professional development of its members through education, collaboration, and networking.’ The Scholarly Kitchen is a moderated and independent blog aimed to help fulfill this mission by bringing together differing opinions, commentary, and ideas, and presenting them openly.”
########
1 See generally Ken Strutin, Strengthening Forensic Science: The Next Wave of Scholarship, LLRX.com, Nov. 23, 2009; Clifford Spiegelman et al., Putting the Science in Forensic Science, Amstat News, Aug. 1, 2011.
2 See Sokal Affair (Sokal Hoax) (Wikipedia)(“The Sokal affair, also known as the Sokal hoax, was a publishing hoax perpetrated by Alan Sokal, a physics professor at New York University. In 1996, Sokal submitted an article to Social Text, an academic journal of postmodern cultural studies. The submission was an experiment to test the publication’s intellectual rigor and, specifically, to learn if such a journal would ‘publish an article liberally salted with nonsense if it (a) sounded good and (b) flattered the editors’ ideological preconceptions.'” (footnotes omitted)); Sokal Social Text Affair Bibliography (Prof. Sokal’s Personal Homepage, NYU Dep’t of Physics).
3 See Gautam Naik, Mistakes in Scientific Studies Surge, Wall St. J., Aug. 10, 2011.
4 See David E. Wright et al., Mentoring and Research Misconduct: An Analysis of Research Mentoring in Closed ORI Cases, 14 Sci. Eng. Ethics 323, 335 (2008)(“[P]otential causes [of misconduct]: (a) sociopathology; (b) increasing pressure on researchers . . . to publish and secure grants; (c) arrogance . . .; (d) ignorance of research standards and ethical norms in research . . . .”).
5 See generally Scientific Misconduct (Wikipedia); Managing Allegations of Scientific Misconduct: A Guidance Document for Editors (ORI 2000)(“The purpose of this document is to provide guidance to editors and their staff on reporting suspect manuscripts, facilitate the investigation of misconduct allegations, improve the correction of the literature, and promote research integrity. ORI is committed to working with editors to address possible scientific misconduct detected in manuscripts and published articles.”).
6 The improvement of legal informatics is an ongoing process. Just recently, the Uniform Law Commission announced the approval of the Uniform Electronic Legal Material Act, which “requires that official electronic legal material be: Authenticated, by providing a method to determine that it is unaltered; Preserved, either in electronic or print form; and Accessible, for use by the public on a permanent basis.” Uniform Electronic Legal Material Act Approved, ULC Press Release, July 12, 2011. This new Act appears to address concerns similar to those raised about information published in scientific journals, i.e., “Is the legal material official, authentic, government data that has not been altered?” Multiple formats, cached files and online archives present problems for any systemic quality assurance of published research.
7 Harold C. Sox and Drummond Rennie, Research Misconduct, Retraction, and Cleansing the Medical Literature: Lessons from the Poehlman Case, 144 Annals Internal Med. 609 (Apr. 18, 2006).
8 Chris Keene, Discovery Services: Next Generation of Searching Scholarly Information, Serials, 24(2), July 2011, at 193.
9 In a comprehensive report issued by The Science and Technology Committee of the British Parliament, they made note of a Shepard’s type tool, described by Robert Campbell of Wiley-Blackwell publishing, which ought to make headway on data quality assurance: “The [publishing] industry is developing […] a new project called CrossMark. Every paper that has gone through the peer review process has the ongoing stewardship of the publisher picking up on retractions or corrections. By clicking on to the CrossMark logo, you can go to the metadata and find out if there have been any updates or even retractions. That is a technical solution which is being launched this year.” Science and Technology Committee – Eighth Report: Peer Review in Scientific Publications par. 251 (House of Commons Rep. Jul 18, 2011).
10 Kath Wright & Catriona McDaid, Reporting of Article Retractions in Bibliographic Databases and Online Journals, 99 J. Med. Libr. Assoc. 164 (2011)( “Although retraction of papers appears to be increasing, it is not a common occurrence, so even an experienced librarian or information specialist may not be familiar with how bibliographic databases deal with annotating records to indicate retraction.”).
11 The added problem of developing safeguards for the citation analysis of grey literature is not considered in this article.