How did NEJM respond when we tried to correct 20 misreported trials?
This blog was written by Ben Goldacre, Henry Drysdale, Ioan Milosevic, Aaron Dale, Philip Hartley, Eirion Slade, Anna Powell-Smith, Kamal Mahtani and Carl Heneghan, on behalf of the COMPare Trials Team. The original blog post can be found here.
It is well established that academic journals routinely permit outcome switching in the trial reports that they publish, despite public commitments to address this problem. For six weeks from October 2015, the COMPare team analysed every RCT published in the top 5 medical journals to check if they had correctly reported their pre-specified outcomes. Where we found discrepancies between the outcomes that were pre-specified and those reported, we wrote a letter to that journal to correct the record. The responses we have received from journals have been extremely varied: from full public engagement and transparent corrections in the BMJ, to inaccurate and concerning responses in the Annals of Internal Medicine.
New England Journal of Medicine (NEJM) is the highest impact factor journal in medicine. So far we have analysed 23 of their trials. To NEJM’s credit, of 32 primary outcomes pre-specified across these papers, 31 were correctly reported. However, 16 publications added in a total of 65 additional outcomes that were not pre-specified in the protocol or trial registry, and only 2 of these were declared as novel. Furthermore, a total of 160 pre-specified secondary outcomes spread across these publications were either not reported, or were reported incorrectly. Our full results and raw data are available here. Only 3 trials were perfectly reported.
We therefore sent 20 letters to NEJM to correct the record on the individual trials that misreported their pre-specified outcomes in the pages of NEJM. None of these letters have been published. At COMPare, we set out to establish why outcome switching persists in academic journals, despite public commitments to best practice. The responses we have received from journals in the course of the project shed important light on the answers to this question. In their correspondence with us NEJM dismiss concerns on outcome reporting and openly state that they do not regard it as problematic. In accordance with our commitment to open science, because we are citing and analysing journal correspondence in our forthcoming research paper, and in order to drive forward discussions on how to fix the problem of outcome switching, we are sharing our correspondence with the editors of NEJM below.
From COMPare to NEJM, 4/11/2015
Dear Editor,
Your recent publication “A Phase 3 Randomized Trial of Nicotinamide for Skin-Cancer Chemoprevention” [1] reports outcomes that are different to those initially registered [2].
There were ten pre-specified secondary outcomes, of which four were reported in the paper; while six are not reported anywhere in the publication.
The New England Journal of Medicine has endorsed the CONSORT guidelines on best practice in trial reporting, which aim to reduce the risk of selective outcome reporting [3]. Our team is currently tracking all trials in five top journals to monitor and signpost incorrect outcome reporting, as part of the COMPare project [4].
Attached is a corrected results table (Table 1)*, giving the results for the pre-specified outcomes, or the words “data not reported”.* We hope you will publish this letter to ensure those using the results of this trial are aware of the discrepancies.
Yours faithfully,
Henry Drysdale, Aaron Dale and Ben Goldacre on behalf of the COMPare project team.
Centre for Evidence Based Medicine,
Nuffield Department of Primary Care Health Sciences,
University of Oxford.
[1] Chen A et al, A Phase 3 Randomized Trial of Nicotinamide for Skin-Cancer Chemoprevention, N Engl J Med 2015;373: 1618-26.
[2] Trial registry entry: https://www.anzctr.org.au/Trial/Registration/TrialReview.aspx?ACTRN=12612000625875
[3] Moher D et al, CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials, BMJ 2010; 340:c869.
[4] COMPare project website www.COMPare-Trials.org
*for clarity we have not reproduced that table here
Below is the email we received from Deputy Editor Prof Dan Longo, in response to that letter (the first of 20 that we sent to NEJM). To set this correspondence in the broader context of ongoing discussions around open science and reproducibility, Prof Dan Longo is lead author of the widely discussed and controversial recent NEJM editorial on “research parasites”.
From NEJM to COMPare, 17/11/2015
Dear Mr. Drysdale,
I am sorry that we will not be able to print your recent letter to the editor regarding the Damian article of 22-Oct-2015. The space available for correspondence is very limited, and we must use our judgment to present a representative selection of the material received. Many worthwhile communications must be declined for lack of space.
Furthermore, I gather that you and your COMPare colleagues intend to make this audit a regular activity. The New England Journal of Medicine finds some aspects of CONSORT useful but we do not, and never have, required authors to comply with CONSORT. Rather we view each piece individually and add the data as appropriate based on the judgment of the peer reviewers, the statistical reviewers, and the editors.
Thank you for your interest in the Journal.
Sincerely,
Dan L. Longo, M.D.
Deputy Editor, New England Journal of Medicine
Professor of Medicine, Harvard Medical School
To NEJM from COMPare, 19/11/2015
Dear Professor Longo,
Thank you for your email, and for the clarification. The aim of the COMPare project is not to assess compliance with CONSORT, but rather to assess prevalence of outcome switching in clinical trials. Specifically, we aim to detect outcomes that are pre-specified but not reported, as well as outcomes that are novel but not declared as such. We are using the CONSORT guidelines as we believe that the methods of outcome reporting they represent are important and widely accepted.
We strongly believe that outcome switching in trials reduces the quality of evidence used to make clinical decisions. In carrying out this project we hope to reduce the prevalence of the problem and to ensure that, when a trial’s outcomes have been switched, those reading the trial report are aware of this fact.
Best wishes,
Henry Drysdale, Ben Goldacre and Carl Heneghan
Centre for Evidence Based Medicine,
Nuffield Department of Primary Care Health Sciences,
University of Oxford.
From NEJM to COMPare, 21/11/2015
Dear Mr. Drysdale,
Thank you for your follow-up letter. Space constraints for articles published in the Journal do not allow for all secondary and other outcomes to be reported. Any interested reader can compare the published article, the trial registration and the protocol (which was published with the article) with the reported results to view discrepancies; thus your letter does not add substantively to the published record.
Dan L. Longo, M.D.
Deputy Editor, New England Journal of Medicine
Professor of Medicine, Harvard Medical School
From COMPare to NEJM, 25/11/2015
Dear Professor Longo,
Thank you for your reply.
You mention space limitations as a justification for selective outcome reporting, and non-reporting of pre-specified outcomes; however our monitoring, and our letters, have regularly identified trials where additional non-prespecified outcomes are reported (and not declared as novel), including several trials published recently in the New England Journal of Medicine [1,2,3].
You also mention that the data needed to detect misreported outcomes are available in the public domain. We agree: indeed this is how we are conducting our work. However, readers reasonably assume that trials published in high-impact journals have been correctly reported. Furthermore, from our extensive experience it takes between 30 minutes and 5 hours to obtain the relevant documents, cross-check the pre-specified and reported outcomes, and assess whether or not these have been reported correctly. Misreported outcomes are therefore highly likely to go undetected, and the evidence used to make clinical decisions remains distorted.
Selective reporting of outcomes is widely regarded as misleading. It undermines the validity of findings, particularly when driven by statistical significance or the direction of the effect [4], and has memorably been described in the New England Journal of Medicine as “Data Torturing” [5]. When pre-specified outcomes are left unreported, the trial report is more likely to overestimate treatment benefits, and more likely to exaggerate the significance of the results that are reported [6]. Even where outcomes are left unreported arbitrarily, regardless of strength or direction of finding, permissiveness around poor reporting gives cover to those who engage in selective reporting of more positive outcomes to exaggerate findings.
Can we please ask you to reconsider publishing our letters, in order to let us correct the record on individual trials; and also to reconsider your policy on permitting trial reports to switch outcomes between the protocol/registration and publication?
Best wishes,
Henry Drysdale, Carl Heneghan and Ben Goldacre, on behalf of the COMPare team.
Centre for Evidence Based Medicine,
Nuffield Department of Primary Care Health Sciences,
University of Oxford.
[1] Chen A et al, A Phase 3 Randomized Trial of Nicotinamide for Skin-Cancer Chemoprevention, N Engl J Med 2015;373: 1618-26.
[2] Kaul U et al, Paclitaxel-Eluting versus Everolimus-Eluting Coronary Stents in Diabetes, N Engl J Med 2015;373:1709-19.
[3] Ellis, S. G. et al, Everolimus-eluting bioresorbable scaffolds for coronary artery disease, N Engl J Med 2015;373:1905-15.
[4] Dwan K et al, Evidence for the selective reporting of analyses and discrepancies in clinical trials: a systematic review of cohort studies of clinical trials. Jun 24; 11(6):e1001666. doi: 10.1371/journal.pmed.1001666
[5] Mills JL, Data torturing, N Engl J Med.1993;329: 1196-1199.
[6] Chan A et al, Empirical Evidence for Selective Reporting of Outcomes in Randomized Trials: Comparison of Protocols to Published Articles, JAMA. 2004;291(20):2457-2465.
We received no further reply to this email. However, we have received the following replies from NEJM in reply to our subsequent letters correcting misreporting of pre-specified outcomes. These replies from NEJM are all identical.
From NEJM, 1/12/2015
Dear Mr. Hartley,
Thank you for your letter. The publication you cite is the summary report of a complex study. In it, the study findings are described with fidelity to the study protocol, which is posted at NEJM.org as a supplement to the publication. We consider the publication to be consistent with the scientific and clinical intent of the trial as described in the protocol; we have not identified any clinically meaningful discrepancies between the protocol and the published paper that are of sufficient concern to require correction of the record.
We appreciate your interest in the New England Journal of Medicine.
Sincerely,
Dan L. Longo, M.D.
Deputy Editor, New England Journal of Medicine
Professor of Medicine, Harvard Medical School
From NEJM, 5/1/2016
Dear Mr. Slade,
Thank you for your letter. The publication you cite is the summary report of a complex study. In it, the study findings are described with fidelity to the study protocol, which is posted at NEJM.org as a supplement to the publication. We consider the publication to be consistent with the scientific and clinical intent of the trial as described in the protocol; we have not identified any clinically meaningful discrepancies between the protocol and the published paper that are of sufficient concern to require correction of the record.
We appreciate your interest in the New England Journal of Medicine.
Sincerely,
John A. Jarcho, M.D.
Deputy Editor, New England Journal of Medicine
From NEJM, 28/1/2016
Dear Mr. Slade,
Thank you for your letter. The publication you cite is the summary report of a complex study. In it, the study findings are described with fidelity to the study protocol, which is posted at NEJM.org as a supplement to the publication. We consider the publication to be consistent with the scientific and clinical intent of the trial as described in the protocol; we have not identified any clinically meaningful discrepancies between the protocol and the published paper that are of sufficient concern to require correction of the record.
We appreciate your interest in the New England Journal of Medicine.
Sincerely,
Julie R. Ingelfinger, M.D.
Deputy Editor, The New England Journal of Medicine
Professor of Pediatrics, Harvard Medical School
From NEJM, 29/1/2016
Dear Mr. Milosevic,
Thank you for your letter. The publication you cite is the summary report of a complex study. In it, the study findings are described with fidelity to the study protocol, which is posted at NEJM.org as a supplement to the publication. We consider the publication to be consistent with the scientific and clinical intent of the trial as described in the protocol; we have not identified any clinically meaningful discrepancies between the protocol and the published paper that are of sufficient concern to require correction of the record.
We appreciate your interest in the New England Journal of Medicine.
Sincerely,
John A. Jarcho, M.D.
Deputy Editor, New England Journal of Medicine
We think the correspondence above speaks for itself, but there are some concerning themes in here that warrant signposting, and discussion.
“We do not, and never have, required authors to comply with CONSORT”.
While it is true that NEJM does not state that it requires compliance with CONSORT guidance, the journal is listed as endorsing the CONSORT statements on clinical trial reporting. This is a matter that we will be taking up with CONSORT, along with various other changes and concerns that have arisen during the course of COMPare.
“We view each piece individually and add the data as appropriate based on the judgment of the peer reviewers, the statistical reviewers, and the editors.”
This seems to state that NEJM is not only unconcerned by outcome switching, but that they actively request it from trialists, according to their own judgement. This seems contrary to good trial reporting practice, and is a clear source of potential bias which the review process is supposed to eliminate. We would be interested to know whether all readers of NEJM are aware that this form of covert outcome switching is permitted in NEJM, and whether they regard it as problematic that this statistical and interpretative context for reported effects is not clearly signposted in the journal.
“Any interested reader can compare the published article, the trial registration and the protocol..”
This is true, but as discussed above, this is an impractical protective factor. We have found the process of checking is long and laborious, and it often takes several hours to identify the outcomes that were specified before trial commencement and then compare those against the outcomes reported. We believe there is an expectation amongst the readers of journals that this kind of checking is conducted as part of the peer review process.
“Space constraints for articles published in the Journal do not allow for all secondary and other outcomes to be reported.”
We agree that space constraints may be a potential problem with print journals. However, at COMPare we have repeatedly found large numbers of additional non-prespecified outcomes reported in journal articles, including repeatedly in NEJM. Furthermore, there are no space constraints to reporting pre-specified outcomes online versions or appendices, and we have coded this as correctly reported where it has occurred (although we think it would be more informative for the reader if journals signposted these from the main body of the text).
In Conclusion
The New England Journal of Medicine is one of the most influential and widely read medical journals, with an impact factor of 55.8. Outcome switching is an important source of bias. It seems NEJM are unwilling to have the record corrected on individual trials when outcome switching is identified, and do not regard outcome switching as problematic (in fact, they may even actively solicit it). Most importantly, readers may not be aware of these issues, and therefore will take reported outcomes at face value.
As previously discussed, we don’t see a problem with trialists switching their outcomes where there are good reasons to do so, as long as this is openly declared in the trial report, as per CONSORT guidelines. Furthermore, we don’t think every trialist switching their outcomes is setting out to deliberately mislead readers, or exaggerate their findings. But a culture of permissiveness around outcome switching leads to lower standards, and it gives cover to those who are setting out to mislead. That’s why we have clear guidelines and reporting standards in academic medicine.
As ever we would welcome any suggestions or views on the best strategies to address this problem.