Trials published fail to adequately report risk estimates

Hi! We’re Izzy, Furqaan and Shanil, a group of 6th-year medical students. As part of our course after finals we get the chance to pick special study modules (SSMs) based on our interests. All of us have had some exposure to evidence-based medicine (EBM) in the past. This has ranged from formal teaching sessions – for instance, the EBM seminars in our 4th year – to more informal discussions with doctors on the wards about new guidelines and trials informing clinical practice. The three-week long SSM provided a perfect opportunity to deepen our understanding and exposure to EBM whilst also being given the chance to perform our own study. This will no doubt be helpful as some of us are considering the possibility of combined clinical and academic careers in the future, and EBM is important to all clinicians to help better inform our decision making.

What did we do?

During our first SSM meeting, we learnt about the standards to which clinical trials are held, including the role of the CONSORT statement which standardises the reporting of results in randomised controlled trials (RCTs). This led us to consider the extent to which trial authors and journals adhere to these guidelines. Our study specifically focused on item 17b of the CONSORT statement:

‘17b – For binary outcomes, presentation of both absolute and relative effect sizes is recommended’

CONSORT offers the following explanation to why this is important:

‘When the primary outcome is binary, both the relative effect (risk ratio, relative risk, or odds ratio) and the absolute effect (risk difference) should be reported (with confidence intervals), as neither the relative measure nor the absolute measure alone gives a complete picture of the effect and its implications.’

This is an important recommendation to investigate since it is easy for doctors to misinterpret treatment effects if only one risk estimate is presented. For example, both doctors and lay people tend to overestimate the effect when it is presented in terms of relative risk. This may be the case, for instance, if the relative risk of an event is high but the absolute effect size is, in fact, quite small. Thus, it is imperative that trials report both risk estimates explicitly to allow readers to correctly interpret and use the results of clinical trials.

To identify relevant trials, we searched for published reports of RCTs with binary primary outcomes in five major medical journals: The Lancet, British Medical Journal (BMJ), Journal of the American Medical Association (JAMA), Annals of Internal Medicine and the New England Journal of Medicine (NEJM) between January and May 2019. Each RCT was then assessed for its adherence to item 17b of the CONSORT statement. Correct adherence was considered when both relative and absolute effect sizes (including their confidence intervals) were presented in the results section of the paper. Screening of trials and data collection was performed independently and then checked by at least one other student with discrepancies resolved by a senior academic.

What did we find?

Our search returned 267 potentially eligible articles. 147 were identified as RCTs and, of these, 93 had a binary primary outcome. Our search is summarised in figure 1.

The number of RCTs with binary outcomes meeting item 17b of the CONSORT reporting guidelines is presented in Table 1. Overall, only 24% of included articles were found to adhere to item 17b of the CONSORT statement.

Table 1: Summary of findings per journal

Implications and discussion

Overall, adherence to item 17b of the CONSORT guideline was poor. Of the 5 journals studied, 2 (BMJ and JAMA) required authors to submit completed checklists that explicitly showed that they met each item of the CONSORT statement. Unsurprisingly, articles within these 2 journals showed greater adherence to the guidelines. However, adherence was still not 100% across these journals: articles published in the BMJ showed 100% adherence whilst those in JAMA only did so 45% of the time. Moreover, the 100% adherence demonstrated by the BMJ is tempered by the small number of trials with binary outcomes published in this journal during our study period.

Our findings suggest that, despite their wide endorsement, many leading medical journals do not currently have effective methods of ensuring that the CONSORT guidelines for reporting binary outcomes in RCTs are adhered to. This has important implications for the interpretation of the research they publish, including fostering the misinterpretation of trial results by health professionals, patients and policymakers.

Figure 2: Number of RCTs published in the top 5 medical journals between

January and May 2019 meeting item 17b of the CONSORT statement.

What did we learn and where next?

This study was a valuable learning opportunity in many ways. Firstly, it has provided us with tools on how to organise a study of our own, and given us a better understanding of the principles underlying the reporting of RCTs. Furthermore it has highlighted to us that journals do not always adhere to standardization protocols, making us more cautious about the way we interpret evidence, and affirming the importance of critical appraisal. Thus, we feel better equipped to appraise evidence as clinicians after graduation and more driven to take part in academic work during our careers.

We’re aiming to finish up our project by obtaining the CONSORT checklists for papers in journals that require them, and then reconcile these with our findings. This may shed light on potential reasons why the guidelines were not adhered to. We hope to eventually write and present these results up more formally for submission to a journal.

Izzy, Furqaan and Shanil