Salt reduction in heart failure: where has the uncertainty come from?

This makes for a great teaching session…. demonstrating the importance of proper application of critical appraisal skills in discerning what the evidence does or does not tell us.

“You might want to look at this”. That’s how it all started. A simple email I sent back in November 2012. The ‘this’ was an update from Evidence Alerts featuring a systematic review entitled ‘Low sodium versus normal sodium diets in systolic heart failure: Systematic review and meta-analysis’ by DiNicolantonio et al., published in the Heart [BMJ] journal in August 2012.

‘You’ was my colleague Dr Kamal Mahtani, deputy director of the Centre for Evidence-based Medicine. We both share an interest in lifestyle medicine and its role in chronic disease management and prevention – Kamal through his practice as a GP, me through my background in exercise medicine. At the time we were working on another study assessing the impact of reduced salt in bread on taste and palatability. So salt was on our radar. The 2012 paper particularly caught our eye for its conclusion: ‘Compared with a normal sodium diet, a low sodium diet significantly increases morbidity and mortality in systolic HF [heart failure].’

Your fast brain may have read this as a normal salt diet (in effect a high salt diet, as the majority of people eat above the recommended 6 g daily intake) increased risk of worse outcomes for people with heart failure. But the conclusion is the opposite – a low salt diet increased the risk of worse outcomes. The numbers also stood out. For example, the pooled relative risk (RR) for overall mortality was 1.95 (1.66 to 2.29) – a 95% relative increase in the risk of earlier death in the group on a low salt diet.

The comments on Evidence Alert were equally interesting. ‘This changes my practice’ stated one family practitioner. There was more caution from an internal medic: ‘This systematic review, since totally opposite to expected results and based on RCTs, needs very careful critique from a good commentator’.

So that’s what we did.

Little did we know just how impactful our careful critique would turn out to be. It also makes for a great critical appraisal teaching session.

The review does well in the appraisal process early on. The PICO (patient, intervention, comparator and outcomes) is clear and given early. Searches are comprehensive enough.  It’s when we get to the criteria for inclusion and their application that cracks start to appear. Only trials of dietary salt interventions could be included. OK. Trials had to have morbidity or mortality as an outcome to be included. Again, fine. And trials needed a sample size greater than 90. No rationale was given for this sample size cut-off.

Six studies were considered eligible for inclusion. On close inspection, it turns out all six trials were from the same research group, all of which were authored by the second author on the Heart BMJ systematic review (Pietro Di Pasquale). That seemed odd.

We also found the following:

• Another three authors from the same group were present on all the included trials (Paterna, Parrinelo, Licata)
• One of the trials was registered as an observational study on clinicaltrials.gov
• All the included trials favoured a normal salt diet for all reported outcomes
• The smallest included trial had a sample size of 94 (the sample size cut-off for inclusion as set by the authors was 90)
• One excluded study from a different group, and showing a favourable effect for a low salt diet, had a sample size of 65
• Another study that was excluded for not reporting morbidity/mortality data actually did report mortality data and showed a favourable effect for a low salt diet (4 deaths in the low salt group, 12 deaths in the normal salt group)
• One included study used a salt tablet as the intervention
• Three trials had serious confounding were by the participants in the normal dietary salt group also received an IV dose of hypertonic saline solution (HSS). The review authors did perform a subgroup analysis excluding these three trials and found similar results for all outcomes.

BUT!

And it’s a big but.

When looking at data for secondary outcomes in two of the included studies we made an interesting observation. The studies were Paterna 2008 published in Clinical Science and Parrinello 2009 published in the Journal of Cardiac Failure. The former enrolled 232 participants to a 6-month trial between Jan 2000 to May 2005, the later 173 participants to a 12-month trial between Sept 2005 and Aug 2007. It took some work as the results were laid out and formatted differently between the two studies, but we observed that both studies reported the same numerical data for 24 secondary outcome variables (see Figure).

Figure: Data for secondary outcome variables reported in Paterna 2008 and Parrinello 2009.

We submitted the above concerns in a letter to the then editor of Heart BMJ journal on 30 November 2012. The Editor did not publish our letter but did forward our concerns to the Editor in Chief at the BMJ and these were subsequently forwarded to the journal’s Publishing Ethics Committee. A request was made to the corresponding authors of the two papers for access to the raw data to substantiate their reliability. The authors’ response to the Committee was ‘the raw data were no longer available having been lost as a result of a computer failure’. The Committee had no choice but to recommend that the Heart systematic review be retracted.

The story doesn’t end there. One of the co-authors of the BMJ Heart review was the lead author of a 2011 Cochrane review on reduced dietary salt in cardiovascular disease. One of the studies with duplicate data (Paterna 2008) was the only included evidence for reduced dietary salt in people with heart failure. Interestingly, the Cochrane review excluded the study that had a salt tablet intervention on the grounds it was ‘not a dietary intervention’. A subsequent 2013 update of the Cochrane review highlighted the ‘doubts raised about the integrity of research from the Paterna group’ and the previous 2011 review was withdrawn. The final update in 2014 excluded studies in patients with heart failure, meaning that there was now no review on the effect of reduced dietary salt in people with heart failure in the Cochrane database.

We happily filled that gap. As of 5 November 2018, we can now finally tell the end of the story. Well, the end of the beginning at least.

Systematic reviews and the results they produce can and do inform and change clinical practice. Taking the evidence from the Heart BMJ review at face value, recommending a low sodium diet has potentially been harming patients, denying them a protective intervention in the form of a high sodium diet. It appeared at least some would change their practice as a result. Our appraisal found these findings to be based on unreliable data.

All this makes for a great teaching session. We have been using it on the core module of the MSc in Evidence-Based Health Care that I lead. It always generates a heated discussion and demonstrates the importance of proper application of critical appraisal skills in discerning what the evidence does or does not tell us.

David Nunan is a Senior Researcher and Director of Postgraduate Certificate in Teaching Evidence-Based Health Care. He is based at the Centre for Evidence-Based Medicine, Nuffield Department of Primary Care Health Sciences, University of Oxford.

You can follow him on Twitter @dnunan79

Acknowledgements: Kamal R Mahtani for helpful comments.

Competing interests: DN has received funding from the NHS NIHR SPCR programme and the Royal College of General Practitioners.

Disclaimer: The views expressed in this article represent the views of the author and not necessarily those of the host institution, the NHS, the NIHR, or the Department of Health.