What does it mean to be ‘living’? In the case of systematic reviews, it depends on who you ask.
DPhil in Evidence-Based Health Care student Melanie Golob investigates definitions of living systematic reviews and whether Cochrane Reviews are the gold standard.
About the author: Melanie Golob is a DPhil in Evidence-Based Health Care student. Watch Melanie’s recent video to learn more about her experiences on the DPhil. She manages a state-level health technology assessment program in the US. Melanie has a master’s degree in biochemistry and molecular biology from the University of California, Los Angeles (UCLA).
As I’m sure we’re all aware, we recently overcame a global pandemic. At the time, the science around what to do and what not to do was changing daily, sometimes even by the minute. Therefore, methods of synthesising the evidence had to keep pace with the constantly evolving science in order to inform healthcare providers, policymakers, and the general public. A little known method (for all but a select few) called ‘living’ systematic reviews (LSRs) rose to prominence and quickly became viewed as the best and only way to keep up with the flood of studies being published during the Covid pandemic.
LSRs are a type of evidence synthesis meant to be continually updated, responding to changes in the primary literature as it is published. But there’s a problem: no one really agrees on what ‘living’ really means. Does an LSR need to be updated immediately following publication of new evidence? What exactly is the threshold for ‘living’ as compared to simply ’updated’? Adding new evidence too quickly can be problematic if data is proven false (ahem, Ivermectin), leading to mistrust in the review. On the other hand, waiting too long to add evidence risks an outdated review, which also results in mistrust. Concerningly, both of these scenarios can cast doubt on science in general.
The only published guidance for conducting LSRs comes from Cochrane, a world leader in systematic review methods. But the question remains as to whether or not that guidance is being followed.
So I checked, conducting an overview and audit of all published Cochrane LSRs (23 as of June 2022) and compared them to non-Cochrane LSRs. Cochrane’s living standards published in 2019 include both the why and the how. It covers why one might conduct an LSR as opposed to a traditional or static systematic review, and how best to perform an LSR. To summarise Cochrane’s LSR standards, it includes three main ‘why’ criteria to for those interested in making a systematic review ‘living’:
-
The question is a priority for decision-making,
-
The evidence is uncertain, and
-
New evidence is likely to be published that would impact conclusions.
Beyond that, five ‘how; guidance items for best practices in conducting LSRs (all pre-defined) are as follows:
-
Planned frequency of searching for new studies,
-
Plans for how often the review is updated with new evidence,
-
An ongoing process of actively monitoring and reviewing new evidence
-
Criteria for updating those plans, and
-
Criteria for stopping living mode.
I compared the 23 existing Cochrane LSRs with the same number of LSRs published outside of Cochrane. With the help of Andrea Leinberger-Jabari and Brenna Loufek, I examined if LSRs explicitly reported the three ‘why’ criteria and the five ‘how’ guidance items defined by Cochrane. This included synonyms of the defined standards, but no value judgements were made. I am also grateful to my supervisors, Jonathan Livingstone-Banks and David Nunan, as well as Nia Roberts, who contributed to this research.
Importance and key findings
The key takeaway is this: Cochrane LSRs mostly follow their own standards, but those outside of Cochrane do not. That being said, interestingly only a few Cochrane LSRs got a perfect score in establishing the ‘why’ and ‘how’ for being living. None of the non-Cochrane LSRs got perfect scores either. Cochrane LSRs met a higher percentage of guidance items (83% met three or more) compared to non-Cochrane LSRs (9%). Though this is not ideal, Cochrane’s living standards were not developed to assess compliance, but rather as items to keep in mind when deciding on and conducting an LSR. But if the only established methodological criteria and guidance for living systematic reviews is not being followed, why is it not being followed? Is it simply a lack of knowledge that the standards exist, or are these standards unachievable for most researchers?
Additionally, the overview showed that only a minority of the first living versions of LSRs (22%) were referenced in clinical practice guidelines; first living versions were examined as it can be assumed that if an LSR was cited in a guideline, it would continue to be cited throughout the LSR lifetime. These provide guidance for healthcare practitioners on how best to treat patients. If the whole idea of a living process is to get evidence more quickly to the point of care, then why are most LSRs not being used in guidelines? There is a gap. Considering adopting a living method is more time and resource-intensive compared to a standard update process, there is research waste created.
Though my perspective may be biased as a fellow University of Oxford researcher, Cochrane’s living systematic review on electronic cigarettes for smoking cessation maintained by researchers in the Centre for Evidence-Based Medicine consistently meets all living criteria and guidance, serving as a good example of transparency in reporting, and has been cited more frequently than any other LSR.
The next step in my research focuses on why so many LSR authors are not following best practices and create solutions to change that pattern. This is being accomplished through a survey and workshop aimed at identifying facilitators and barriers to living evidence synthesis generally, followed by a Delphi consensus process for living evidence methodology that can be achieved by Cochrane and non-Cochrane review authors alike.
Dissemination
Data from my research work were presented as a 15-minute oral presentation at the 2023 EBHC Conference in Taormina, Italy, prior to completion of data analysis. The presentation was titled, ’Dead on Arrival? An overview of living systematic reviews and their methodological rigor.’ The work was also mentioned by Hilda Bastian in her blog Absolutely Maybe, in a post titled ‘A View From the Sicily Evidence-Based Healthcare Conference.’ It has been submitted for publication in a peer-reviewed journal.
Since the original search in 2022 that found 23 Cochrane LSRs, an additional search in 2025 showed an increasing rate of LSR publications: 16 new Cochrane LSRs were published in the intervening 3 years. Without a common understanding of what it means to be living, those creating LSRs risk wasting time and resources. LSRs are also not consistently cited in guidelines, potentially diminishing their value to inform clinical practice.
What to read next
Hear Ranin Soliman's experiences during her recently completed DPhil in EBHC about generating evidence to improve childhood cancer health outcomes and resource use in Egypt
Ranin Soliman completed her DPhil in Evidence-Based Health Care in May 2023. Her DPhil research work focused on generating evidence to improve childhood cancer health outcomes and resource use in Egypt.
Arsenio Paez shares with us his experiences during his DPhil in Evidence Based-Health Care and his plans to utilise these in his research, and with the students he teaches.
27 July 2023
Arsenio Paez completed his DPhil in Evidence-Based Health Care in May 2023. Here he shares how this experience will inform his research focused on sleep, Alzheimer's Disease, and dementia and his teaching, in Canada.