The RoB NMA Tool: development of a tool to assess risk of bias (RoB) in network meta-analysis (NMA)

Oral
2024 Prague [Global Evidence Summit]
Dias S1, Higgins J2, Hutton B3, Lunny C4, Tricco A5, Veroniki A5, White I6, Whiting P7, Wright J4
1Centre for Reviews and Dissemination, University of York, York, UK
2Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, Bristol, UK; NIHR Applied Research Collaboration West (ARC West) at University Hospitals Bristol and Weston NHS Foundation Trust, Bristol, Bristol, UK
3Ottawa Research Institute, Ottawa, Ontario, Canada
4Cochrane Hypertension Group, University Of British Columbia, Vancouver, BC, Canada
5Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Unity Health Toronto, Toronto, Ontario, Canada; Institute for Health Policy, Management, and Evaluation, Toronto, Ontario, Canada
6MRC Clinical Trials Unit, University College London, London, London, UK
7Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Unity Health Toronto, Toronto, Ontario, Canada

Background:

Network meta-analysis (NMA) is an extension of pairwise meta-analysis that aims to synthesize evidence simultaneously from multiple primary studies on healthcare interventions of interest. Our aim was to develop a tool to assess the potential risk of bias (RoB) in the results and conclusions at the NMA level.

Methods:

A steering committee of 9 experts was convened, and conceptual decisions were made about the type of tool that would be developed. The steering committee initially decided that the tool would focus on aspects specific to NMA bias and not general systematic review biases. The development of the tool consisted of 6 steps: project management, item generation, decision-maker survey, Delphi exercise, tool refinement, and pilot testing and refinement. For item generation, we conducted a methodological review of tools, scientific papers, and editorial standards that present items related to NMA bias, reporting, or methodological quality. We searched MEDLINE, the Cochrane library, and unpublished literature. A decision-maker survey solicited feedback about the structure of the tool. A 2-round Delphi exercise asked respondents to rate whether items should be included (>70% agreement). The tool was piloted and refined.

Results:

A RoB in NMA (RoB NMA) tool was developed consisting of 3 domains (interventions and network geometry, effect modifiers, and statistical synthesis) and 2 overall judgments of the results and the interpretation of the findings into conclusions. Each domain and overall judgment are assessed in terms of RoB. Seventeen signaling statements are included to help judge RoB. Problems identified at the domain level feed into the assessment of the author’s conclusions. The RoB NMA tool is not designed to assess the systematic review portion of the NMA, and it therefore should be used in combination with ROBIS (which we recommend) or other similar tools (eg, AMSTAR-2) that aim to evaluate RoB in a systematic review.

Conclusions:

We developed the first tool for assessing bias in the findings and conclusions of NMAs. Being able to appraise the findings of published NMAs critically is central to evidence-based medicine.