Article type
Abstract
Background: Network meta-analysis (NMA) is an extension of pairwise meta-analysis that aims to synthesize evidence simultaneously from multiple primary studies on healthcare interventions of interest. Our aim was to develop a tool to assess the potential risk of bias (RoB) in the results and conclusions at the NMA level.
Methods: A steering committee of 9 experts was convened, and conceptual decisions were made about the type of tool that would be developed. The steering committee initially decided that the tool would focus on aspects specific to NMA bias and not general systematic review biases. The development of the tool consisted of 6 steps: project management, item generation, decision-maker survey, Delphi exercise, tool refinement, and pilot testing and refinement. For item generation, we conducted a methodological review of tools, scientific papers, and editorial standards that present items related to NMA bias, reporting, or methodological quality. We searched MEDLINE, the Cochrane library, and unpublished literature. A decision-maker survey solicited feedback about the structure of the tool. A 2-round Delphi exercise asked respondents to rate whether items should be included (>70% agreement). The tool was piloted and refined.
Results: A RoB in NMA (RoB NMA) tool was developed consisting of 3 domains (interventions and network geometry, effect modifiers, and statistical synthesis) and 2 overall judgments of the results and the interpretation of the findings into conclusions. Each domain and overall judgment are assessed in terms of RoB. Seventeen signaling statements are included to help judge RoB. Problems identified at the domain level feed into the assessment of the author’s conclusions. The RoB NMA tool is not designed to assess the systematic review portion of the NMA, and it therefore should be used in combination with ROBIS (which we recommend) or other similar tools (eg, AMSTAR-2) that aim to evaluate RoB in a systematic review.
Conclusions: We developed the first tool for assessing bias in the findings and conclusions of NMAs. Being able to appraise the findings of published NMAs critically is central to evidence-based medicine.
Methods: A steering committee of 9 experts was convened, and conceptual decisions were made about the type of tool that would be developed. The steering committee initially decided that the tool would focus on aspects specific to NMA bias and not general systematic review biases. The development of the tool consisted of 6 steps: project management, item generation, decision-maker survey, Delphi exercise, tool refinement, and pilot testing and refinement. For item generation, we conducted a methodological review of tools, scientific papers, and editorial standards that present items related to NMA bias, reporting, or methodological quality. We searched MEDLINE, the Cochrane library, and unpublished literature. A decision-maker survey solicited feedback about the structure of the tool. A 2-round Delphi exercise asked respondents to rate whether items should be included (>70% agreement). The tool was piloted and refined.
Results: A RoB in NMA (RoB NMA) tool was developed consisting of 3 domains (interventions and network geometry, effect modifiers, and statistical synthesis) and 2 overall judgments of the results and the interpretation of the findings into conclusions. Each domain and overall judgment are assessed in terms of RoB. Seventeen signaling statements are included to help judge RoB. Problems identified at the domain level feed into the assessment of the author’s conclusions. The RoB NMA tool is not designed to assess the systematic review portion of the NMA, and it therefore should be used in combination with ROBIS (which we recommend) or other similar tools (eg, AMSTAR-2) that aim to evaluate RoB in a systematic review.
Conclusions: We developed the first tool for assessing bias in the findings and conclusions of NMAs. Being able to appraise the findings of published NMAs critically is central to evidence-based medicine.