The update of the JBI critical appraisal tools for risk of bias assessment: rationale and justification

Oral
2024 Prague [Global Evidence Summit]
Aromataris E1, Barker T2, Hasanaoff S2, Klugar M3, Leonardi-Bee J4, Liu X5, Munn Z2, Sears K6, Stone J1
1JBI, Faculty of Health and Medical Sciences, The University of Adelaide, Adelaide , South Australia, Australia
2Health Evidence Synthesis, Recommendations and Impact (HESRI), School of Public Health, The University of Adelaide, Adelaide, South Australia, Australia
3Cochrane Czech Republic, The Czech Republic: a JBI Centre of Excellence, Czech GRADE Network, Institute of Health Information and Statistics of the Czech Republic, Prague, Czech Republic; Center of Evidence-based Education and Arts Therapies: A JBI Affiliated Group, Faculty of Education, Palacký University, Olomouc, Czech Republic
4The Nottingham Centre for Evidence Based Healthcare: A JBI Centre of Excellence, School of Medicine, University of Nottingham, Nottingham, England
5Hong Kong Metropolitan University, Hong Kong, China
6Queen’s Collaboration for Health Care Quality, Queen’s University, Kingston, Ontario, Canada

Background:

JBI offers a suite of critical appraisal tools that are freely available to evidence synthesizers. These tools have been developed by JBI collaborators and approved by the JBI Scientific Committee following extensive consultation. However, recent methodological advancements have resulted in a movement from critical appraisal to using risk-of-bias assessments. Therefore, the existing suite of instruments do not align with these developments and could potentially conflate and confuse the process of critical appraisal with that of risk-of-bias assessment.

Objective:

This presentation aims to provide the rationale and justification for the aforementioned update, including what has changed and what has remained the same from the previous iterations of these tools.

Methods:

The JBI Effectiveness Methodology Group began the update by cataloguing the questions asked in each JBI critical appraisal tool for quantitative study designs. These questions were ordered into constructs of validity (internal, statistical conclusion, comprehensiveness of reporting, external) using consensus methods. For questions that were related to the internal validity construct, they were further catalogued to a domain of bias through a series of mapping exercises. Finally, questions were then separated based on whether they were answered at the study, outcome, or result level.

Results:

A strength of the JBI critical appraisal tools has been their simplicity and flexibility to facilitate assessments of risk of bias following different approaches. However, due to their presentation, using the tools following a domain-based approach was not intuitive to all users. By presenting the questions to the validity construct to which they belong and the domain of bias they address, users of the tools are better placed to follow a domain-based approach using these new tools. These redeveloped tools offer new options to evidence synthesizers to assess the risk of bias of a broad range of study types, to ensure their systematic reviews are informed by the best-available evidence and are therefore meaningful to end users.

Conclusion:

The revision to the JBI critical appraisal tools has provided greater flexibility to their users. This work will increase the usability and applicability of these instruments while maintaining consistency with modern advances in evidence synthesis.