Article type
Abstract
Background: JBI offers a suite of critical appraisal tools that are freely available to evidence synthesizers. These tools have been developed by JBI collaborators and approved by the JBI Scientific Committee following extensive consultation. However, recent methodological advancements have resulted in a movement from critical appraisal to using risk-of-bias assessments. Therefore, the existing suite of instruments do not align with these developments and could potentially conflate and confuse the process of critical appraisal with that of risk-of-bias assessment.
Objective: This presentation aims to provide the rationale and justification for the aforementioned update, including what has changed and what has remained the same from the previous iterations of these tools.
Methods: The JBI Effectiveness Methodology Group began the update by cataloguing the questions asked in each JBI critical appraisal tool for quantitative study designs. These questions were ordered into constructs of validity (internal, statistical conclusion, comprehensiveness of reporting, external) using consensus methods. For questions that were related to the internal validity construct, they were further catalogued to a domain of bias through a series of mapping exercises. Finally, questions were then separated based on whether they were answered at the study, outcome, or result level.
Results: A strength of the JBI critical appraisal tools has been their simplicity and flexibility to facilitate assessments of risk of bias following different approaches. However, due to their presentation, using the tools following a domain-based approach was not intuitive to all users. By presenting the questions to the validity construct to which they belong and the domain of bias they address, users of the tools are better placed to follow a domain-based approach using these new tools. These redeveloped tools offer new options to evidence synthesizers to assess the risk of bias of a broad range of study types, to ensure their systematic reviews are informed by the best-available evidence and are therefore meaningful to end users.
Conclusion: The revision to the JBI critical appraisal tools has provided greater flexibility to their users. This work will increase the usability and applicability of these instruments while maintaining consistency with modern advances in evidence synthesis.
Objective: This presentation aims to provide the rationale and justification for the aforementioned update, including what has changed and what has remained the same from the previous iterations of these tools.
Methods: The JBI Effectiveness Methodology Group began the update by cataloguing the questions asked in each JBI critical appraisal tool for quantitative study designs. These questions were ordered into constructs of validity (internal, statistical conclusion, comprehensiveness of reporting, external) using consensus methods. For questions that were related to the internal validity construct, they were further catalogued to a domain of bias through a series of mapping exercises. Finally, questions were then separated based on whether they were answered at the study, outcome, or result level.
Results: A strength of the JBI critical appraisal tools has been their simplicity and flexibility to facilitate assessments of risk of bias following different approaches. However, due to their presentation, using the tools following a domain-based approach was not intuitive to all users. By presenting the questions to the validity construct to which they belong and the domain of bias they address, users of the tools are better placed to follow a domain-based approach using these new tools. These redeveloped tools offer new options to evidence synthesizers to assess the risk of bias of a broad range of study types, to ensure their systematic reviews are informed by the best-available evidence and are therefore meaningful to end users.
Conclusion: The revision to the JBI critical appraisal tools has provided greater flexibility to their users. This work will increase the usability and applicability of these instruments while maintaining consistency with modern advances in evidence synthesis.