Article type
Abstract
Background
Many errors can arise when preparing for, conducting, and interpreting the results of a meta-analysis [eg, using standard errors (of means) instead of standard deviations]. These errors can importantly impact the meta-analysis effect estimate (and associated statistics) and may result in misleading conclusions being drawn.
Objectives
To develop a checklist to help peer reviewers detect errors in the application and interpretation of pairwise meta-analyses in systematic reviews of interventions.
Methods
To develop the checklist, we are undertaking a 4-step process. Step 1: Generate a comprehensive item bank of possible errors by undertaking a systematic review of studies describing errors in the conduct and interpretation of meta-analysis. Step 2: Reduce the item bank by seeking input from individuals with expertise in methods for meta-analysis and journal editors via an online survey. Survey participants will be asked to provide feedback on the inclusion of each item, its phrasing, how easy the item is to assess as a peer reviewer, and whether the error is likely to be consequential. Step 3: Seek consensus on the items by holding a consensus meeting. Step 4: Draft the checklist and guidance document and pilot.
Results
In this presentation, we will describe the results from the first 2 steps. Step 1: In the systematic review, 50 articles were included, from which 139 errors were identified. The errors covered data extraction/manipulation (25/139), statistical analysis (74/139), and interpretation (40/139). Of the 139 errors, we judged that 105 may be identifiable from inspection of the systematic review only, while the remaining 34 errors would also require review of the primary studies. Step 2: We will present the results from the survey, including a preliminary set of checklist items to be discussed in the consensus meeting planned for September 2024.
Conclusions
The checklist may help journal editors and peer reviewers identify errors in the conduct and interpretation of meta-analyses in systematic reviews.
Relevance and importance to patients
By developing a tool to reduce statistical errors in meta-analyses, the evidence that underpins policy and clinical practice decisions can be improved.
Many errors can arise when preparing for, conducting, and interpreting the results of a meta-analysis [eg, using standard errors (of means) instead of standard deviations]. These errors can importantly impact the meta-analysis effect estimate (and associated statistics) and may result in misleading conclusions being drawn.
Objectives
To develop a checklist to help peer reviewers detect errors in the application and interpretation of pairwise meta-analyses in systematic reviews of interventions.
Methods
To develop the checklist, we are undertaking a 4-step process. Step 1: Generate a comprehensive item bank of possible errors by undertaking a systematic review of studies describing errors in the conduct and interpretation of meta-analysis. Step 2: Reduce the item bank by seeking input from individuals with expertise in methods for meta-analysis and journal editors via an online survey. Survey participants will be asked to provide feedback on the inclusion of each item, its phrasing, how easy the item is to assess as a peer reviewer, and whether the error is likely to be consequential. Step 3: Seek consensus on the items by holding a consensus meeting. Step 4: Draft the checklist and guidance document and pilot.
Results
In this presentation, we will describe the results from the first 2 steps. Step 1: In the systematic review, 50 articles were included, from which 139 errors were identified. The errors covered data extraction/manipulation (25/139), statistical analysis (74/139), and interpretation (40/139). Of the 139 errors, we judged that 105 may be identifiable from inspection of the systematic review only, while the remaining 34 errors would also require review of the primary studies. Step 2: We will present the results from the survey, including a preliminary set of checklist items to be discussed in the consensus meeting planned for September 2024.
Conclusions
The checklist may help journal editors and peer reviewers identify errors in the conduct and interpretation of meta-analyses in systematic reviews.
Relevance and importance to patients
By developing a tool to reduce statistical errors in meta-analyses, the evidence that underpins policy and clinical practice decisions can be improved.