Article type
Abstract
"Background
Despite the growth in systematic reviews, many meta-analyses in education are poorly conducted using questionable methods. Low quality reviews have consequences for the use of evidence in practice, making it difficult for policymakers to recognize what is strong evidence for decision making. In the past decade, meta-analysis methods have evolved including important innovations for the modeling of dependent effect size data structures and exploration of selection bias. It is critical to understand the quality of meta-analyses in education given their use for consequential decisions for student learning and growth.
Objectives
This meta-review aims to examine the following research questions: (a) To what extent are best-practice methods used in education meta-analyses? (b) How do the quality of the methods used in education meta-analyses differ across publication outlet, funding status and across time? and (c) In what ways do education meta-analyses report results to support the use of research evidence in practice?
Methods
A prospective protocol for this study was published in the Nordic Journal of Systematic Review in Education (Pellegrini et al., 2024). The included published meta-analyses were published between January 2011 and September 2023. 248 meta-analyses focusing on an education intervention with general education students in kindergarten through 12th grade were included in the meta-review.
Results
The team is currently coding and analyzing the results of this meta-review. There is an increase in the use of meta-analysis models that better reflect the complex structure of effect size data and that describe the heterogeneity across study results. Published meta-analyses often fail to report details of the samples used in primary studies, limiting the ability of decisionmakers to understand how results can apply to their own context.
Conclusions
The quality of education meta-analyses is improving but more support is needed for reviewers to use best-practice methods. Education meta-analyses should involve more stakeholders in the process of systematic review and provide more details to allow the use of the results in practice.
Note: The public/consumers were not involved in this study.
"
Despite the growth in systematic reviews, many meta-analyses in education are poorly conducted using questionable methods. Low quality reviews have consequences for the use of evidence in practice, making it difficult for policymakers to recognize what is strong evidence for decision making. In the past decade, meta-analysis methods have evolved including important innovations for the modeling of dependent effect size data structures and exploration of selection bias. It is critical to understand the quality of meta-analyses in education given their use for consequential decisions for student learning and growth.
Objectives
This meta-review aims to examine the following research questions: (a) To what extent are best-practice methods used in education meta-analyses? (b) How do the quality of the methods used in education meta-analyses differ across publication outlet, funding status and across time? and (c) In what ways do education meta-analyses report results to support the use of research evidence in practice?
Methods
A prospective protocol for this study was published in the Nordic Journal of Systematic Review in Education (Pellegrini et al., 2024). The included published meta-analyses were published between January 2011 and September 2023. 248 meta-analyses focusing on an education intervention with general education students in kindergarten through 12th grade were included in the meta-review.
Results
The team is currently coding and analyzing the results of this meta-review. There is an increase in the use of meta-analysis models that better reflect the complex structure of effect size data and that describe the heterogeneity across study results. Published meta-analyses often fail to report details of the samples used in primary studies, limiting the ability of decisionmakers to understand how results can apply to their own context.
Conclusions
The quality of education meta-analyses is improving but more support is needed for reviewers to use best-practice methods. Education meta-analyses should involve more stakeholders in the process of systematic review and provide more details to allow the use of the results in practice.
Note: The public/consumers were not involved in this study.
"