Educational Intervention Outcome Reporting Bias

Article type
Authors
Wiliams R1, Pigott T1, Valentine J2, Canada D2
1Research Methodology, Loyola University Chicago, Chicago, Illinois, United States
2Educational & Counseling Psychology, University of Louisville, Louisville, Kentucky, United States
Abstract
Background: There is a well-known tendency for studies to be published in peer-review journals when the results are statistically significant. Bias can also exist within the study when details about study methods are missing or omitted and when outcomes gathered in the original research are not reported in the final publication. When conducting a meta-analysis, these circumstances can obscure the magnitude and or direction of the treatment effect under study. Objective: The goal of this research is to estimate the magnitude of publication bias in the educational sciences. Methods: Dissertations related to education from 96 ‘‘very high’’ research universities were collected from 2001 through 2005. Those dissertations using an intervention involving preK-12 students and were ultimately published comprised the universe of relevant studies. All intervention outcomes were coded in both the dissertation and published version. The discrepancies in study outcome reporting between dissertation and publication were examined. Results: The initial search yielded 12,904 dissertations. Of those, we have screened 4,102 dissertations which resulted in 199 studies containing and intervention on preK-12 students. Only 16 of those 199 were ultimately published. Examination of these 16 studies provides evidence of prominent outcome reporting bias based on statistically significant treatment effects (OR = 2.43, RR = 1.48, p & .0031). Conclusions: The results of this study provide evidence of non-trivial outcome reporting bias in education research. Outcome reporting standards may help alleviate the magnitude of this effect in education but to our knowledge no such initiatives have been implemented specifically targeting educational intervention outcomes. The remaining two thirds of the study pool will be analyzed for this presentation.