Article type
Year
Abstract
Introduction: Reports of meta-analyses and systematic reviews are playing an increasingly important role in the dissemination of evidence-based health care. As part of the conduct of these studies there is uncertainty about the merits of including 'grey' literature. This literature can be defined as 'difficult to identify and retrieve' and includes: unpublished studies, conference proceedings, theses, industrial reports and applications. The nature of grey literature can make its exclusion more convenient for meta-analysts
Objectives: To determine whether the inclusion of grey literature in meta-analyses, compared to its exclusion, influences the point estimate, precision or statistical heterogeneity of the reported results.
Methods: A random sample of 135 meta-analyses published between 1983 and 1995. All meta-analyses were retrieved and reviewed to determine if they contained grey literature. For each meta-analysis that included grey literature, the included studies were retrieved, and the analyses replicated and then repeated with all grey items removed. Impact of grey literature on treatment effect estimates was assessed using a logistic regression model which related an unwanted event with study covariates: treatment, trial, meta-analysis, grey literature and trial by grey literature interaction. The impact of grey literature was quantified using a ratio of odds-ratios (i.e. ratio of treatment effect odds ratios of without and with grey literature) and its 95% confidence interval
Results: Of the 135 meta-analyses retrieved, 38 (28%) included grey literature. When grey literature was excluded from the analysis, compared to this information was included, there was a statistically significant increase in the reported effectiveness of the intervention, by 12% (95%CI: 1.01, 1.23), on average. In 14 (34%) of the studies, the point estimate shifted by s10%, which was considered to be a clinically important difference. An overall shift towards more significant results was observed after the exclusion of the grey literature. In three (7%) studies, the significance of the results changed, in 2 cases, removing the grey literature makes the results significant.
Discussion: There does not appear to be any trend toward the inclusion or exclusion of grey literature in meta-analyses. However, meta-analysts that exclude this information run the risk of overestimating, and biasing, their intervention effectiveness. Due to the impact the grey literature has on the treatment effect, it is important that it be sought and included when it meets pre-defined inclusion criteria.
Objectives: To determine whether the inclusion of grey literature in meta-analyses, compared to its exclusion, influences the point estimate, precision or statistical heterogeneity of the reported results.
Methods: A random sample of 135 meta-analyses published between 1983 and 1995. All meta-analyses were retrieved and reviewed to determine if they contained grey literature. For each meta-analysis that included grey literature, the included studies were retrieved, and the analyses replicated and then repeated with all grey items removed. Impact of grey literature on treatment effect estimates was assessed using a logistic regression model which related an unwanted event with study covariates: treatment, trial, meta-analysis, grey literature and trial by grey literature interaction. The impact of grey literature was quantified using a ratio of odds-ratios (i.e. ratio of treatment effect odds ratios of without and with grey literature) and its 95% confidence interval
Results: Of the 135 meta-analyses retrieved, 38 (28%) included grey literature. When grey literature was excluded from the analysis, compared to this information was included, there was a statistically significant increase in the reported effectiveness of the intervention, by 12% (95%CI: 1.01, 1.23), on average. In 14 (34%) of the studies, the point estimate shifted by s10%, which was considered to be a clinically important difference. An overall shift towards more significant results was observed after the exclusion of the grey literature. In three (7%) studies, the significance of the results changed, in 2 cases, removing the grey literature makes the results significant.
Discussion: There does not appear to be any trend toward the inclusion or exclusion of grey literature in meta-analyses. However, meta-analysts that exclude this information run the risk of overestimating, and biasing, their intervention effectiveness. Due to the impact the grey literature has on the treatment effect, it is important that it be sought and included when it meets pre-defined inclusion criteria.