Background: The Cochrane Handbook claims that the evaluation of the validity of included studies is essential for the analysis, interpretation and conclusions of systematic reviews. Therefore, reviews should include not only a risk of bias assessment but also should discuss how the limitations in the design and execution could affect the validity of their conclusions. Objectives: To evaluate to what extent the risk of bias assessment is properly considered to formulate the conclusions in set of published systematic reviews. Methods: Two reviewers will independently assess a set of systematic reviews used to develop a practice guideline on the surgical management of femoral fractures. We will use AMSTAR (Shea 2007), paying special attention to the relevant items that address whether the reviews documented any formal quality assessment (item 7), and if this assessment was used accurately to formulate the reviews conclusions (item 8). We will estimate an agreement coefficient between these two items, but will also qualitatively check the extent to which the quality of information was used to draw conclusions in the reviews. Results: So far, we have assessed 15 systematic reviews (5 Cochrane vs. 10 non Cochrane). Although 9 of these reviews properly considered the limits in design and execution of the included trials to formulate their conclusions, the rest (6 reviews, 40%) did not assess the quality of the included studies or did not use the quality assessment to discuss its impact on the review results. At this moment we are increasing the sample of assessed reviews to accurately estimate if these preliminary results could be considered significant. Conclusions: There still exists clear room for improvement in the way that the quality assessment of included studies is used to draw conclusions in systematic reviews. This issue is of special relevance for the usefulness of reviews in the process of knowledge translation, given the importance of bias assessment for allowing an appropriate evaluation of the confidence in the estimates derived from reviews.
Reference: Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C,
Porter AC, Tugwell P, Moher D, Bouter LM. Development of AMSTAR:
a measurement tool to assess the methodological quality of systematic
reviews. BMC Med Res Methodol. 2007;7:10. PMID: 17302989