Article type
Year
Abstract
Background: Assessing methodological quality is an important filter for selecting potentially eligible studies for a systematic review. The Quality Assessment of Studies of Diagnostic Accuracy included in Systematic Reviews (QUADAS) tool differs from previous instruments in that formal Delphi rounds were employed to assemble its 14 items. As yet, little practical experience was gained with QUADAS.
Objectives: We were interested in the inter-rater agreement with individual QUADAS elements. Also, we wanted to clarify whether adherence to QUADAS standards affects measures of diagnostic accuracy.
Methods: In a systematic review of the efficacy of focused abdominal sonography for trauma (FAST), we used QUADAS as the primary methodological scoring tool. Manuscripts of any language were obtained by scanning various medical databases and a manual search. Two reviewers independently checked whether individual studies fulfilled QUADAS standards. After the initial evaluation phase, we calculated kappa statistics (k) and solved inconsistencies by consensus. Heterogeneity of reported sensitivity (SN) and specificity (SP) was assessed by Galbraith plots. We used random-effects meta-regression to model the impact of single QUADAS items on reported measures of test accuracy.
Results: Of 957 citations originally identified, 62 studies enrolling 18,167 subjects were considered eligible. Studies fulfilled a median of 7 QUADAS items (range, 2 - 10). We noted poor inter-rater agreement with the appropriateness of the reference test (k=0.12) and its independent application (k=0.03, mainly clinical observation). Specification of drop-outs and the time interval between index and reference test yielded k-values of 0.23 and 0.34, respectively. The remaining ten features were rated with substantial to almost perfect agreement.
Strong heterogeneity was observed among studies with respect to reported SN, whereas SP was markedly constant. By multivariate meta-regression, the use of a single reference test (rated with k=0.61) was the strongest predictor of SN. Whereas the overall SN of FAST was estimated at 79%, studies relying on a single reference test reported a sensitivity of only 66%.
Conclusions: In this study, most QUADAS items showed substantial inter-rater agreement. The appropriateness of verification procedures must be defined according to the test under investigation. Adherence to certain standards is closely associated with reported test accuracy.
Objectives: We were interested in the inter-rater agreement with individual QUADAS elements. Also, we wanted to clarify whether adherence to QUADAS standards affects measures of diagnostic accuracy.
Methods: In a systematic review of the efficacy of focused abdominal sonography for trauma (FAST), we used QUADAS as the primary methodological scoring tool. Manuscripts of any language were obtained by scanning various medical databases and a manual search. Two reviewers independently checked whether individual studies fulfilled QUADAS standards. After the initial evaluation phase, we calculated kappa statistics (k) and solved inconsistencies by consensus. Heterogeneity of reported sensitivity (SN) and specificity (SP) was assessed by Galbraith plots. We used random-effects meta-regression to model the impact of single QUADAS items on reported measures of test accuracy.
Results: Of 957 citations originally identified, 62 studies enrolling 18,167 subjects were considered eligible. Studies fulfilled a median of 7 QUADAS items (range, 2 - 10). We noted poor inter-rater agreement with the appropriateness of the reference test (k=0.12) and its independent application (k=0.03, mainly clinical observation). Specification of drop-outs and the time interval between index and reference test yielded k-values of 0.23 and 0.34, respectively. The remaining ten features were rated with substantial to almost perfect agreement.
Strong heterogeneity was observed among studies with respect to reported SN, whereas SP was markedly constant. By multivariate meta-regression, the use of a single reference test (rated with k=0.61) was the strongest predictor of SN. Whereas the overall SN of FAST was estimated at 79%, studies relying on a single reference test reported a sensitivity of only 66%.
Conclusions: In this study, most QUADAS items showed substantial inter-rater agreement. The appropriateness of verification procedures must be defined according to the test under investigation. Adherence to certain standards is closely associated with reported test accuracy.