Article type
Year
Abstract
Background: It is difficult to adequately classify the study designs of included studies in the process of performing systematic review.
Objectives: To develop study designs typology in systematic reviews of intervention and test the inter-rater reliability, feasibility and comparison with an existing typology tool (Figure1).
Methods: To develop the typology, our study was conducted as follows: 1)establish a principle of development, 2) investigation of the existing study design typology, 3) develop new typology making up for the weak points in existing typology, 4) develop instructions for application of the tool to intervention studies, and 5) consult with experts about systematic reviews and clinical practice guidelines. To test the validity of the new typology, we used a sample of 70 studies. To assess the reliability of agreement among the three raters for all study design classification ratings and the agreement between the new typology and the National Institute for Health and Clinical Excellence (NICE) typology, we calculated the weighted kappa. We measured time and feasibility to apply the new typology tool compared with NICE.
Results: Inter-rater reliability was almost perfect among the testers (κ = 0.82) and the mean time to complete the new typology was significantly shorter than for the NICE typology (113.4 seconds (95% CI 99.4 to 127.4) vs. 132.6 seconds (95% CI 116.1 to 149.1), P = 0.046 for paired t-test). To evaluate the clarity and simplicity using 7 point Likert scale, completeness, and user-friendliness, the new typology was rated with mean scores 6, 6.6, and 6.3, respectively.
Conclusions: We suggest the use of the new typology to classify study designs of systematic reviews and guideline development for interventions. Application of the new tool to classify study designs should be accompanied by adequate training and decision-making based on explicit instruction.
Objectives: To develop study designs typology in systematic reviews of intervention and test the inter-rater reliability, feasibility and comparison with an existing typology tool (Figure1).
Methods: To develop the typology, our study was conducted as follows: 1)establish a principle of development, 2) investigation of the existing study design typology, 3) develop new typology making up for the weak points in existing typology, 4) develop instructions for application of the tool to intervention studies, and 5) consult with experts about systematic reviews and clinical practice guidelines. To test the validity of the new typology, we used a sample of 70 studies. To assess the reliability of agreement among the three raters for all study design classification ratings and the agreement between the new typology and the National Institute for Health and Clinical Excellence (NICE) typology, we calculated the weighted kappa. We measured time and feasibility to apply the new typology tool compared with NICE.
Results: Inter-rater reliability was almost perfect among the testers (κ = 0.82) and the mean time to complete the new typology was significantly shorter than for the NICE typology (113.4 seconds (95% CI 99.4 to 127.4) vs. 132.6 seconds (95% CI 116.1 to 149.1), P = 0.046 for paired t-test). To evaluate the clarity and simplicity using 7 point Likert scale, completeness, and user-friendliness, the new typology was rated with mean scores 6, 6.6, and 6.3, respectively.
Conclusions: We suggest the use of the new typology to classify study designs of systematic reviews and guideline development for interventions. Application of the new tool to classify study designs should be accompanied by adequate training and decision-making based on explicit instruction.
Images
