Background:

QI studies often employ quasi-experimental or observational designs. Conclusions of these studies can be misleading if they are overzealous in stating the causal relationship between the intervention and outcomes. The extent of this practice is, however, unclear. We hypothesize that studies using weaker designs frequently overstate the causal inferences in the conclusion.

Objective:

To assess the appropriateness of conclusions reported in recently published QI intervention studies in relation to the study design.

Methods:

This study consists of three phases. In Phase I, we hand-searched 11 major medical journals or health services research journals for studies of QI interventions published between January 2002 and December 2003. Eligible studies were those evaluating interventions that aimed to change health professional behaviours based on research evidence. This included various forms of continuing education, quality assurance, and behavioural interventions that could affect the ability of health care professionals to deliver services more effectively and efficiently. This phase has been completed.

During Phase II, all eligible studies will receive a full review by two reviewers. Information, including the study characteristics, design, and methodology will be collected. Further, key statements, addressing the casual effect between the intervention and outcomes in the abstract and the conclusion section, will be recorded.

Assessment of the strength of the causal inference suggested by the studies will be done by a 15-member expert panel using Delphi method in Phase III. This panel consists of leaders in the fields of QI intervention research, clinical study design and evidence-based practice. The members, who are blinded to the title, authors and the study design, will rate the strength of causal relationships for each set of the concluding statements on an 11-point scale. Student s t-tests will be used to compare the ratings between the randomized controlled trials (RCTs) and the non-RCTs. Separate analyses will be done for statements recorded from the abstract and those recorded from the text.

Preliminary

results:

Of the 4543 titles / abstracts, 80 eligible articles were identified (Year 2002 = 34; Year 2003 = 46). Half of these studies employed quasi-experimental or observational designs. The feasibility and comprehensiveness of the data extraction form was assessed in a pilot study using 10 of the eligible studies (5 randomized controlled trials (RCTs); 5 non-RCTs). The result showed that the reviewers were able to identify the same concluding statements concerning causal relationships.

Discussion:

Both experimental and non-experimental designs have been used by the currently published QI intervention studies. Further, it was feasible for the reviewers to identify statements discussing the causal relationship between the intervention and outcomes. Findings of this study will be the first step in improving the precision of conclusions stated in QI intervention studies.