Including interrupted time series (ITS) designs in systematic reviews

Tags: Poster
Thomson-O'Brien MA, Grilli R, Freemantle N, Ramsay C, Campbell M

Introduction/Objective: Although randomised controlled trials provide the most valid evidence about the impact of interventions, such designs are not always feasible. Useful information about the effectiveness of an intervention can be derived from less rigorous designs, such as an interrupted time series. We describe a process to assess the methodological quality of ITS designs and provide an example of one approach to the analysis.

Methods: In a systematic review by Grilli and colleagues (1998) of the impact of mass media campaigns on health services utilisation, they included 22 papers reporting 17 time series. Most campaigns were aimed at promoting of the use of specific health services (cancer screening, immunisation programmes, or emergency services for patients with suspected myocardial infarction). The quality of the studies was assessed by using criteria developed by EPOC based upon the work of Cook and Campbell (1979). When information about individual observations was only reported graphically in the original papers, Grilli et al (1998) derived the data set by computer scanning the figures. Consistency between the data collected with this approach and those explicitly reported in the paper was good and discrepancies were never greater than 1%. Grilli et al (1998) analysed the data using an autoregressive integrated moving average (ARIMA) model to isolate the effect of the intervention from existing time trends. A regression coefficient (with its standard error) that described the effect of the campaign was estimated. The direction of effect (e.g. positive or negative) was standardised so that a negative coefficient described an improvement in outcome attributable to the intervention.

Results: Only four studies relied on at sufficient data points to enable reliable statistical inference. In 13 studies, contextual changes could not be excluded. Most of the original studies described the time series without a formal statistical analysis. The ARIMA analysis showed that mass media campaigns appeared to be effective in improving immunisation (-1.19, 95%CI -2.73 to -1.96); cancer screening activities (-0.49, 95%CI -2.46 to -1.42); and emergency services for myocardial infarction (-0.16, 95%CI -2.74 to -1.45).

Discussion: Studies using time series designs can be incorporated into systematic reviews. Because these studies are prone to bias, their methodological quality must be carefully assessed. If insufficient data cannot be obtained in the published report, it may be possible to re-analyse data presented graphically. The use of ARIMA modelling which takes into account trends in event rates before the intervention and the interrelationship between data points is preferred.