Article type
Year
Abstract
Background: The testing of multiple hypotheses is a well known problem in biomedical research. It has received widespread attention in the past, especially within the framework of randomized clinical trials (RCTs). Guidelines give clear advice how to handle this problem. Multiplicity can arise through several mechanisms also in systematic reviews (SRs), e.g. by multiple outcome variables, multiple treatments to be compared, etc. Astonishingly, this issue seems to be neglected in the methodological literature with regard to SRs.
Objectives: It was the aim of our study to get an impression whether multiplicity is an implicit or explicit issue in SRs.
Methods: A random sample of 100 SRs was drawn from the Cochrane Library, 25 each for the periods 1998-1999, 2000-2001, 2002-2003, and 2004-2005. To be included, the reviews had to deal with therapeutic interventions, and the reviews had to summarize the results of at least 5 single trials. Protocols and / or updates were excluded. Two independent reviewers looked (i) whether multiplicity was stated as a problem in the reviews themselves (explicit issue), (ii) whether a distinction was made between primary and secondary outcomes, and (iii) how many outcomes should be investigated (implicit issue).
Results: (i) Multiplicity was stated as a problem of the review itself in only 4 reviews. (ii) A distinction was made between primary and secondary outcomes in 47 reviews. However, there is a trend for an increase with regard to such a distinction, from 24% in the first time period to 75% in 2004-2005. (iii) A median (1st, 3rd quartile) of 6 (4, 9) outcomes was used in the reviews, with no clear time trend for an increase or decrease. A median of 2 (1, 3) outcomes were defined as primary in those reviews which made a distinction.
Conclusions: In SRs, multiplicity is only rarely mentioned as a problem of the reviews themselves. In only about one fifth of all reviews one single outcome was predefined as primary as it is a part of good statistical practice in RCTs. Methodological advice is urgently necessary when considering how to deal with the multiplicity issue in SRs.
Objectives: It was the aim of our study to get an impression whether multiplicity is an implicit or explicit issue in SRs.
Methods: A random sample of 100 SRs was drawn from the Cochrane Library, 25 each for the periods 1998-1999, 2000-2001, 2002-2003, and 2004-2005. To be included, the reviews had to deal with therapeutic interventions, and the reviews had to summarize the results of at least 5 single trials. Protocols and / or updates were excluded. Two independent reviewers looked (i) whether multiplicity was stated as a problem in the reviews themselves (explicit issue), (ii) whether a distinction was made between primary and secondary outcomes, and (iii) how many outcomes should be investigated (implicit issue).
Results: (i) Multiplicity was stated as a problem of the review itself in only 4 reviews. (ii) A distinction was made between primary and secondary outcomes in 47 reviews. However, there is a trend for an increase with regard to such a distinction, from 24% in the first time period to 75% in 2004-2005. (iii) A median (1st, 3rd quartile) of 6 (4, 9) outcomes was used in the reviews, with no clear time trend for an increase or decrease. A median of 2 (1, 3) outcomes were defined as primary in those reviews which made a distinction.
Conclusions: In SRs, multiplicity is only rarely mentioned as a problem of the reviews themselves. In only about one fifth of all reviews one single outcome was predefined as primary as it is a part of good statistical practice in RCTs. Methodological advice is urgently necessary when considering how to deal with the multiplicity issue in SRs.