Investigation of within-study selective reporting in clinical research: Follow-up of applications submitted to a local research ethics committee

Hahn S, Williamson P, Hutton J

Abstract: Publication bias is a widely recognised problem which may give rise to misleading results in a meta-analysis. A related problem, defined as within-study selective reporting, has been less well investigated. For example, it may be that several outcomes are measured and analysed but only a selected subset is reported. A similar problem exists when subgroup analyses are reported or not depending on the results. The potential biases due to selective reporting of outcomes or subgroup analyses have particular relevance in meta-analysis. Although widely believed to exist, to date there have been no empirical studies to assess the extent of the problem of within-study selective reporting. Here we report the results of a pilot study, involving only a single LREC (Local Research Ethics Committee), in which we compared the original study protocol and the subsequent study report for approved applications. Our aim was to assess the feasibility of examining within-study selection via this approach with a view to undertaking a larger study if successful. We received 41 (73%) replies from lead researchers of 56 clinical research applications approved in a particular time period by the LREC. Fifteen of these projects, which were completed and published at the time of our study, were further investigated. Only six (40%) stated which outcome variables were of primary interest and 4 (67%) of these showed consistency in the reports. Eight (53%) of the 15 studies mentioned an analysis plan in some way, but studies were rarely specific about the plan in the protocol. Seven (88%) of these eight studies did not follow their prescribed analysis plan: the analysis of stated outcome variables or planned analyses to explore the association between certain variables mentioned in the protocol were found to be missing from the report. Our pilot study to compare LREC approved protocols with subsequent study reports showed that within-study selective reporting may be examined qualitatively by this approach but that it is difficult to quantify the bias as protocols are not sufficiently precise.