Development of a new instrument to assess the credibility of effect modification

Tags: Oral
Schandelmaier S1, Briel M2, Sun X3, Dahabreh I4, Walsh M1, Thabane L1, Hoes AW5, Gagnier JJ6, Sauerbrei W7, Hayward RA6, Ioannidis JP8, Van der Heijden GJ9, Borenstein M10, Schmid CH4, Varadhan R11, Guyatt GH1
1McMaster University, 2University Hospital Basel, 3Cochrane China, 4Brown University, 5University Medical Center Utrecht, 6University of Michigan, 7Freiburg University, 8Stanford University, 9University of Amsterdam, 10Biostat, 11John Hopkins University

Background:

Analyses of effect modification in randomized trials and meta-analyses are important to inform personalized medicine, but can result in spurious inferences. For example, debates regarding the credibility of subgroup analyses and meta-regression are often contentious. A formal, consensus-based, and user-tested instrument to assess the credibility of effect modification in randomized trials and meta-analyses remains unavailable. We are currently developing such an instrument.

Methods:

We are following a rigorous instrument development process. Our systematic survey of the methodological literature on the credibility of effect modification analyses identified 36 candidate items for the new instrument (Table 1). In collaboration with 10 experts (all authors of at least two methodological publications addressing effect modification), we are currently conducting a consensus study in which the team provides feedback on sequential versions of a new instrument. In addition, we will involve potential users of the new instrument including trialists, systematic reviewers, guideline developers, and journals editors. They will apply the draft instrument to a sample of subgroup analyses using formal user testing methods. Once the instrument is finalized, we will perform a sensibility testing with a second group of experts, and a reliability study with a second group of users.

Results:

Having presented the concept and candidate items at the last Cochrane Colloquium, we will now present the instrument that emerges from the consensus study and the user testing. We will demonstrate the use of the credibility instrument in a number of examples and discuss its compatibility with Cochrane Reviews and the GRADE approach.

Conclusion:

We expect that the new instrument will improve the analysis, interpretation, and reporting of effect modification in individual trials and meta-analyses.

Patient or healthcare consumer involvement: Not applicable