Article type
Year
Abstract
Background: Patient report outcome measures (PROMs) vary in their quality (psychometric properties) and clinical trials use a heterogeneous array of PROMs. Little empirical evidence exists on how poor-quality PROMs influence treatment effect estimates in clinical research.
Objectives: To assess the potential for bias treatment effects associated with PROMs of varying psychometric quality in randomized clinical trials (RCTs).
Methods: We searched for RCTs published in the past five years (January 2011 to December 2016) in the top five 2015 impact factor orthopaedic journals. We accepted RCTs including human participants with rotator cuff disease, published in English, and using PROMs specific to rotator cuff disease. We extracted data on study design, sample size, risk of bias (ROB) for RCTs, characteristics of PROM used, estimates of effect and associated measures of variance. PROMs were given numerical ratings of psychometric quality from a prior publication. We transformed continuous measures of effect by dividing the effect estimate by the standard deviation. We performed multi-level linear regression analyses to determine whether PROM quality was associated with the magnitude of effect.
Results: Overall, we included 72 RCTs reporting 174 separate outcomes. Mean sample size was 66.8 (95% CI 62.30 to 71.27), mean ROB score across all studies was 7.00/10 (95% CI 6.72 to 7.29), psychometric quality summary scores ranged from -2 to 10 and the standardized mean effect (SME) estimate was 0.47 (95% CI -0.17 to 1.11; Figure 1). Regression revealed that higher quality PROMs had smaller estimates of effect (β = -0.32; 95% CI -0.51 to -0.13; P = 0.001). We also found that a longer follow-up period predicted slightly increased effect estimates (β = 0.08; 95% CI 0.02 to 0.13; P = 0.007).
Conclusions: PROMs with poor or unknown psychometric properties overestimate treatment effects in clinical research of rotator cuff disease by 68.4% (β -0.32/SME 0.47). To our knowledge, this is the first empirical evidence that variations in the quality of PROMs bias treatment effect estimates. Researchers and clinicians using data from PROMs should be cautious and explore the quality of PROMs so as to not mislead decision-making base on potentially biased outcomes.
Patient or healthcare consumer involvement: This research focuses on PROMs, which are patient relevant.
Objectives: To assess the potential for bias treatment effects associated with PROMs of varying psychometric quality in randomized clinical trials (RCTs).
Methods: We searched for RCTs published in the past five years (January 2011 to December 2016) in the top five 2015 impact factor orthopaedic journals. We accepted RCTs including human participants with rotator cuff disease, published in English, and using PROMs specific to rotator cuff disease. We extracted data on study design, sample size, risk of bias (ROB) for RCTs, characteristics of PROM used, estimates of effect and associated measures of variance. PROMs were given numerical ratings of psychometric quality from a prior publication. We transformed continuous measures of effect by dividing the effect estimate by the standard deviation. We performed multi-level linear regression analyses to determine whether PROM quality was associated with the magnitude of effect.
Results: Overall, we included 72 RCTs reporting 174 separate outcomes. Mean sample size was 66.8 (95% CI 62.30 to 71.27), mean ROB score across all studies was 7.00/10 (95% CI 6.72 to 7.29), psychometric quality summary scores ranged from -2 to 10 and the standardized mean effect (SME) estimate was 0.47 (95% CI -0.17 to 1.11; Figure 1). Regression revealed that higher quality PROMs had smaller estimates of effect (β = -0.32; 95% CI -0.51 to -0.13; P = 0.001). We also found that a longer follow-up period predicted slightly increased effect estimates (β = 0.08; 95% CI 0.02 to 0.13; P = 0.007).
Conclusions: PROMs with poor or unknown psychometric properties overestimate treatment effects in clinical research of rotator cuff disease by 68.4% (β -0.32/SME 0.47). To our knowledge, this is the first empirical evidence that variations in the quality of PROMs bias treatment effect estimates. Researchers and clinicians using data from PROMs should be cautious and explore the quality of PROMs so as to not mislead decision-making base on potentially biased outcomes.
Patient or healthcare consumer involvement: This research focuses on PROMs, which are patient relevant.