Article type
Year
Abstract
Background: Monte Carlo simulations, which are computational algorithms that use random sampling to generate numerical results, have been used by Carlisle et al. to prove the extremely low probability of randomization for fabricated randomized controlled trials (RCTs). This objective method could also be used in Cochrane review where the quality of random sequence generation, at present, is only assessed by the subjective description of randomization from RCT authors.
Objectives: We aim to demonstrate that Monte Carlo simulations could be used to evaluate the quality of random sampling in Cochrane review by applying this method to two published Cochrane reviews about the effectiveness of endometrial scratching for in-vitro fertilization (IVF) and intrauterine insemination (IUI)/natural intercourse.
Methods: We extracted all the baseline characteristics across intervention groups from RCTs with full-text included in the two Cochrane reviews. Monte Carlo simulations were used to generate a p-value for differences between means for each baseline continuously-valued variable or proportions for each baseline categorical variable. If randomization has been done correctly then the set of p-values from all baseline variables in studies should follow a uniform [0,1] distribution, that is, they should be randomly drawn values between 0 and 1. Stouffer’s method was used to combine the p-values for all baseline variables in a study to generate a single combined p-value for that study. We then used the Kolmogorov–Smirnov test, against a uniform distribution [0,1], for the p values of baseline variables and RCTs, to check for the effectiveness of randomization across studies.
Results: For RCTs included in the Cochrane review for IVF, there was no evidence against the assertion that p-values from all baseline variables followed the expected uniform distribution, p=0.8654; whereas there was a strong evidence against the null hypothesis that the p-values followed the uniform distribution in RCTs concerning IUI/intercourse, p= 1.754*10^-5 (Figure 1). Similarly, the distribution of pooled p values for RCTs with respect to IVF was likely to follow the expected uniform distribution, p=0.5825, in contrast, RCTs regarding IUI/intercourse did not follow the expected uniform distribution, p=7.707*10^-5.
Conclusions: Monte Carlo simulations could be used to evaluate the probability of randomization across RCTs in Cochrane review. In the case of a low probability, additional quality assessments such as acquiring and analyzing individual participant data should be considered before pooling RCTs.
Patient or healthcare consumer involvement: None.
Objectives: We aim to demonstrate that Monte Carlo simulations could be used to evaluate the quality of random sampling in Cochrane review by applying this method to two published Cochrane reviews about the effectiveness of endometrial scratching for in-vitro fertilization (IVF) and intrauterine insemination (IUI)/natural intercourse.
Methods: We extracted all the baseline characteristics across intervention groups from RCTs with full-text included in the two Cochrane reviews. Monte Carlo simulations were used to generate a p-value for differences between means for each baseline continuously-valued variable or proportions for each baseline categorical variable. If randomization has been done correctly then the set of p-values from all baseline variables in studies should follow a uniform [0,1] distribution, that is, they should be randomly drawn values between 0 and 1. Stouffer’s method was used to combine the p-values for all baseline variables in a study to generate a single combined p-value for that study. We then used the Kolmogorov–Smirnov test, against a uniform distribution [0,1], for the p values of baseline variables and RCTs, to check for the effectiveness of randomization across studies.
Results: For RCTs included in the Cochrane review for IVF, there was no evidence against the assertion that p-values from all baseline variables followed the expected uniform distribution, p=0.8654; whereas there was a strong evidence against the null hypothesis that the p-values followed the uniform distribution in RCTs concerning IUI/intercourse, p= 1.754*10^-5 (Figure 1). Similarly, the distribution of pooled p values for RCTs with respect to IVF was likely to follow the expected uniform distribution, p=0.5825, in contrast, RCTs regarding IUI/intercourse did not follow the expected uniform distribution, p=7.707*10^-5.
Conclusions: Monte Carlo simulations could be used to evaluate the probability of randomization across RCTs in Cochrane review. In the case of a low probability, additional quality assessments such as acquiring and analyzing individual participant data should be considered before pooling RCTs.
Patient or healthcare consumer involvement: None.