Article type
Year
Abstract
Background: Monte Carlo simulations, which are computational algorithms that use random sampling to generate numerical results, have been used to demonstrate an extremely low probability of randomization for trials that were subsequently confirmed to be fabricated. This objective method could also be used in Cochrane Reviews where the quality of random sequence generation, at present, is only assessed subjectively by the description of randomization in publications or by correspondence with randomized controlled trial (RCT) authors.
Objectives: we aim to demonstrate that Monte Carlo simulations could be used to evaluate the quality of random sampling in Cochrane Reviews by applying this method to two published Cochrane Reviews investigating the effectiveness of endometrial scratching for in-vitro fertilization (IVF) and intrauterine insemination (IUI)/natural intercourse.
Methods: we extracted the baseline characteristics across intervention groups from RCTs with full text included in the two Cochrane Reviews on endometrial scratching to improve fertility. Monte Carlo simulations were used to generate a P value for differences between means for each baseline continuously-valued variable or proportions for each baseline categorical variable. If randomization has been done correctly then the set of P values from all baseline variables in studies should follow a uniform (0,1) distribution, that is, they should be randomly drawn values between 0 and 1. Stouffer’s method was used to combine the P values for all baseline variables in a study to generate a single combined P value for that study. We then used the Kolmogorov–Smirnov test, against a uniform distribution (0,1), for the P values of baseline variables and RCTs, to check for the effectiveness of randomization across studies. This analysis is a part of a larger project that aims to evaluate the quality of included RCTs from multiple perspectives.
Results: for 11 RCTs included in the Cochrane Review for IVF, there was no evidence against the assertion that P values from all baseline variables followed the expected uniform distribution, P = 0.8654; whereas there was a strong evidence against the null hypothesis that the P values followed the uniform distribution in seven RCTs concerning IUI/intercourse, P = 0.00001754 (Figure 1). Similarly, the distribution of pooled P values for RCTs with respect to IVF were likely to follow the expected uniform distribution, P = 0.5825, in contrast, RCTs regarding IUI/intercourse did not follow the expected uniform distribution, P = 0.00007707.
Conclusions: Monte Carlo simulations could be used to evaluate the probability of randomization across RCTs in Cochrane Reviews. In the case of a low probability, additional quality assessment such as acquiring and analyzing individual participant data could be considered before including or pooling RCTs.
Patient or healthcare consumer involvement: none
Objectives: we aim to demonstrate that Monte Carlo simulations could be used to evaluate the quality of random sampling in Cochrane Reviews by applying this method to two published Cochrane Reviews investigating the effectiveness of endometrial scratching for in-vitro fertilization (IVF) and intrauterine insemination (IUI)/natural intercourse.
Methods: we extracted the baseline characteristics across intervention groups from RCTs with full text included in the two Cochrane Reviews on endometrial scratching to improve fertility. Monte Carlo simulations were used to generate a P value for differences between means for each baseline continuously-valued variable or proportions for each baseline categorical variable. If randomization has been done correctly then the set of P values from all baseline variables in studies should follow a uniform (0,1) distribution, that is, they should be randomly drawn values between 0 and 1. Stouffer’s method was used to combine the P values for all baseline variables in a study to generate a single combined P value for that study. We then used the Kolmogorov–Smirnov test, against a uniform distribution (0,1), for the P values of baseline variables and RCTs, to check for the effectiveness of randomization across studies. This analysis is a part of a larger project that aims to evaluate the quality of included RCTs from multiple perspectives.
Results: for 11 RCTs included in the Cochrane Review for IVF, there was no evidence against the assertion that P values from all baseline variables followed the expected uniform distribution, P = 0.8654; whereas there was a strong evidence against the null hypothesis that the P values followed the uniform distribution in seven RCTs concerning IUI/intercourse, P = 0.00001754 (Figure 1). Similarly, the distribution of pooled P values for RCTs with respect to IVF were likely to follow the expected uniform distribution, P = 0.5825, in contrast, RCTs regarding IUI/intercourse did not follow the expected uniform distribution, P = 0.00007707.
Conclusions: Monte Carlo simulations could be used to evaluate the probability of randomization across RCTs in Cochrane Reviews. In the case of a low probability, additional quality assessment such as acquiring and analyzing individual participant data could be considered before including or pooling RCTs.
Patient or healthcare consumer involvement: none