Background: In complementary and alternative medicine there is need for evidence-based assessments, especially meta-analyses. In meta-analyses, as the number of trials in primary analyses decreases, the risk of false positives or false negatives increases. The same is true for meta-regression in multiple subgroup analyses. This is due to the assumption of normality which may not hold in small samples. Therefore, creation of a distribution from the observed trials using permutation and bootstrap methods to calculate p-values and confidence intervals may allow for less spurious findings. Objectives: To perform an empirical study to explore the differences in results for meta-analyses on a small number of trials using standard large sample approaches verses permutation and bootstrap based methods. Methods: We isolated a sample of randomized controlled clinical trials (RCTs) for of an intervention that has a small number of trials (herbal medicine trials). Trials were then assessed using the Jadad scale and data was extracted. Finally, we performed meta-analyses on the primary outcome of each group of trials and meta-regression for methodological quality subgroups within each meta-analysis. We used permutation and bootstrap methods to arrive at p-values and confidence intervals. We then compared p-values and confidence intervals between methods. Results: We collected 145 RCTs for this analysis. In the first subgroup of papers (n = 11), we performed a random effects analysis on trials testing Hypericum perforatum (St. John’s Wort; SJW) against placebo for depression, for change from baseline on the Hamilton depression scale (HAM-D). We found a weighted mean difference (WMD) of -2.95 (95% CI: -4.0 to -1.91; p = 0.01) and an I² of 56.2%. We are currently performing permutation and bootstrap methods and will present comparisons of the p-values and confidence intervals to those arrived at with standard large sample methods for several groups and numbers of trials.