Background: The risk of bias (ROB) tool was recently introduced by the Cochrane Collaboration to assess internal validity of randomized trials. Objectives: To evaluate (1) inter-rater agreement of the ROB tool, (2) time to access supplemental study information, (3) concurrent validity compared to Jadad and Schulz allocation concealment, and (4) relationship between ROB and effect estimates. Methods: We conducted a systematic review of long-acting beta agonists combined with inhaled corticosteroids for adults with persistent asthma. Two reviewers independently assessed trials using ROB, Jadad and allocation concealment. One reviewer searched for supplemental information including protocols. Median and interquartile range (IQR) time for ROB assessments and searches for supplemental information are reported. We assessed inter-rater agreement using weighted Kappa (κ) and the degree of correlation between tools using Kendall’s Tau (τ ). Results: ROB has been assessed for 36/107 trials. The trials were of high methodological quality based on Jadad scores (median 4, IQR 3 to 4); however, 85% had unclear allocation concealment and 81% high ROB. The factor that most influenced ROB was the potential inappropriate influence of study sponsors (95% industry funded). Agreement on ROB domains was moderate (κ=0.47) to substantial (κ=0.80), and moderate for overall ROB (κ=0.56). Median time to complete ROB assessments was 28 minutes (IQR: 22 to 36) and 13 minutes (IQR: 9 to 16) to search for supplemental information. Supplemental information was identified for five studies (13%); in two cases the assessment of selective outcome reporting changed. There was poor correlation between overall ROB vs. Jadad (τ =0.013, p=0.9) and allocation concealment (τ =−0.02, p=0.8). Analyses comparing effect estimates and risk are ongoing. Conclusion Inter-rater agreement on ROB assessments was better than previously reported suggesting that training and clear review-specific guidelines are critical. The correlation between ROB and Jadad was low suggesting measurement of different constructs (i.e., true risk of bias vs. quality of reporting).