Background: The ‘Risk of bias’ (ROB) tool was recently introduced to assess internal validity of randomized controlled trials based on six domains: sequence generation, allocation concealment (AC), blinding, incomplete outcome data, selective outcome reporting, and other sources of bias. Objectives: To evaluate: (1) the feasibility and inter-rater agreement of the ROB tool; (2) the concurrent validity compared to the Jadad Scale and Schulz approach to AC; and (3) the relationship between overall risk as assessed by the ROB tool and study effect estimates. Methods: A random sample of 80 trials was selected from an existing database. Two reviewers independently applied the ROB tool for a single outcome that was pre-specified; the trials had previously been assessed using the Jadad scale. We tracked the time to complete the tool and assessed inter-rater agreement using weighted Kappa. Using Kendall’s Tau statistic, we assessed the degree of correlation for: overall ROB risk assessment vs. Jadad scores and Schulz AC; high/low risk based on ROB vs. low/high quality based on Jadad (score < vs. >= 3). Results: The median time using the ROB tool was eight minutes per study for a single outcome (interquartile range 7-10; range 4-22). Agreement on individual domains ranged from poor (k = 0.15, selective reporting) to moderate (k = 0.59, sequence generation); agreement for overall risk of bias was fair (k = 0.38). There was a positive correlation between ROB overall risk vs. Jadad scores (Kendall’s Tau = 0.491, p < 0.0005); ROB overall risk vs. Schulz AC (0.438, p < 0.0005); and ROB high/low risk vs. Jadad low/high quality (0.406; p = 0.002). Analyses comparing effect estimates and assessment of risk are ongoing. Conclusions: There was substantial variation in agreement across domains of the ROB tool. Generally, the items with poor agreement were those that required judgement regarding the potential for the study methods to yield biased results. There was high correlation between overall assessments using the ROB tool and two commonly used approaches to quality assessment. Careful training and decision rules are required when applying the ROB tool.