Background: The new Cochrane ‘Risk of bias’ tool (RoB) assesses whether the results of a randomized controlled trial (RCT) are potentially biased. Objectives: To examine the feasibility, ease of use, and reliability of using RoB in a pilot study. Methods: The feasibility and ease of use of RoB were examined among experienced systematic reviewers new to RoB by assessing the problems and questions raised with the first 5 RCTs and again after 25 studies using RoB. Inter-rater reliability was assessed with the kappa coefficient for the 6 individual domains of RoB and for the overall risk for bias at the study level after 5 and then 25 studies. Two systematic reviews of clinical drug efficacy and safety were used for data collection which was performed independently by two persons. Results: Reviewers generally liked the separation of the ‘‘description’’ from the ‘‘reviewer authors’ judgment’’ and felt comfortable with the domains of sequence generation, allocation concealment, and blinding. Reviewers had some difficulty assessing the domains of incomplete outcome data and selective outcome reporting. The most commonly reported ‘‘other source of bias’’ was industry funding of the study. There was uncertainty among reviewers over the meaning ‘‘unclear risk of bias’’ at the study level. Reviewers who had already read and abstracted data from a study initially took 30 to 40 additional minutes to use RoB, but that time rapidly decreased to 10 to15 minutes after the examination of more than 5 trials. Kappa coefficients for the first 5 trials were low (o0.6); results after reviews of 25 studies are pending. Conclusions: In this pilot study, the constructs and explicit nature of RoB were well received by experienced systematic reviewers. The assessment of incomplete outcome data and selective outcome reporting were initially problematic, with improvement with usage. The time required for RoB completion decreased significantly over time. Follow-up data after extended use are pending. This assessment of RoB is limited by the narrow focus of the topics and may not be applicable to broader groups of systematic reviewers with different experience levels.