Background: The appropriate classification of study designs is essential in systematic reviews to guide (1) decisions around inclusion, (2) risk of bias assessments, (3) pooling of studies, and (4) interpretation of the body of evidence for end-users. Objectives: To identify tools used to classify study designs, select a single tool for evaluation, develop guidelines for application of the tool, and test the tool and accompanying guidelines for concurrent validity and inter-rater reliability. Methods: Classification tools were identified through communication with a sample of key informants. The steering committee developed and applied criteria to select one tool for closer evaluation; then, developed a glossary and guidelines to accompany the tool. Three members of the steering committee classified 30 studies representing all considered designs (reference standard). Six reviewers, varying in training and experience, will independently apply the tool to the same 30 studies. Inter-rater reliability and accuracy against the reference standard will be measured. Results: We requested information from 24 individuals and identified 10 classification schemes and algorithms. The top-ranked tools were algorithms starting either from assignment of the intervention/exposure or the use of a comparison group. None of the tools included a comprehensive range of study designs, particularly for single group types. Nomenclature was inconsistent across tools. The algorithm developed by the Cochrane Non-Randomized Studies Methods Group was selected for testing and was modified to include study designs that were captured in other schemes and algorithms. A glossary and guidelines have been developed. Testing of the revised algorithm is ongoing; results will be available in October 2009. Conclusion: Classification tools are important for systematic reviewers as the scope of reviews extends beyond efficacy and randomized trials. The end-product of this study will be a comprehensive and reliable tool for classifying designs that can then be used to inform the review process.