Background: Unlike systematic reviews of the literature that rely on randomized control trials, reviews that require the inclusion of observational studies are handicapped by the absence of validated instruments for evaluating the quality of these studies. The idiosyncrasies of each systematic review of observational studies coupled with the absence of a validated tool often mean that each new project entails the development of a new quality rating tool. Objectives: Our goal was to create a practical tool for evaluating the quality of observational studies that: (1) captures all of the domains critical for evaluating this type of research; (2) can be easily lifted ‘off the shelf’ by different researchers; and (3) can be adapted to different topic areas and observational study types (e.g., case control, cross-sectional, and case series studies). Methods: In Phase I of our study (completed), we developed an instrument based on domains identified by Deeks et al. (2003) which considered all relevant items used in earlier AHRQ-sponsored systematic evidence reports. We created a prototype instrument containing a parsimonious list of items within all relevant domains. We also included directions for Principal Investigators and abstractors for using and interpreting individual items. In Phase II (ongoing), we are validating and cognitively testing the instrument. Results: As described above, we developed an instrument that can be used for evaluating the quality of individual observational studies. In addition, this tool may be relevant for evaluating the risk of bias across studies when grading the strength of a body of evidence. Current followup study: In Phase II of our project we are: (1) meeting with experts to finalize the face validity of the instrument; (2) cognitively testing the instrument for readability; and (3) further validating the instrument through inter-rater reliability tests. We anticipate that we will present findings from the cognitive testing and validation study along with our tool in October 2008.