Background: Randomized trials (RCTs) cannot answer numerous important policy questions in the social sciences-experiments may be politically or ethically impossible, RCTs may not identify the effect of interest (such as variation in response), or a decision may be needed before (lengthy) RCTs can be conducted. Yet almost no work has been done to develop quality guidelines for observational empirical work. Unfortunately, as such work becomes easier to conduct, more and more contradictory claims abound, and separating the wheat from the chaff becomes harder and more critical. Objectives: The goal of this presentation is to examine why the social sciences have not developed effective evidence-based (EB) quality guidelines and systematic reviews for observational work and to lay out how they should begin to rectify this problem. Argument and conclusions: I argue that the lack of guidelines in the social sciences is partly due to a misguided philosophy of science in the social sciences, but also partly due to the fact that the EB policy movement has (often intentionally) ignored the non-experimental, observational research that social scientists must use. Developing substantive-as opposed to reporting-guidelines, however, is challenging. To demonstrate the difficulties analysts face, I develop prototype quality guidelines for one methodological issue: controlling for simultaneity in criminological research. I show that the number of required criteria grows rapidly, and that there is little evidence about how to select between various methodological options. Given its methodological complexity and sensitivity to small errors, observational research will require guidelines that are more complex than those used in experimental settings, but this argues for developing such guidelines. And guidelines will force researchers to carefully evaluate the empirical tradeoffs of various methodological options. Furthermore, systematic reviews of observational research will help policymakers better understand the collective implication of wide-spread, and often seeminglycontradictory, research programs.