Background: Systematic review methodology pioneered in health care has been applied increasingly to questions of importance to lower- and middle-income countries. The development industry itself is also engaging with the evidence-based approach with initiatives such as the Abdul Latif Jameel Poverty Action Lab (J-PAL) and the International Initiative for Impact Evaluations (3ie). However, the evidence-based approach does not necessarily translate smoothly to the field of development. Systematic review methodology is based on the gold standard of randomised controlled trials. This gold standard is now widely accepted in health care, but such a narrow approach to evidence is not so easily accepted in a field where trials are often lacking, solutions urgently required, and regional and local variations common.

Methods: In the last twelve months, the UK Department for International Development (DFID) has undertaken a pilot, commissioning academics to undertake systematic reviews in development. We undertook one such review and in doing so made a number of pragmatic methodological compromises. Having completed and disseminated our review, we have now reflected on these compromises to assess what might be learnt for future reviews.

Results: We have identified lessons for future commissioners and researchers employing systematic review methodology in development. These relate to multi-disciplinary teamwork, application of regional filters, drawing on evidence from a range of study types, and exploring mechanisms for change through the development and testing of causal pathways.

Conclusion: There are still unanswered questions for reviewers in the field of development, particularly surrounding questions of study design, generalizability and applicability. Whilst adhering to the highest standards of systematic review methodology may not be possible (or supported by all), through transparent and complete reporting, we believe these discussions can advance and a balance between purity and pragmatism achieved.