Background: As guidelines are increasingly used in Korea, there is also increasing concern whether guidelines are developed by evidence-based methods. Objectives: The objectives of this study were to systematically examine guidelines published by medical specialty societies in order to determine to what degree they are based on scientific evidence and using methodological standards for developing guidelines. Methods: Guidelines produced by medical specialty societies in Korea published between 1998 and 2006 were identified through database search and mails. Each guideline’s quality and validity was assessed in terms of the strategy to identify primary evidence, literature selection, evaluation of evidence, data synthesis, and links between the strength of evidence and the recommendation. Results: A total of 54 guidelines were assessed by two independent appraisers per guideline using established criteria. Only small numbers of guidelines met the criteria: 29.6% of the guidelines report information on searchers for published studies; 16.7% give information on the evaluation of evidence or data synthesis; 9.3% report any explicit grading of the strength of recommendations. There was improvement over time in adherence to standards on identification and summary of evidence 18.6% prior to 2000 to 33% after 2000. Conclusions: The guidelines developed by specialty societies are not developed by evidence-based methods. Professional organizations or specialty societies that aim to develop guidelines should adopt explicit methodological criteria including identification, evaluation and synthesis of scientific evidences.