Background: Several measures for assessing statistical heterogeneity are available. Similar to high degree statistical heterogeneity the presence of considerable clinical heterogeneity (also called clinical diversity) precludes combining of study results by means of metaanalysis. Objective: To assess the inter-rater agreement regarding clinical heterogeneity. Methods: Five published meta-analyses that investigated cardiovascular endpoint in type 2 diabetes provided data on several RCTs. From these, seven case scenarios (each consisting of five to seven studies) were presented containing various patient characteristics (e.g. age, diabetes duration, cardiovascular diseases, blood pressure) and eligibility criteria in tabulated form. Based on these data, eight raters (from four institutions) with experience in clinical diabetes treatment, conducting systematic reviews in diabetes or both (most common) decided whether the extent of clinical heterogeneity precluded meta-analysis. Free marginal multi-rater kappa was calculated overall and for each of four distinct outcomes: mortality, cardiovascular events, health-related quality of life and glycosylated haemoglobin A1c (HbA1c). Results: The overall kappa was 0.29 (95%-confidence interval [95%-CI] 0.13 to 0.45). For the three patient-relevant outcomes mortality, cardiovascular events and health related quality of life, the results were−0.01 (95%-CI−0.07 to 0.05), −0.08 (95%-CI −0.13 to 0) and 0.44 (95%-CI 0.20 to 0.66), respectively. Regarding HbA1c a kappa of 0.81 (95%-CI 0.56 to 1) was found. Conclusions: With the exemption of HbA1c, the observed inter-rater agreement was low, much lower than we expected. For the endpoints mortality and cardiovascular events, kappa was about zero indicating agreement in the range of chance only. There is a definite need to further explore what constitutes clinical heterogeneity, ways to operationalise it and to determine possible implications for pooling effect estimates.