Background: Outcomes measured on ranking scales and continuous scales can be challenging for meta-analysts. We have previously reported a high rate of data extraction errors in meta-analyses using the standardised mean difference (SMD). Objectives: To study the interobserver variation when several independent observers extracted the same type of data and the reasons for disagreements. Methods: We selected a random sample of 10 recent Cochrane reviews that presented a result as an SMD and retrieved the trial reports (n = 45) that corresponded to the first SMD result in each review and the protocols for the reviews. Ten observers (five experienced methodologists and five PhD students) independently extracted the necessary data from the trial reports for calculation of the SMD. The observers did not have access to the original review and were only given the review protocols and the trial reports as pdf files. In the protocols, an additional researcher had highlighted the relevant outcome and other important factors. Based on the extracted data, this researcher calculated the SMDs. Agreement was defined as SMDs that differed less than 0.1 in their point estimates or confidence intervals. Results: The results were analysed at two levels, meta-analysis and trial level, pairing the observers in all possible ways (45 pairs). Thus, the 10 meta-analyses yielded a total of 450 pairs, and the 45 trials yielded 2025 pairs for agreement analysis. The agreement was 30.4% (137/450) at the meta-analysis level and 52.8% (1070/2025) at the trial level. Important reasons for disagreement were calculation errors, oversights, and differences in selection of time points, scales, control groups, and type of calculations. Conclusions: There was considerable disagreement between observers. Despite the limitations of this study due to its experimental design, it is clear that the multiplicity of data in trial reports calls for more detailed review protocols. The potential for error highlights the need for more than one observer and statistical knowledge or help from a statistician.