Background: Meta-analysts use a wide variety of graphs in assessments of heterogeneity and publication bias. To date, however, there have been few comparative studies of the performance of these graphs. Objectives: Our objective was to assess the reproducibility and validity of meta-analysis graph ratings. Methods: We simulated 100 meta-analyses from four scenarios that covered situations with and without heterogeneity and publication bias. From each meta-analysis we produced 11 graphs (box plot, weighted box plot, standardized residual histogram, normal quantile plot, forest plot, three funnel plots, trim and fill plot, Galbraith plot, and L’Abbeplot) and assessed the resulting 1100 graphs with three reviewers. We used intra-class correlation coefficients to assess between-rater reproducibility and assessed the validity of graph ratings by means of regression analyses with heterogeneity and publication bias parameters as dependent variables. Results: The intra-class correlation coefficients for reproducibility of the graph ratings ranged from poor (0.34) to high (0.91). Ratings of the forest plot and the standardized residual histogram were best associated with heterogeneity. Association between plot ratings and publication bias (censorship of studies) was poor, with the weighted box plot and the trim and fill plot showing the best combination of reproducibility and validity. Conclusions: Although pictures may say more than a thousand words, meta-analysts should be selective in the graphs they choose for the exploration of their data.