Article type
Abstract
Background: Cochrane recommends several methods for synthesizing data when meta-analysis is not appropriate, including vote counting based on direction of effect (VC-DE). There is lack of standardization for how to apply VC-DE to observational data that include multiple relevant results per outcome.
Objective: To assess how different adaptations of VC-DE for systematic reviews of observational data can affect conclusions about exposure/outcome pairs when multiple results for an outcome are reported.
Methods: A systematic review (n = 40 articles) assessing relationships between a prenatal dietary exposure (PDE) and neurocognitive development in the child included n = 436 results. Results were coded based on direction of the point estimate in relation to improved child neurodevelopment: "favored lower PDE," "favored higher PDE," or "neither." VC-DE approaches included the percent of results in each category based on total number of results per outcome [results method (RM)] or an aggregate result per article based on the percent of individual results [aggregate method (AM)] followed by the application of a threshold (>50% or >70%). With RM, the number of results per article influences the weight of each study, whereas with AM, each study is weighted equally.
Results: With RM for cognitive development (n = 156 results), as an example, 51% of results "favored higher PDE," 31% "favored lower PDE," and 18% were "neither." With AM, as the threshold increases, the percent favoring PDE in either direction decreases while the percent favoring neither increases. Using AM and cognitive development (n = 21 results), with a 50% threshold, 57% of the results favored higher PDE, 23% favored lower PDE, and 20% favored neither, which may lead to a conclusion favoring higher PDE. However, at the 70% threshold, the percent favoring higher PDE dropped to 43%, the percent favoring lower PDE dropped to 14%, and the percent favoring neither increased to 43%, indicating uncertainty in the findings.
Conclusions: Analytical decisions for adapting VC-DE for observational data can meaningfully influence conclusions about exposure/outcome pairs, emphasizing the need for transparent methods reporting. Guidance for VC-DE is needed for observational study designs that challenge conventional synthesis methods.
Objective: To assess how different adaptations of VC-DE for systematic reviews of observational data can affect conclusions about exposure/outcome pairs when multiple results for an outcome are reported.
Methods: A systematic review (n = 40 articles) assessing relationships between a prenatal dietary exposure (PDE) and neurocognitive development in the child included n = 436 results. Results were coded based on direction of the point estimate in relation to improved child neurodevelopment: "favored lower PDE," "favored higher PDE," or "neither." VC-DE approaches included the percent of results in each category based on total number of results per outcome [results method (RM)] or an aggregate result per article based on the percent of individual results [aggregate method (AM)] followed by the application of a threshold (>50% or >70%). With RM, the number of results per article influences the weight of each study, whereas with AM, each study is weighted equally.
Results: With RM for cognitive development (n = 156 results), as an example, 51% of results "favored higher PDE," 31% "favored lower PDE," and 18% were "neither." With AM, as the threshold increases, the percent favoring PDE in either direction decreases while the percent favoring neither increases. Using AM and cognitive development (n = 21 results), with a 50% threshold, 57% of the results favored higher PDE, 23% favored lower PDE, and 20% favored neither, which may lead to a conclusion favoring higher PDE. However, at the 70% threshold, the percent favoring higher PDE dropped to 43%, the percent favoring lower PDE dropped to 14%, and the percent favoring neither increased to 43%, indicating uncertainty in the findings.
Conclusions: Analytical decisions for adapting VC-DE for observational data can meaningfully influence conclusions about exposure/outcome pairs, emphasizing the need for transparent methods reporting. Guidance for VC-DE is needed for observational study designs that challenge conventional synthesis methods.