Article type
Year
Abstract
Background: Networkmeta-analysis relies on the agreement between direct and indirect evidence defined as consistency. Empirical evidence about the prevalence of inconsistency is limited to simple loops of evidence about three interventions.
Objectives: To evaluate empirically the prevalence of inconsistency in full complex networks and explore factors that might control its statistical detection.
Methods: We evaluated inconsistency in 40 published networkswith dichotomous outcomes (303 loops of evidence). We employed four different approaches: (1) loop-specific: we evaluated each loop in the network separately by contrasting direct and indirect estimates (2) separating direct and indirect evidence (SIDE): we evaluated the agreement between a particular comparison and the rest of the network (3) Lu and Ades model: we jointly assessed all possible inconsistencies in the network to obtain an omnibus test (4) Design-by-Treatment interaction model (DbT): we evaluated the agreement between estimates from different study designs in the network. In each approach we assessed the assumption of consistency using odds ratio, risk ratio, and risk difference, and we considered different estimators for the heterogeneity.
Results: The loop-specific method showed that inconsistency was prevalent in up to 10% of the tested loops. Loops including comparisons informed by a single study were more likely to show inconsistency. The SIDE method showed that about 10% of the tested comparisons were inconsistent. The Lu and Ades model yielded two (5%) inconsistent networks in total. The DbT model suggested inconsistency in 13% of the networks. Important heterogeneity was associated with a small decrease in statistical inconsistency. Different effect measures had no important impact on the detection of inconsistency.
Conclusions: Inconsistency can occur in 1 in 10 of the loops and in 1 in 8 networks. A sensitivity analysis for the estimation of heterogeneity might be needed before reaching a conclusion about consistency.
Objectives: To evaluate empirically the prevalence of inconsistency in full complex networks and explore factors that might control its statistical detection.
Methods: We evaluated inconsistency in 40 published networkswith dichotomous outcomes (303 loops of evidence). We employed four different approaches: (1) loop-specific: we evaluated each loop in the network separately by contrasting direct and indirect estimates (2) separating direct and indirect evidence (SIDE): we evaluated the agreement between a particular comparison and the rest of the network (3) Lu and Ades model: we jointly assessed all possible inconsistencies in the network to obtain an omnibus test (4) Design-by-Treatment interaction model (DbT): we evaluated the agreement between estimates from different study designs in the network. In each approach we assessed the assumption of consistency using odds ratio, risk ratio, and risk difference, and we considered different estimators for the heterogeneity.
Results: The loop-specific method showed that inconsistency was prevalent in up to 10% of the tested loops. Loops including comparisons informed by a single study were more likely to show inconsistency. The SIDE method showed that about 10% of the tested comparisons were inconsistent. The Lu and Ades model yielded two (5%) inconsistent networks in total. The DbT model suggested inconsistency in 13% of the networks. Important heterogeneity was associated with a small decrease in statistical inconsistency. Different effect measures had no important impact on the detection of inconsistency.
Conclusions: Inconsistency can occur in 1 in 10 of the loops and in 1 in 8 networks. A sensitivity analysis for the estimation of heterogeneity might be needed before reaching a conclusion about consistency.