Estimating the contribution of studies in network meta-analysis: paths, flows and streams

In network meta-analysis, it is important to assess the influence of the limitations or other characteristics of individual studies on the estimates obtained from the network. The proportion contribution matrix, which shows how much each direct treatment effect contributes to each treatment effect estimate from network meta-analysis, is crucial in this context. We use ideas from graph theory to derive the proportion that is contributed by each direct treatment effect. We start with the ‘projection’ matrix in a two-step network meta-analysis model, called the H matrix, which is analogous to the hat matrix in a linear regression model. We develop a method to translate H entries to proportion contributions based on the observation that the rows of H can be interpreted as flow networks, where a stream is defined as the composition of a path and its associated flow. We present an algorithm that identifies the flow of evidence in each path and decomposes it into direct comparisons. To illustrate the methodology, we use two published networks of interventions. The first compares no treatment, quinolone antibiotics, non-quinolone antibiotics and antiseptics for underlying eardrum perforations and the second compares 14 antimanic drugs. We believe that this approach is a useful and novel addition to network meta-analysis methodology, which allows the consistent derivation of the proportion contributions of direct evidence from individual studies to network treatment effects.


Introduction
Decision making around multiple alternative healthcare interventions is increasingly based on meta-analyses of a network of relevant studies, which contribute direct and indirect evidence to different treatment comparisons 1,2 . Limitations in the design and flaws in the conduct of studies synthesized in network meta-analysis (NMA) reduce the confidence in the results: a treatment comparison in the network may be directly or indirectly informed by studies at high risk of bias. A relative treatment effect from NMA (hereafter the NMA effect estimate) is estimated as a linear combination of the available direct estimates of the treatment effect (i.e. the results from pairwise meta-analyses) and the indirect evidence on the treatment effect.
Salanti et al. suggested that in order to assess the impact of study deficiencies on an NMA effect estimate, the limitations of studies contributing to direct estimates should be considered jointly, taking into account their relative contribution to the overall NMA effect estimate 3 . The proportion contribution matrix plays a key role in this approach: a matrix that shows how much each direct effect contributes to the estimation of the NMA effect.
The proportion contribution matrix is derived from the absolute contribution matrix. The absolute contributions of direct effects to an NMA effect is the projection matrix from a two-step NMA model 4,5 . In the first stage, all direct effects are derived from pairwise meta-analyses. In the second stage, the NMA effect estimates are produced as a linear combination of the derived direct effects. The respective projection matrix is called the H matrix and it is analogous to the hat matrix in a linear regression model. The elements in the H matrix can be viewed as generalized weights from pairwise meta-analysis, but they do not add up to 1 and depend on the precision of the available studies, the degree of between-study heterogeneity and the network structure.
To translate the entries of the H matrix into proportion contributions, Salanti et al. suggested normalizing the absolute entries of each row of H and interpret them as proportions 3 . However, H represents the flow of evidence in different paths; the weight of each path is assigned to each direct effect involved. Thus, ignoring the multiple occurrences of the same values by taking standardized absolute values is incorrect. In particular, such a process overestimates the contribution of comparisons involved in long paths and underestimates the weights of the shortest paths. In this paper, we address this issue and present a method that properly translates the entries of the H matrix into proportions. The methodology is based on the observation that the rows of the H matrix can be interpreted as flow networks 4,6 .

Motivating example
To illustrate the ideas presented in this paper, we will use a network of topical antibiotics for the treatment of chronic otitis media with ear discharge in patients with eardrum perforations 7 . This network was used in Salanti et al. 3 and compares no treatment (x), quinolone antibiotic treatment (y), non-quinolone antibiotic treatment (u) and antiseptic treatment (v) 7 . The study outcome was the proportion of patients with persistent discharge from the ear after 1 week, measured using the odds ratio (OR). The network plot shown in Figure 1a shows that direct evidence exists for all comparisons except u versus x (non-quinolone antibiotic versus no treatment).
In order to assess the confidence that should be placed in an NMA effect estimate, Salanti et al. suggested considering the quality of all pieces of evidence that contributed to it 3 . For example, the studies directly comparing 'u versus v' were judged to be at high risk of bias; however, in order to judge the quality of the NMA effect estimate of 'u versus v', we need to consider the amount of data that these studies contributed to its estimation.

Methods
We first present the random-effects two-stage NMA model first described by Lu et al. 5 . We will employ a simplified version of the H matrix described by König et al. 4 that does not take into account the correlation induced by multi-arm trials. We ignore this correlation for the sake of ease of interpretation of the entries in the H matrix; we discuss implications of multi-arm trials at the end of the Methods section. Taking advantage of previous findings on how the flow of evidence can be considered in NMA 4,6 , we present an algorithm to decompose the flow in a network and subsequently approximate the proportion contributions of direct effect estimates for each NMA effect estimate.
Two-stage network meta-analysis model Consider a network of T competing treatments. The set of treatments is denoted by V = {x, y, u, v, ...} and let x denote the reference treatment. The number of NMA effects to be estimated is ( ) 2 T but the estimation of T -1 effects allows the derivation of the remaining effects via linear combination. We collect the T -1 effects against the reference treatment x in a vector of basic parameters θ = (θ xy ,θ xu ,θ xv , ...)′. In the case of a dichotomous outcome, θ is the parameter vector of all log-ORs compared to the common reference treatment x.
We assume that the distribution of effect modifiers is similar across comparisons and thus the transitivity assumption is plausible. The consistency assumption refers to the statistical manifestation of transitivity and implies that all sources of evidence are in agreement; this is expressed via the consistency equations

Amendments from Version 2
The revised version of this manuscript includes suggestions by Drs. Annette M. O'Connor and John R. Thomson. In particular: 1. We changed 'percentages' to 'proportions' throughout.
Thus, Figure 1 has also been revised.
2. We clarify in which situations our algorithm can be used in sections 'Proportion contributions of direct comparisons' and 'Algorithm to decompose flows into proportion contributions'.  where H is Matrix X is a D × (T -1) design matrix expressing the linear relationships between the available direct effects and the basic parameters and Y is a ( ) 2 T × (T -1) design matrix that links the NMA estimates with the basic parameters. Note that X is identical to Y only when there are direct studies for all treatment comparisons in the network.
Matrix H is of dimensions ( ) 2 T × D and describes the influence of each direct effect (specified in the column) to an NMA effect (specified in the row). As Equation 2 implies, H is derived as a function of the variances of the direct effects θ D and the network structure; therefore, the exact (absolute) contribution of each direct comparison depends on the precision of the available direct data and the comparison's connectivity to the rest of the network. Note that it resembles the hat matrix in a linear regression model. To explain the method, we will continue focusing on one row of the H matrix, say h xy , corresponding to the 'x versus y' comparison. Box 1 includes the definitions, along with the notation, of some of the notions used in this paper.

Set of vertices
The set of vertices is defined as the set of treatments examined in the network, V = {x, y, u, v ...}.

Set of directed edges
The set of directed edges E is defined as the set of direct comparisons respecting the signs of the entries of h xy , the row of the H matrix corresponding to the 'x versus y ' comparison. Edges are given a direction upon the definition of flows; the network itself corresponds to an undirected graph.

Set of flows
The set of flows is defined as F = {f uv , ∀ uv ∈ E} where f uv is equal to |h uv |.

Comparison graph
A comparison graph is defined as a graph G xy = (V, E, F) constructed from a row of the H matrix, h xy ; its definition derives from a set of vertices V, a set of edges E, and a set of flows, F.

Source
Source is defined as a vertex with no incoming edges.

Sink
Sink is defined as a vertex with no outgoing edges.

Path
A path π i is defined as a sequence of connected directed edges belonging to E.

Stream
A stream S i is defined as the composition of a path and its associated flow, S i = (φ i , π i ). with i = 1, ... , I where I is the total number of streams.

Comparison graph
König et al. showed that every row of the H matrix, h xy , can be interpreted as a flow network with source x and sink y, and visualised in a directed acyclic graph (DAG) 4 . Thus, we create a graph G xy = (V, E, F) from h xy ; its definition derives from a set of vertices V, a set of edges E, and a set of flows, F. The set of vertices is defined as the set of treatments examined in the network, V = {x, y, u, v ...}. Set E is defined as a set of directed edges that correspond to observed direct comparisons respecting the signs of the entries of h xy . To simplify the notation, we drop from now on all superscripts assuming they all refer to xy. Then, the set E contains uv if h uv > 0 or contains vu if h uv < 0.
The set of flows is defined as F = {f uv , ∀ uv ∈ E} where f uv is equal to |h uv |.
The following conditions hold for the elements of set F (see Supplementary File 1 for proof): a. The sum of outflows of node x (source) is 1 The flow passing through each internal node (any node except x or y) is conserved there is no path (sequence of edges) that visits the same vertex twice.
Consider, for example, the graph G xy in Figure 1b, which corresponds to the xy comparison of the network of four treatments of Figure 1a. The set of vertices is V = {x, y, u, v} and the set of directed edges is E = {xy, xv, vy, vu, uy}. Flows f uv are given along the edges; their numerical values are equal to the respective absolute entries of h xy and the direction of their corresponding edge is indicated in the subscript. As properties (a) to (d) imply, the arrows in Figure 1b indicate that the outflows of x, as well as the inflows of y, equal 1, and that the inflows equal the outflows in the intermediate nodes u and v.

Streams
In Figure 1b there are three different paths from x to y, one based on direct evidence, {xy}, and two based on indirect evidence, {xv, vy} and {xv, vu, uy}. A path is a sequence of connected directed edges belonging to E, and we denote it as π i . As property (d) implies, each node occurs at most once in π i . Then, given the above properties of f uv , we can assign a flow φ i to each path π i . Flow φ i is equal to the smallest f uv in the path π i . Figure 1c shows the three paths from x to y; π 1 , π 2 and π 3 , and their corresponding flows. Path π 1 corresponds to xy and its flow, φ 1 , equals the flow of the single edge in path, f xy = 0.635. Path π 2 is constituted from two edges, xv and vy; thus, flow φ 2 = min(f xv , f vy ) = 0.251. The flow corresponding to the third path π 3 is φ 3 = min(f xv , f vu , f uy ) = 0.114.
We define a stream, S i , as the composition of a path and its associated flow, S i = (φ i , π i ) with i = 1, ..., I where I is the total number of streams; here I = 3. Note that it holds

Proportion contributions of direct comparisons
In order to assign proportion contributions to each direct comparison, we need to split each stream's flow to the involved edges in the stream's path. Equation 3 can be re-written as

Equation 5
with 3 1 1 i i= ϕ = ∑ . It follows from the properties of the elements of set F, that each NMA effect can be written as a linear combination of direct and indirect effects, in the form of Equation 5. The effects are stochastically interdependent and, hence, their aggregation is different from the aggregation of studies in a pairwise meta-analysis.
To approximate the proportion contributions per comparison, we suggest dividing φ i by the length of the respective path π i , #π i . This will leave the proportion contribution of the direct evidence of the same treatment comparison equal to the diagonal of the H matrix and assign to each comparison involved in an indirect route a portion of the respective stream's flow. Note that directed edges might be involved in more than one path; we thus define the proportion contribution of an edge uv as θ θ θ θ θ to ^N xy θ , we approximated their proportion contribution as 63.5%, 16.4%, 3.8%, 12.6% and 3.8%, respectively.

Algorithm to decompose flows into proportion contributions
In this section, we present an iterative algorithm that generalizes the process outlined above to derive proportion contributions of each direct effect to the estimation of a 'x versus y' NMA effect. We start by defining a graph G xy from h xy .
The algorithm is described as follows: Then, repeat the process below I times, equal to the number of streams in G i , until E i = {Ø}.
1. In G i-1 , find the shortest path from x to y, π i , and define its flow as φ i = min{f i-1,uv , uv ∈ π i }. Then, use π i and φ i to define the stream S i = (φ i , π i ).
2. Recalculate the flow of edges uv ∈ π i by subtracting φ i from the flow of the edges of the stream found: The flow of the rest of the edges that do not belong to π i remain unchanged: 3. Define E i as the set of edges uv for which f i,uv > 0; this is E i-1 after removing the edges with zero flow, and go to step 1.
When the algorithm terminates, all streams S i = (φ i , π i ) have been identified and Equation 6 is used to derive the proportion contributions p uv .
Repeating the same process for all NMA effects, we derive all xy uv p and collect them in a matrix P of the same dimensions as H. The presented algorithm could be described as a reverse maximum flow Edmonds Karp algorithm 8 , but instead of adding we remove augmenting paths.
It is possible that multiple shortest paths exist; in this case, the order in which one chooses such a path could in principle result in different proportion contributions per comparison. We can, thus, use the following modification in the algorithm to impose consistency. Instead of selecting the shortest path, we assign cost values c i,uv to each edge uv as follows: c i,uv = 2f i-1,uv . Then, we select the path from x to y with the minimum cost across comparisons included in π i . The definition of the cost values c i,uv assures that paths are selected from shortest to longest and removes any ambiguity regarding the selection of paths.
The starting point for the developed algorithm was a simplified version of the H matrix that does not consider the correlation induced by multi-arm trials. Alternatively, one could use the H matrix as described by König et al. 4 that extends the definition of the matrix for multi-arm designs. Note that any matrix whose rows can be interpreted as flow networks can be used as the starting point of the algorithm. The estimator of heterogeneity as well as the assumption of a different or common heterogeneity across comparisons in the network does not modify any aspect of the method.
Calculations in this paper were performed using R.

Application
We apply the algorithm described above to the network of topical antibiotics 7 .
Proportion contributions of direct relative treatment effects to the estimation of the NMA effect between non-quinolone antibiotic and no treatment Direct effects are obtained using the random effects model and the H matrix of dimension 6×5 is calculated using Equation 2. The H matrix, along with NMA effects, is given in Table 1.
We begin by applying step 0 of the algorithm. We construct the network G 0 = G xy = (V, E 0 , F 0 ) with source x and sink y corresponding to row h xy . The set of vertices is V = {x, y, u, v} and the set of directed edges, taking into account the signs of the elements of h xy , is E 0 = {xy, xv, vy, vu, uy}. The set of flows is F 0 = {f 0,uv | uv ∈ E 0 }, where f 0,uv equal the respective absolute values of Table 1 and are given along the edges of Figure 1b.
Then, we apply the developed iterative algorithm until E i = {Ø}. The iterations of the algorithm equal the number of existing streams from x to y and are illustrated in Figure 2.
3. Define E 1 as the set of edges uv for which f 1,uv > 0; edge xy is removed since its flow is zero, E 1 = E 0 \{xy}. Collect f 1,uv to form the set F 1 = {f 1,uv | uv ∈ E 1 } ( Figure 2c).
3. Define E 3 as the set of edges uv for which f 3,uv > 0; edges xv, vu and uy are removed since their flow is zero, The set of direct edges is E 3 = {Ø} and the algorithm is terminated at this point ( Figure 2i). Figure 1c shows the flows of the three streams identified when applying the above algorithm. We then calculate the proportion contributions of each comparison to the 'x versus y' NMA treatment effect estimate using Equation 6 ( Figure 1d). For instance, to calculate p xv we first have to identify the relevant paths; these were π 2 and π 3 . Consequently, The calculations for deriving the proportion contributions of the other comparisons are shown in Table 2. Applying the algorithm to all NMA treatment effect estimates we get the entire proportion contribution matrix P.

Proportion study contributions
Matrix P ( to the xy NMA treatment effect estimate. The application of this process to the entire matrix P leads to the matrix P* shown in Table 4. Adjusted weights as proposed by Rücker & Schwarzer 9 are used for multi-arm studies.
Using proportion study contributions to quantify the impact of a characteristic in a direct comparison The algorithm translating the H matrix into study proportion contributions can be applied to quantify the influence that a study-level characteristic has in the estimation of the x, no treatment; y, quinolone antibiotic; u, non-quinolone antibiotic; v, antiseptic.   Figure 3 shows such a bar plot using the algorithm described in this paper and distributing comparison proportion contributions to study proportion contributions; inspecting Figure 3 can support judgements regarding the importance of study limitations for different NMA treatment effect estimates. For instance, studies with high risk of bias contribute more than 50% in the estimation of the 'u versus v' comparison, potentially reducing the confidence that we can place in this particular NMA treatment effect estimate.
Proportion contributions of direct comparisons in a large complex network of interventions So far, we have illustrated how to derive proportion contributions for a network with four treatments. However, the algorithm can be straightforwardly applied to large networks of any structure, as soon as the involved treatments are connected. Consider for example a large network examining antimanic drugs (Figure 4) 10 . Let us concentrate on the comparison PLA versus  OLA ('placebo versus olanzapine'); the algorithm starts by applying step 0 and constructing network G 0 . Then, we continue by finding the shortest path in the first iteration, which corresponds to the direct comparison, and define its flow and stream S 1 = (φ 1 , π 1 ). The number of algorithm's iterations is equal to the number of streams from placebo to olanzapine, which turns out to be 16. The resulting entire proportion matrix is given in Supplementary File 2. Data labels: study, name of individual studies; id, id of the individual studies; t; treatment; r, number of events; n, sample size; rob, risk of bias per study.

Discussion
In this paper, we present a new approach to derive proportion contributions of individual studies to the treatment effect estimates in NMA. We made use of the fact that the composition of network treatment effect estimates can be interpreted as a flow of evidence. An assumption that underlies our algorithm is the equal split of the stream flow to the involved comparisons. Although indirect effects are not weighted averages, we find this approximation to be a pragmatic approach that reasonably reflects the amount that each comparison contributes to network effects. The derivation of the contribution of sources of evidence in a Bayesian NMA is not straightforward, as no analogue to the H matrix exists. The application of the method described in this paper, however, can be used to derive useful approximations of the contributions of studies.
Applying the algorithm to networks of interventions can be used to quantify the contribution of potential study limitations to the NMA treatment effect estimates. Study limitations may lead to biased NMA treatment effect; however, the amount and direction of bias in the NMA treatment effect as a result of the within-study bias is not straightforward to define and is not currently accommodated within the proportion contribution matrix. First, a single biased trial may affect an entire indirect route; thus, even if its proportion contribution is small, its consequences in the estimation of the NMA treatment effect may be important. Second, the direction of bias across studies involved in a stream may vary. For example, bias in two comparisons in the same stream may either cancel out or add-up in favor of one of the two treatments. We aim to extend the methods presented in this paper to develop a network meta-regression model that will use the direction and the amount of bias to determine whether and how much NMA treatment effect estimates will be biased as the result of within-study bias.
Alternative methods to derive the relative contribution of all sources of evidence have been developed 12,13 . Krahn et al. define influence functions to describe the extent to which changes in study effects would be translated into NMA treatment effects 13 . An alternative approach, based on the decomposition of Fisher's information matrix, has been proposed to derive proportion study weights in a variety of meta-analysis models, including meta-regression, network meta-analysis and individual patient data meta-analysis 12 . Further investigation of the degree of agreement between our algorithm and that of Riley et al. 12 would be of interest.
In the example implemented in the Application, there is no other possible set of paths, and associated streams, that could be selected from x to y in order to partition the inflow of x: π 1 , π 2 and π 3 is the only possible set of streams ( Figure 1c). Thus, even if we were taking paths using different criteria, i.e. from longest to shortest, according to values from the H matrix or even randomly, the proportion contributions given in Table 2 would be identical. However, cases exist where the selection of paths does influence the derivation of the P matrix. In Supplementary File 3, we elaborate on the selection of direct paths in the algorithm and discuss some alternative modifications of the algorithm. We are planning to examine the properties of the different approaches in greater detail in a follow up project.
We offer an R package 14 , which we also use in the software application CINeMA (Confidence In Network Meta-Analysis) 11 , that aims to simplify the evaluation of confidence in the findings from NMA. While CINeMA largely follows the framework previously developed by Salanti et al. 3 , the refinement of several methodological aspects is currently under development. Core aspects of the approach include the consideration of the relative contributions of each direct comparison to each NMA treatment effect estimate. To this end, CINeMA uses the proportion contribution matrix as described in this paper. The command netweight in Stata has also been updated to use the described approach.
We believe that the approach described in this paper is a useful and novel addition to network meta-analysis methodology, which allows the consistent derivation of the proportion contributions of direct evidence from individual studies to network treatment effects.

Data availability
Dataset 1: Outcome data from the example network of topical antibiotics for the treatment of chronic otitis media with ear discharge in patients with eardrum perforations 7 . The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Supplementary material
Supplementary File 1. Proof that the sum of all outflows from the source is equal to the sum of all inflows to the sink and both are 1.
Click here to access the data.
In criticising the terminology, I have to accept that the authors are largely following common practice and so my criticisms are partly aimed at the meta-analysis community. None the less there were three things that irritated me. Firstly, throughout the paper, proportions are referred to as percentages. Next, the title of the paper says it is about "the contribution of studies in network meta-analysis" while actually it is about the contribution of different effect estimates. Finally, the measure adopted is the weight given to each effect estimate when calculating the pooled estimate. In this paper and elsewhere in the meta-analysis literature, this weight is called a contribution. Perhaps I am being too pedantic but the weight and contribution are different ideas and it does not help with the terminology confuses them.
Now the proposed method. Taking the example from the paper, the authors note that the pooled estimate of the xy effect can be calculated using , where is the indirect estimate of xy along the path equation 1 θ xy. Further it is true, at least for this example, that Σ = 1 where L is the length of the path. So we can L attach the weights to the edges of the network, sum them when an edge contributes to more than one φ estimate and the resulting weights will sum to one over the whole network.
This argument works for the example presented in the paper but it is not clear to me what conditions have to hold for it to work generally. The authors note in the paper that including a multi-arm trial in the meta-analysis would cause a problem, presumably because some of the direct or indirect estimates would not be independent. Are there any other conditions that have to hold? For example, can we have any structure of random effects in the meta-analysis model? What about the Bayesian models that are I xvy iφi i any structure of random effects in the meta-analysis model? What about the Bayesian models that are often used for network meta-analysis?
The authors take the matrix, H, which projects individual estimates such as xy, xy, uy, etc. into their predicted values under the meta-analysis model and they present an algorithm for converting those values into weights that are equivalent to the 's.The algorithm is sensible and works for the simple φ example in the paper but one is again left wondering whether or not it works under all circumstances. After all, the algorithm is presented without any proof that it works.
My own feeling is that a contribution is best measured by the sledgehammer approach of analysing the network with and without a particular edge, but the authors' suggested approach is much less computationally demanding and I think that it would be appreciated by many applied researchers provided they were certain that it could be safely used with their particular network and their particular meta-analysis model.

Are sufficient details provided to allow replication of the method development and its use by others? Yes
If any results are presented, are all the source data underlying the results available to ensure full reproducibility? Partly

Are the conclusions about the method and its performance adequately supported by the findings presented in the article? Partly
No competing interests were disclosed. Competing Interests:

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
Author Response 05 Dec 2018 , University of Bern, Bern, Switzerland Adriani Nikolakopoulou We are grateful to Dr. John R. Thompson for his time and valuable comments and for recommending approval of our paper. We believe that we have been able to address all the comments raised and we have prepared a next version (Version 3) of the paper. Below are the detailed replies to reviewer's comments; our responses to the first reviewer, Dr. Jochem König, can be found in I like the idea of associating a descriptive measure of importance to each edge in a network and the authors of this paper present an interesting step in that direction, though my feeling is that this proposal will not turn out to be the final solution. I have two areas of concern, one is to do with terminology and the other with the scope of applicability of the proposal solution.
In criticising the terminology, I have to accept that the authors are largely following common practice and so my criticisms are partly aimed at the meta-analysis community. None the less there were three things that irritated me. Firstly, throughout the paper, proportions are referred to as percentages. Next, the title of the paper says it is about "the contribution of studies in network meta-analysis" while actually it is about the contribution of different effect estimates. Finally, the measure adopted is the weight given to each effect estimate when calculating the pooled estimate. In this paper and elsewhere in the meta-analysis literature, this weight is called a contribution. Perhaps I am being too pedantic but the weight and contribution are different ideas and it does not help with the terminology confuses them.
Thank you for these interesting notes on meta-analysis terminology. At first, we agree that referring to "percentage contributions" might not be fitting and calling them "proportions" is more appropriate. We, thus, replaced throughout percentages with proportions.
Regarding the reporting of contribution of studies in the title of our paper, we have extended the method to quantify the contribution of studies as suggested by the first reviewer, Dr. Jochem König (comment 6). Thus, we left the title of the paper unmodified.
Weights and contributions are indeed sometimes used interchangeably. In this paper, however, we believe that the definition of matrix and the set of flows render the distinction clear. We only talk about "weights" of paths in the Introduction, where we have not yet explained the terminology to be used and thus it is convenient describing the elements of matrix as generalized weights from pairwise meta-analysis. We apologise in advance if we have misunderstood your comment. Now the proposed method. Taking the example from the paper, the authors note that the pooled estimate of the xy effect can be calculated using , where is the equation 1 θ indirect estimate of xy along the path xy. Further it is true, at least for this example, that Σ L = 1 where L is the length of the path. So we can attach the weights to the edges of φ the network, sum them when an edge contributes to more than one estimate and the resulting weights will sum to one over the whole network.

I xvy iφi i because some of the direct or indirect estimates would not be independent. Are there any other conditions that have to hold? For example, can we have any structure of random effects in the meta-analysis model? What about the Bayesian models that are often used for network meta-analysis?
Thank you for this insightful comment. We have clarified at the beginning of the Methods section that we consider [1] "a simplified version of the matrix described by König et al. that does not take into account the correlation induced by multi-arm trials." In this case, and considering the properties of the elements of set it always holds that Equation 3 can be re-written as Equation 5. We now clarify in the subsection 'Proportion contributions of direct comparisons':

"It follows from the properties of the elements of set that each NMA effect can be written as a linear combination of direct and indirect effects, in the form of Equation 5".
We clarify at the end of 'Algorithm to decompose flows into percentage contributions' section that multi-arm studies can be handled if a modified matrix is used. We also clarify that the structure of random effects does not play a role in the calculations. In particular, we write: "The starting point for the developed algorithm was a simplified version of the matrix that does not consider the correlation induced by multi-arm trials. Alternatively, one could use the matrix as [1] described by König et al. that

extends the definition of the matrix for multi-arm designs. Note that any matrix whose rows can be interpreted as flow networks can be used as the starting point of the algorithm. The estimator of heterogeneity as well as the assumption of a different or common heterogeneity across comparisons in the network does not modify any aspect of the method."
Within a Bayesian framework, the matrix is not defined. We added in the Discussion: "The derivation of the contribution of sources of evidence in a Bayesian NMA is not straightforward, as no analogue to the matrix exists. The application of the method described in this paper, however, can be used to derive useful approximations of the contributions of studies." The authors take the matrix, H, which projects individual estimates such as xy, xy, uy, etc. into their predicted values under the meta-analysis model and they present an algorithm for converting those values into weights that are equivalent to the 's.The algorithm is φ sensible and works for the simple example in the paper but one is again left wondering whether or not it works under all circumstances. After all, the algorithm is presented without any proof that it works.
Thank you for pointing this out. The algorithm we present in the 'Algorithm to decompose flows into proportion contributions' is described for the general case of a network of interventions containing any number of studies, treatments and loops. Indeed, we do not present a proof that it works under all circumstances. We have, however, applied the algorithm in numerous NMA applications apart from the simple example used in the results of this paper. Moreover, as mentioned in the Discussion, the web application CINeMA (Confidence In Network Meta-Analysis) [2], that aims to simplify the evaluation of confidence in the findings from NMA, uses the proportion contribution matrix as described in this paper. Users of CINeMA have not reported any case that the algorithm does not work. We are planning to examine the properties of the different approaches on the selection of direct paths in the algorithm in a follow up project (preliminary work is described in Supplementary File 3), which could be combined with a proof that the algorithm (and its modifications) works.

modifications) works.
My own feeling is that a contribution is best measured by the sledgehammer approach of analysing the network with and without a particular edge, but the authors' suggested approach is much less computationally demanding and I think that it would be appreciated by many applied researchers provided they were certain that it could be safely used with their particular network and their particular meta-analysis model.
We totally agree that such 'leave one out' approaches would be also of interest. Relevant methods have been proposed by Krahn et al. [3] for contribution to the effect and by Riley et al. [4] for contribution to the variance. The former is useful in measuring how much a direct effect could bias the NMA effect and thus is not relevant in this case where we opted for a representation of contribution of pieces of evidence according to a study characteristic. The latter is based on the decomposition of Fisher's information matrix and is associated with some issues that need further investigation.
We had already mentioned and discussed the Riley et al. approach in the Discussion. We now added:

"Krahn et al. define influence functions to describe the extent to which changes in study effects
[3] would be translated into NMA treatment effects ." No competing interests were disclosed.

Competing Interests:
I have to admit that this is an example of why I appreciate the open review system. I enjoyed the paper and found it easy to read and follow, but the others reviewer's comments were a significant contribution also. I don't have any additional critique of the proposed approach, but I look forward to seeing how the investigators address those or provided some discussion, as this would help the less statistically inclined reader like me. Again, the reason I wanted this, is that I am an applied user and such discussions are very helpful. In particular, I would like to have seen some discussion of the 1st comment. I would also like to Concerning the approach to calculating the % contribution and how to weight the flows, I came away with the impression that the decision of equal split of the stream flow as arbitrary (perhaps not precisely the correct term) or perhaps pragmatic is better. Therefore, it is not surprising this is a debatable approach. Again, this was another reason why I am looking forward to see the investigators responses to the 1st reviewer's comments. I agree that the comparison of the methods of deriving the percentage contribution would be of interest but was not expecting that to be included. I look forward to this comparison being published.

Are sufficient details provided to allow replication of the method development and its use by others? Yes
If any results are presented, are all the source data underlying the results available to ensure full reproducibility? Yes

Are the conclusions about the method and its performance adequately supported by the findings presented in the article? Yes
No competing interests were disclosed.

Competing Interests:
Reviewer Expertise: Epidemiology, infectious diseases, research synthesis

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
Author Response 05 Dec 2018 , University of Bern, Bern, Switzerland

Adriani Nikolakopoulou
We are grateful to Dr. Annette M. O'Connor for her time and valuable comments and for recommending approval of our paper. We believe that we have been able to address all the comments raised and we have prepared a next version (Version 3) of the paper. Below are the detailed replies to reviewer's comments; our responses to the first reviewer, Dr. Jochem König, can be found in . https://f1000research.com/articles/7-610/v2 Referee Report: Annette M. O'Connor, Department of Veterinary Diagnostic and Production Animal Medicine, College of Veterinary Medicine, Iowa State University, Ames, IA, USA Approved Approved This paper described an updated approach to deriving the percentage contributions of the direct comparison used in treatment estimates. In the interests of full disclosure, I am not a statistician and was reviewing from the view point of an applied user of NMA. There are several papers on this topic in recent years, and it is an active body of work in this area. The article is straightforward to read, although made considerably more readable if one is aware of the content in references, 3,4 and 5.
I have to admit that this is an example of why I appreciate the open review system. I enjoyed the paper and found it easy to read and follow, but the others reviewer's comments were a significant contribution also. I don't have any additional critique of the proposed approach, but I look forward to seeing how the investigators address those or provided some discussion, as this would help the less statistically inclined reader like me. Again, the reason I wanted this, is that I am an applied user and such discussions are very helpful. In particular, I would like to have seen some discussion of the 1st comment. I would also like to see if the comment about Figure 3 could be incorporated -perhaps not feasible.
Thank you very much for your positive feedback. Our answers to the first reviewer, Dr. Jochem König, can be found in . In summary, regarding his first https://f1000research.com/articles/7-610/v2 comment, we distinguish between representation of contribution of pieces of evidence according to a study characteristic (addressed in this paper) and quantification of the impact of within-study bias in NMA treatment effects (subject of future research). A new paragraph in the Discussion highlighting these concerns reads: "Study limitations may lead to biased NMA treatment effect; however, the amount and direction of bias in the NMA treatment effect as a result of the within-study bias is not straightforward to define and is not currently accommodated within the proportion contribution matrix. First, a single biased trial may affect an entire indirect route; thus, even if its proportion contribution is small, its consequences in the estimation of the NMA treatment effect may be important. Second, the direction of bias across studies involved in a stream may vary. For example, bias in two comparisons in the same stream may either cancel out or add-up in favor of one of the two treatments. We aim to extend the methods presented in this paper to develop a network meta-regression model that will use the direction and the amount of bias to determine whether and how much NMA treatment effect estimates will be biased as the result of within-study bias." As we explain in detail in our response to Dr. Jochem König (comment 4), we did not amend Figure  3 to incorporate directionality.
Concerning the approach to calculating the % contribution and how to weight the flows, I came away with the impression that the decision of equal split of the stream flow as arbitrary (perhaps not precisely the correct term) or perhaps pragmatic is better. Therefore, it is not surprising this is a debatable approach. Again, this was another reason why I am looking forward to see the investigators responses to the 1st reviewer's comments. I agree that the comparison of the methods of deriving the percentage contribution would be of interest but was not expecting that to be included. I look forward to this comparison being published.
We agree that equal splitting of stream flow is to some extent arbitrary but also pragmatic. We say 1.
We agree that equal splitting of stream flow is to some extent arbitrary but also pragmatic. We say in the Discussion: "An assumption that underlies our algorithm is the equal split of the stream flow to the involved comparisons. Although indirect effects are not weighted averages, we find this approximation to be a pragmatic approach that reasonably reflects the amount that each comparison contributes to network effects." No competing interests were disclosed. Competing Interests:

6.
7. and three edges, respectively, each with flow 1/3. Then each direct comparison has the same influence on the network based treatment effect estimate: A bias of 1 unit in one direct estimate translates into a bias of 1/3 unit in the network based estimate. These proportions are conserved in Salanti's notion of percentage contribution, but not so in the newly introduced concept. Salanti [3] attaches a percentage contribution of 1/6 to each comparison just as if we had pooled six equally precise trials comparing the same pair of treatments. But the influence of direct comparisons is different in this network and should be characterized by the number 1/3.
That is why I have objections against Salanti's [3] concept of percentage contribution and even more so against the newly introduced concept, which attaches unequal contribution quantities of 1/9, 1/6 and 1/3, respectively.
The new method should not be introduced into routine of network meta-analysis. If this paper is going to be indexed, its draw backs and caveats should be very clearly set out.
The concept of streams is a nice one. It allows to represent a network based treatment effect as a weighted sum of a direct and indirect effect estimates each corresponding to one path from source to sink. Note however that these effects are stochastically interdependent and hence, the aggregation of streams is different from the usual process of pooling evidence.
To be somewhat less destructive, I propose to arrange the network graph with the source to the left and the sink to the right and all other vertices placed, such that all directed edges point from left to right. Then, due to the flow properties, any vertical cross-section gives a set of flows that sum to one. In that sense, the untransformed rows of the H matrix already contain percentage contributions. Displays used to present voter hiking can be used to represent these flows (see e.g. ). www.zeit.de/politik/deutschland/2017-05/waehlerwanderung-nrw-landtagswahl-cdu-spd-fdp I propose to modify Table 3 and Figure 3 in a way that the direction of influence becomes clear. Does a positive bias in one contributing direct effect translate into a positive or negative bias of the network based effect estimate? The colored fields in Figure 3 could be labelled with meaningfully ordered pairs of treatment letters.
Discussion "The calculation of the percentages uses the estimated variances of direct effects and thus incorporate associated uncertainty in their estimation." Note however, that in a linear network each direct effect estimate contributes equally, irrespective of the size and precision of the study.
Discussion "Applying the algorithm to networks of interventions can be used to quantify the contribution of potential study limitations to the NMA treatment effect estimates." The two stage approach discussed in this paper is not essential. Both the flows and the percentage contribution defined in the paper can be distributed to single trials according to the weights used when pooling them into direct effect estimates. Then the risk of bias of single trials can be analyzed. A display similar to Figure 3 is possible that contains separately colored fields for individual studies.
Discussion, 2 paragraph: Concerning multi-armed designs, König et al.[4] have defined the flow into and out of the vertices in a way that does not depend on the choice of reference treatment.

Supplementary File 3:
1st paragraph: What is the vector space of G over R? nd 7.
1st paragraph: What is the vector space of G over R? 2nd paragraph: 'The number of directed paths' obviously refers to the number of directed paths "which suffice to 'spend' the flow from source to sink".
The authors state that, if no bridges exist, this number equals to df+1. However, it depends on the flows, too. Consider a network composed of two squares connected at one vertex. It has df=2 inconsistency degrees of freedom and generally 3 paths are needed to spend the flow between the two most distant vertices. However, if all edges have equal flows, only two paths are needed. Moreover, the connecting vertex has the same effect as the bridging edges. Hence the argument has to be extended to the case of graphs that can be split by cutting through a vertex.
Note also, that a network estimate that compares treatments from different subgroups of treatments connected through a bridge or a 'breaking' vertex can be written as the sum of three respective two subnetwork effect estimates: one comparing the source treatment to the bridge post, one for the bridge, if present, and one comparing the second bridge post to the sink. Because it is an unweighted sum, the contribution of the three (or two) summands should be equal.

Are sufficient details provided to allow replication of the method development and its use by others? Yes
If any results are presented, are all the source data underlying the results available to ensure full reproducibility? Yes Are the conclusions about the method and its performance adequately supported by the findings presented in the article? Partly No competing interests were disclosed.

Competing Interests:
I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
Author Response 19 Aug 2018 , University of Bern, Bern, Switzerland Adriani Nikolakopoulou We are grateful to Dr. Jochem König for the time and effort he spent to review our paper. We believe that his valuable comments and suggestions have substantially improved the manuscript. We have addressed all his comments and revised the paper accordingly. Below you can find detailed replies to the reviewer's suggestions and comments. Approved with Reservations 1. The authors undertake a redefinition of the term 'percentage contribution' in network meta-analysis. In ordinary meta-analysis of randomized clinical trials, pooled estimates of a treatment effect can be represented as a weighted mean of treatment effects from all contributing trials. Weights are positive and sum to one and naturally constitute the 'percentage contribution' of each study. In network meta-analysis treatment effect estimates are defined for all pairs of treatments. Each can be represented as a two-stage estimate, firstly pooling all trials comparing the same set of treatments, respectively, which gives direct effect estimates, and then pooling direct estimates into a network based mixed treatment comparison. The linear coefficients, used in the second stage constitute the 'contribution of direct evidence to a network based comparison'. They can be read off, from the H matrix, as described by the authors. They share some but not all aspects with weights of a two-armed meta-analysis. Aspects shared in some sense are the flow properties described by the authors and in citation [4]. The aspect not shared, is that summation to one.
Indeed, in a purely linear graph connecting two extreme treatments through a chain of, say 10, pairwise intermediary comparisons, the network estimate is the sum of all direct effects. Any bias present in one direct effect estimate translates 1:1 into the network based estimate. Accordingly, the entries of the H matrix are all equal to 1 in the relevant row.
Both, Salanti [3] and the present paper aim to rescale or transform the rows of the H matrix in order to achieve a 'sum to one' condition. In the case of the linear graph, the percentage contribution is 10% for each direct comparison. I am not convinced, that this number adequately captures the role of each direct effect estimate in this example. Consider now, an example where two treatments are connected through three paths with one, two, and three edges, respectively, each with flow 1/3. Then each direct comparison has the same influence on the network based treatment effect estimate: A bias of 1 unit in one direct estimate translates into a bias of 1/3 unit in the network based estimate. These proportions are conserved in Salanti's notion of percentage contribution, but not so in the newly introduced concept. Salanti [3] attaches a percentage contribution of 1/6 to each comparison just as if we had pooled six equally precise trials comparing the same pair of treatments. But the influence of direct comparisons is different in this network and should be characterized by the number 1/3. That is why I have objections against Salanti's [3] concept of percentage contribution and even more so against the newly introduced concept, which attaches unequal contribution quantities of 1/9, 1/6 and 1/3, respectively.
The new method should not be introduced into routine of network meta-analysis. If this paper is going to be indexed, its draw backs and caveats should be very clearly set out.
paper is going to be indexed, its draw backs and caveats should be very clearly set out.
Thank you for this insightful comment. We agree with the point you make which we find that enlightens potential applications or extensions of the presented method to account for bias in network meta-analysis. We would like to make a distinction between 'representation of the contribution of pieces of ' and evidence in a 0 to 1 scale 'using this percentage contributions to represent and measure the '. In this paper, we focus on the first aspect of translating impact and directionality of potential bias the matrix into percentages and we subsequently apply it to depict the contribution of pieces of evidence according to a study characteristic. Our aim is to give a representation of the evidence that each comparison -or study, see our response to comment 6 below-contributes to the estimation of each NMA treatment effect without considering how this evidence could bias the results. Note that the bar plot of figure 3 gives a picture of the contribution of high, moderate or low risk of bias evidence without giving insight on the amount or direction of bias.
Quantification of bias in the direct evidence and its impact on the NMA estimates is subject of further research but not addressed within this paper. We have now added a paragraph in the Discussion to highlight the limitations of the presented method as you recommend. The paragraph reads: "Study limitations may lead to biased NMA treatment effect; however, the amount and direction of bias in the NMA treatment effect as a result of the within-study bias is not straightforward to define and is not currently accommodated within the percentage contribution matrix. First, a single biased trial may affect an entire indirect route; thus, even if its percentage contribution is small, its consequences in the estimation of the NMA treatment effect may be important. Second, the direction of bias across studies involved in a stream may vary. For example, bias in two comparisons in the same stream may either cancel out or add-up in favor of one of the two treatments. We aim to extend the methods presented in this paper to develop a network meta-regression model that will use the direction and the amount of bias to determine whether and how much NMA treatment effect estimates will be biased as the result of within-study bias." 2. The concept of streams is a nice one. It allows to represent a network based treatment effect as a weighted sum of a direct and indirect effect estimates each corresponding to one path from source to sink. Note however that these effects are stochastically interdependent and hence, the aggregation of streams is different from the usual process of pooling evidence.
Thank you for this comment; indeed the aggregation of streams is not equivalent to the aggregation of studies in a pairwise meta-analysis. We write in 'Percentage contributions of direct comparisons': "It can be shown that an NMA effect is a linear combination of the direct effects combined within paths." And we now added in the same section: "Equation 5 represents θ as a weighted sum of direct and two indirect effect estimates; the effects are stochastically interdependent and, hence, their aggregation is different from the aggregation of studies in a pairwise meta-analysis." xy N 3. To be somewhat less destructive, I propose to arrange the network graph with the source to the left and the sink to the right and all other vertices placed, such that all directed edges point from left to right. Then, due to the flow properties, any vertical cross-section gives a set of flows that sum to one. In that sense, the untransformed rows of the H matrix already contain percentage contributions. Displays used to present voter hiking can be used to represent these flows (see e.g. www.zeit.de/politik/deutschland/2017-05/waehlerwanderung-nrw-landtagswahl-cdu-spd-fdp ).
Thank you, we agree that horizontal representation is more intuitive. We re-arranged figures 1 and 2 accordingly. Table 3  Thank you very much for this comment which we believe is related to your comment 1. We agree that such a representation would be useful but we believe it could confuse readers with respect to the scope of this paper which is not to quantify the impact of within-study bias. As said in comment 1, the scope of this paper is to represent the percentage contributions of each piece of evidence; combining this information with information of study characteristics in a figure such as figure 3 may be a useful representation of the contribution of these characteristics to the estimation of NMA treatment effects. The influence of these characteristics, though, is not straightforwardly derived and it is subject to further research. Thus, we would prefer to leave Table 3 and Figure 3 unmodified and construct such representations of the direction of bias in future work where we aim to develop models to account for study-level bias using the presented methodology.

I propose to modify
Please also see the reply to comment 1 and the addition to the Discussion.
5. Discussion "The calculation of the percentages uses the estimated variances of direct effects and thus incorporate associated uncertainty in their estimation." Note however, that in a linear network each direct effect estimate contributes equally, irrespective of the size and precision of the study.
We agree this statement is confusing and hence deleted this phrase from the Discussion. The paragraph now reads: "In this paper, we present a new approach to derive percentage contributions of individual studies to the treatment effect estimates in NMA. We made use of the fact that the composition of network treatment effect estimates can be interpreted as a flow of evidence. An assumption that underlies our algorithm is the equal split of the stream flow to the involved comparisons. Although indirect effects are not weighted averages, we find this approximation to be a pragmatic approach that reasonably reflects the amount that each comparison contributes to network effects." 6. Discussion "Applying the algorithm to networks of interventions can be used to quantify the contribution of potential study limitations to the NMA treatment effect estimates." The two stage approach discussed in this paper is not essential. Both the flows and the Supplementary File 3: 1st paragraph: What is the vector space of G over R?
Thank you for this comment; as it was not clear and in order to simplify this part, we eliminated the notions of 'vector space' and 'basis' in the document. It now reads: "However, in the general case, the number of paths used in the algorithm is smaller than the number of elements of Π. The number of directed paths which suffice to 'spend' the flow from source to sink is less or equal to df+1 where df is the inconsistency degrees of freedom in the Lu and Ades inconsistency model (1), df=D-T+1." 2nd paragraph: 'The number of directed paths' obviously refers to the number of directed paths "which suffice to 'spend' the flow from source to sink".
Thank you, we corrected it.
The authors state that, if no bridges exist, this number equals to df+1. However, it depends on the flows, too. Consider a network composed of two squares connected at one vertex. It has df=2 inconsistency degrees of freedom and generally 3 paths are needed to spend the flow between the two most distant vertices. However, if all edges have equal flows, only two paths are needed. Moreover, the connecting vertex has the same effect as the bridging edges. Hence the argument has to be extended to the case of graphs that can be split by cutting through a vertex. Note also, that a network estimate that compares treatments from different subgroups of treatments connected through a bridge or a 'breaking' vertex can be written as the sum of three respective two subnetwork effect estimates: one comparing the source treatment to the bridge post, one for the bridge, if present, and one comparing the second bridge post to the sink. Because it is an unweighted sum, the contribution of the three (or two) summands should be equal.
Thank you very much for these interesting notes with which we totally agree. We extended the 'Number of directed paths' section in Supplementary File 3 to include discussion on the 'breaking' vertex situation and the consequences you describe. In particular we wrote: "Another situation where the number of directed paths which suffice to 'spend' the flow from source to sink is less than df+1 elements occurs when a vertex has at least two inflows and two outflows, all equal between them. In such a situation the flow from source to sink may be spent in less than df+1 directed paths. Such a vertex will be called breaking vertex. Are the conclusions about the method and its performance adequately supported by the findings presented in the article? Partly Competing Interests: No competing interests were disclosed. I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
No competing interests were disclosed.

Competing Interests:
The benefits of publishing with F1000Research: Your article is published within days, with no editorial bias You can publish traditional articles, null/negative results, case reports, data notes and more The peer review process is transparent and collaborative Your article is indexed in PubMed after passing peer review Dedicated customer support at every stage For pre-submission enquiries, contact research@f1000.com