A simple formula for enumerating comparisons in trials and network meta-analysis

We present use of a simple formula to calculate the number of pairwise comparisons of interventions within a single trial or network meta-analyses. We used the data from our previous network meta-analysis to build a study-based register and enumerated the direct pairwise comparisons from the trials therein. We then compared this with the number of comparisons predicted by use of the formula and finally with the reported number of comparisons (indirect or direct) within the network meta-analysis. A total of 133 trials of 8 interventions were selected which included 163 comparisons. The network of these showed 16 unique direct comparisons. The formula predicted an expected 28 indirect or direct comparisons and this is the number that were indeed reported. The formula produces an accurate enumeration of the potential comparisons within a single trial or network meta-analysis. Its use could help transparency of reporting should a shortfall occur between comparisons actually used and the potential total.


Introduction
The pairwise comparisons reported within each randomized controlled trial are being documented in study-based registers 1 . This lends itself to accurate indexing and enumeration of these comparisons within the studies and then subsequent supply of immediate, highly sensitive and highly specific search results to those wishing to investigate one or more particular comparisons within systematic reviews and meta-analyses or overviews and network meta-analysis 1,2 .
To gain a perspective on the absolute effectiveness of a treatment it is ideal to compare all the existing medications with placebo and for relative effects with each other in pairwise comparison trials. However, some of pairwise comparisons of the medications have not been tested within trials at all. Finally, even if some of the possible pairwise comparisons have been directly tested within trials not all may be eligible for inclusion in a network meta-analysis 3 . This leaves a gap between the research has been done and the research that should or could have been undertaken and finding this highlights gaps in the fair testing of treatments 4 .
A two-arm trial will generate one pairwise comparison. A threearm trial, however. will generate three, and a six-arm study, 15 pairwise comparisons. It is easy to lose track of how many comparisons one study can generate. This is more likely when it comes to the many direct, indirect or mixed comparisons within a network. This paper describes a simple formula for enumerating the possible number of comparisons within a single trial or planned network meta-analysis in advance.

The formula
The formula below solves this where n is the number of arms in a single study or network and N is the number of pairwise comparisons: n is a natural number; Then every intervention is compared to every other intervention except itself so: n*(n-1); Because N is a bidirectional comparison (X vs. Y = Y vs. X) so: (n*(n-1))/2; This is an established formula from combinatorics for calculating number of pairs for a number of items in a set.
The networks of 2 to 10 interventions will create networks in shapes of line, triangle, rectangle, pentagon, hexagon, heptagon, octagon, nonagon and decagon, respectively. A visual proof of a network of five interventions and (5*(5-1))/2=10 pairwise comparisons is presented in Figure 1. Adding any new intervention to the trial or network will create n-1 new pairwise comparisons. For example, where there are 6 arms in a trial-or 6 nodes in network meta-analysis-there will be (6*(6-1))/2=15 comparisons; adding a new intervention (6+1=7) will create 7-1=6 new pairwise direct comparisons in an individual trial and 6 direct or indirect comparisons in a network meta-analysis. Although this formula has been used for other purposes such as Metcalfe's law in telecommunication, its use in the current context is novel.
Testing the formula: working back from existing network meta-analyses We used the open data 5 from our previously published network meta-analysis 6 to re-create and enumerate the comparisons within the network. Using the direct comparisons reported in the trials within the network, we applied the formula and then compared the number of potential or expected comparisons (formula-derived) and the actual or observed number reported within the network analysis.

Number of direct and indirect comparisons
We built a small study-based register based-thus avoiding the pitfall of multiple counting-containing all 133 included studies in our previous network meta-analysis 6,7 . These trials reported comparisons from 8 interventions. Using our formula, 8 interventions should create 28 unique comparisons: (8*(8-1))/2=28 ( Figure 2).

Reported comparisons within the trials
We extracted the separate intervention arms from the open data to re-create the direct comparisons from within trials. The trials had either two or three arms so each study could create either two or three comparisons. As a result the 133 studies had 163 comparisons, the majority of which were duplicated.
After removing these duplicates, this created 16 unique direct comparisons with between 1 and 47 studies per comparison for 8 interventions (Table 1). These 16 observed comparisons are 57% of the 28 expected by use of the formula above.
Direct comparisons eligible for network meta-analysis Among five networks reported in the final paper, the number of comparisons in these five network meta-analyses, however, varies from 6 (for 3 networks) to 11 (for 1 network) and 13 (for 1 network) ( Figure 3). As visualized in Figure 3, only 21.42% to 46.42% of comparisons were eligible for pairwise meta-analysis ( Table 2).
Comparisons in network meta-analysis plots From Figure 3 we can calculate that about 42% of comparisons expected through use of the formula have not been tested directly in trials. This is a direct evidence-gap. The number of missing comparisons varies between nine out of 15 in three networks with six interventions, 17 out of 28 in one network with eight interventions, and 15 out of 28 in another network with eight interventions ( Figure 3). However, all 28 comparisons expected by use of the formula were utilized and reported within the network meta-analysis. It is possible that some of the comparisons predicted by the formula would have been deemed ineligible-either by adherence to a network review protocol or through post hoc exclusions-but this was not the case in this particular review ( Figure 4). This diagram shows that only some of the comparisons from trials in study-based register could be included in pairwise meta-analysis. In addition, the number of comparisons in network meta-analysis (calculated by formula) is larger and inclusive of all the comparisons in the network of interventions and includes all the possible unique comparisons even if the comparisons are not in trials or in pairwise meta-analysis.    Network meta-analysis √ √ √ √ 28 (Figure 2) 100.00 * There are five networks in Figure 3 and each has 6, 11, or 13 eligible comparisons. Three out of 16 comparisons from trials have not been included in any of five network plots.

Discussion
This formula can be employed when estimating the total number of comparisons (direct and indirect combined) theoretically possible within a proposed network meta-analysis. It would be possible that there would sometimes be a discrepancy between the number of comparisons theoretically possible and those actually employed within any given network meta-analysis. The formula would highlight this for researchers and readers and, before and after analyses, facilitate descriptions of why particular comparisons have not been included.

Conclusion
The formula produces an accurate enumeration of the potential comparisons within a single trial or network meta-analysis.

Figure 4. Venn diagram showing the coverage of comparisons by the network meta-analysis (from formula), and pairwise meta-analysis (from network plots), and trials (from study-based register).
Any shortfall between the full potential of the data and the actual number of comparisons within a network meta-analysis should be possible to explain through reference to pre-stipulated eligibility criteria or post hoc exclusions.

Data availability
The data analyzed in the present study have been published previously 6,7 .

Grant information
The authors declared that no grants were involved in funding this work. 1.

G. Mustafa Soomro
St. James Hospital, Portsmouth, UK This is a useful paper for demonstrating and discussing how an established formula could be used for calculating all possible pairs of comparisons for interventions in a network meta-analysis. Thus, reviewers would be able to find out how many potential comparisons have not been carried out.
I have some minor suggestions numbered from 1) to 4): In the abstract the following sentence should be amended for clarity perhaps as follows: "A total of 133 trials included in the network generated 163 comparisons (16 unique direct comparisons for 8 interventions)". Amendment: "A total of 133 trials of 8 interventions were selected which included 163 comparisons. The network of these showed 16 unique direct comparisons.". On page 2, it says that N is a triangular number. Either this point is not relevant, or why being a triangular number is important should be described. I think they should say that the formula they have used is an established formula from combinatorics for calculating number of pairs for a number of items in a set. On page 3 under "Comparisons in network meta-analysis plots", line 2 and line 11 "though" should be changed to "through".

If applicable, is the statistical analysis and its interpretation appropriate? Yes
Are all the source data underlying the results available to ensure full reproducibility?
No source data required

Are the conclusions drawn adequately supported by the results? Yes
No competing interests were disclosed.

Competing Interests:
No competing interests were disclosed.

Competing Interests:
Reviewer Expertise: Systematic reviews and meta-analysis and secondary data analysis.

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
Author Thanks for spending your time for reviewing our work and commenting on it. In the following lines, we replied to your comments and made changes to the paper to cover your suggestions.
: This is a useful paper for demonstrating and discussing how an established formula COMMENT could be used for calculating all possible pairs of comparisons for interventions in a network meta-analysis. Thus, reviewers would be able to find out how many potential comparisons have not been carried out. I have some minor suggestions: In the abstract the following sentence should be amended for clarity perhaps as follows: "A total of 133 trials included in the network generated 163 comparisons (16 unique direct comparisons for 8 interventions)". Amendment: "A total of 133 trials of 8 interventions were selected which included 163 comparisons. The network of these showed 16 unique direct comparisons." : Thank you. We revised the sentence as suggested.

REPLY
: "A total of 133 trials of 8 interventions were selected which included 163 comparisons.

CHANGE
The network of these showed 16 unique direct comparisons".
: On page 2, it says that N is a triangular number. Either this point is not relevant, COMMENT or why being a triangular number is important should be described.
: We agree. We deleted this part.

REPLY
: "N is a triangular number" was deleted. CHANGE : I think they should say that the formula they have used is an established formula from COMMENT combinatorics for calculating number of pairs for a number of items in a set.
: Thank you for adding this. We agree and we added your suggestion in the text right after REPLY formula.
: "This is an established formula from combinatorics for calculating number of pairs for a CHANGE number of items in a set". : On page 3 under "Comparisons in network meta-analysis plots", line 2 and line 11 COMMENT "though" should be changed to "through".
: Thank you for detecting these errors. We collected both.

REPLY
: Thanks for spending your time for reviewing our work and commenting on it. In the following lines, we replied to your comments and made changes to the paper to cover your suggestions.
: In my opinion, this is a novel approach which allows the quantification of comparators COMMENT in trials and network meta-analyses. Following on from this, the comparators which are possible versus how many were reported can be discussed in further depth. This formula has great applicability in terms of network meta-analysis methodology which is still very much a developing area. Some very minor points; please add a table which demonstrates the use of the formula to quantify how many comparators are possible for trials from 2 interventions up to 10.
: Thanks for your positive comment. We added a single visual proof for this formula to show REPLY how it works. The networks of 2 to 10 interventions will create networks in shapes of line, triangle, rectangle, pentagon, hexagon, heptagon, octagon, and decagon.
: We added: "The networks of 2 to 10 interventions will create networks in shapes of line, CHANGE triangle, rectangle, pentagon, hexagon, heptagon, octagon, and decagon".
: Pg 3, paragraph with the heading 'Reported comparisons within the trials' -line 2 COMMENT change 'there' to 'the'.
: Thank you and sorry for this typo.

REPLY
: We change 'there' to 'the'. CHANGE : Explanation of Figure 4 in text is not clear.

COMMENT
: We added a few sentences in the text to clarify it.

REPLY
: "This diagram shows that only some of the comparisons from trials in study-based CHANGE register could be included in pairwise meta-analysis. In addition, the number of comparisons in network meta-analysis (calculated by formula) is larger and inclusive of all the comparisons in the network of interventions and includes all the possible unique comparisons even if the comparisons are not in trials or in pairwise meta-analysis". The benefits of publishing with F1000Research: Your article is published within days, with no editorial bias You can publish traditional articles, null/negative results, case reports, data notes and more The peer review process is transparent and collaborative Your article is indexed in PubMed after passing peer review Dedicated customer support at every stage For pre-submission enquiries, contact research@f1000.com Page 12 of 12