## Introduction

Agglomerative hierarchical clustering (HC) is one of the classic and yet still very popular cluster analysis methods in data exploration^{1,2}. Its implementation is widely available and execution of the clustering requires only a few settings, such as a choice of distance metric and linkage algorithm^{3}. The clustering process begins with individual input elements as singleton clusters and successively merges a pair of most similar clusters until only one cluster remains. The dissimilarity, or the distance, between two clusters is defined by a distance metric and updated by a linkage algorithm. The output of HC is typically represented in a form of a binary tree, called a dendrogram. In a dendrogram, the similarity of two clusters is encoded in the height of the branch where two clusters merge. Two very similar elements are merged in the early stages of clustering, thus the height of the branches between these elements is relatively small. The dissimilarity between two clusters increases with each successive merge, resulting in a binary hierarchical structure with a monotonic property^{4}. Therefore, a dendrogram represents both cluster-subcluster relationships as well as the order in which the clusters were merged^{5}.

There are two unique uses of a dendrogram in exploratory data analysis. First, clusters of input elements can be inferred from the subtree structures below a certain threshold by “cutting the tree”. It is an advantage of hierarchical clustering that this threshold value can be adjusted based on domain-specific knowledge to result in clusters of different sizes. Second, a linear order of observations (rows) or attributes (columns) of an associated matrix can be derived. This linear order of observations is typically used to reorder the columns or rows of the data matrix. Then, the matrix is visualized as cluster heat maps^{1}, where dendrograms and heat map visualizations are coupled (Figure 1).

The linear order derived from a dendrogram is more meaningful than a random order, as it groups similar items together^{6,7}. However, two consecutive items in this order are not necessarily similar, since these leaves could belong to different subtree structures, or simply be quite distant from each other. This is a common misinterpretation of a dendrogram: one may expect similarity between two input elements based on the proximity in the leaf order^{8,9}. In addition, there are 2^{n−1} possible orderings given *n* input elements, because the orientation of clusters at each merge can be flipped without affecting the underlying hierarchical structure, thus rendering a unique optimization challenge.

To address the misinterpretation of dendrograms and the optimization problem, a number of methods have been proposed to rearrange the structure of a dendrogram. Gruvaeus and Wainer^{10} proposed a method (GW) to order leaves such that two singleton clusters at the edge of adjacent subtrees are most similar, given the constraint of the binary tree structure. Bar-Joseph *et al.*^{6} proposed a method, called the optimal leaf ordering (OLO), to maximize the sum of the similarity of any adjacent elements in the ordering. Similarly, Chae and Chen^{11} proposed a method for ordering by minimizing the bilateral symmetric distance between two adjacent clusters. All these methods aim to homogenize the linear order in one way or another and are evaluated in terms of either a loss function, such as the Hamiltonian path length, or a merit function, such as the number of anti-Robinson events^{12}.

Even though these seriation-based leaf ordering methods exploit the binary tree structure to reduce the number of permissible permutations, these methods have short-comings. First, they homogenize and optimize the distance between items in the linear order, and this still encourages the common misinterpretation of dendrograms, reading a dendrogram horizontally. Second, the dendrogram structure is only a means to reduce the number of permissible permutations, and the graphical representation of the resulting dendrogram obscures the intrinsic properties of the hierarchical clustering result, such as the cluster-subcluster relationship and the order in which clusters are merged.

In the biological domain, Eisen *et al.*^{13} have introduced and established a cluster analysis method for high throughput gene expression data using cluster heat maps. The method includes a leaf orderings by weighting genes based on genome coordinates or the average expression level. The resulting linear order is more meaningful in terms of biology, but the method requires prior knowledge or additional information for the weighting.

In this paper, we present leaf ordering heuristics, named modular leaf ordering (MOLO), to address the aforementioned shortcomings by constructing a dendrogram that reflects a) the monotonic order in which clusters are merged and b) the nested cluster relationships. We compare dendrogram and cluster heat map visualizations created using our heuristics to the default heuristic in R and seriation-based leaf ordering methods. The implementation is available as an R package, named "dendsort", from the CRAN package repository. The R script for generating figures in this paper is available as a supplementary material. Further examples, documentations, and the source code are available at [https://bitbucket.org/biovizleuven/dendsort/].

## Open Peer Review

## Current Referee Status: ?
Key to Referee Statuses
VIEW
HIDE
ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested
Approved with reservationsKey revisions are required to address specific details and make the paper fully scientifically sound
Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions

Read more about the unique F1000Research publication and peer review model here.3:177(doi: 10.5256/f1000research.5108.r5926)

http://f1000research.com/articles/3-177/v1#referee-response-5926

NOTE:it is important to ensure the information in square brackets after the title is included in this citation.It is well written and the authors go into sufficient detail to describe the issues with the available ways to display dendrograms. They have applied their method ... Continue reading

It is well written and the authors go into sufficient detail to describe the issues with the available ways to display dendrograms. They have applied their method to a number of case studies, and here they show that their method improves the distinction between clusters in the data.

The method is made available in the form of an R package (dendsort). Creating re-ordered dendrograms is quite easy with this package, but I found it more cumbersome to apply the method to heatmaps. The standard heatmap() function has a 'reorderfun' parameter, but it expects other parameters than the dendsort() function. The documentation of the package should be improved to make this easier.

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.Competing Interests:No competing interests were disclosed. CloseF1000Research2014,3:177 (doi: 10.5256/f1000research.5108.r5926)http://f1000research.com/articles/3-177/v1#referee-response-5926

it is important to ensure the information insquare brackets after the titleis included in all citations of this article.3:177(doi: 10.5256/f1000research.5108.r5627)

http://f1000research.com/articles/3-177/v1#referee-response-5627

NOTE:it is important to ensure the information in square brackets after the title is included in this citation.Within the visualization community, we know that heat map representations work much better when clustered ... Continue reading

Within the visualization community, we know that heat map representations work much better when clustered appropriately since correlations with colour are more difficult to detect when colour grouping is not present, see Haroz and Whitney (2012). This technique presented by the authors goes some way to making heat map representations more effective.

Their evaluation is complete and shows clear advantages over existing methods.

We have read this submission. We believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.Competing Interests:No competing interests were disclosed. CloseF1000Research2014,3:177 (doi: 10.5256/f1000research.5108.r5627)http://f1000research.com/articles/3-177/v1#referee-response-5627

it is important to ensure the information insquare brackets after the titleis included in all citations of this article.