Interactive Clustered Heat Map Builder: An easy web-based tool for creating sophisticated clustered heat maps

Implementation

The Interactive Builder⁹ is web-based application that accepts an uploaded data matrix and then walks the user through several steps to transform the data, perform hieratical cluster, and format the resulting CHM. The application is implemented as HTML, CSS, and JavaScript on the browser-side and Java servlets on the web server. Data manipulation and heat map generation are implemented in Java classes used by the servlets. The clustering is performed by a servlet using the Renjin engine (https://www.renjin.org) to perform R clustering functions in Java. Browser sessions are tracked by the server to create a working area for each user and prevent users from seeing each other’s data or maps. In addition to the working version of the data matrix on which transformations are performed, an original version of the matrix is preserved. Returning to a previous matrix state is accomplished by restoring the original version and then re-applying transformations until the requested state is restored. The site retains constructed heat maps and the related uploaded data only for the duration of the HTTP session.

A Java NG-CHM heat map generator .jar file is used to construct the heat map repeatedly as options are selected in each step of the builder. The heatmapProperties.json file, which contains all options selected by the user, conveys the selected options to the generator. The current NG-CHM file set is stored in a directory under the session ID. The NG-CHM file is a zipped version of the NG-CHM directory. The downloaded .ngchm file can be saved locally and viewed interactively using a local instance of the NG-CHM viewer that can also be downloaded from the builder site. An overview is given in Figure 1.

Figure 1. High-level overview of the interaction of heat map builder components.

Heat maps are built on a webserver. A browser session ID is used to create a separate, temporary working area for each user. Heat map construction sessions are cleaned up when the session is ended, but PDF and NG-CHM heat map files can be downloaded.

The full source code for the Interactive Builder is available in GitHub.

Operation

There is no need to install software to use the Interactive Builder⁹ it is available for public use on our server at https://build.ngchm.net/NGCHM-web-builder/. If, however, a local private installation of Interactive Builder is preferred, there are two simple installation methods.

Organizations familiar with Docker can run the Builder as a Docker container. To do this, clone the git repository. The base folder of this repository has a docker build file. Run the docker build command in this directory with a –t option to name the resulting docker image. Then use the docker run command to start a container using the image. The heat maps created by the software are transient and last only for the duration of a user http session so there is no need to mount an external directory to the container for persistent storage. The port for connecting to the webserver in the container does need to be specified in the docker run command. Connect the desired external port to the tomcat instance in the container, for example –p 80:8080. Users should then be able to connect to Interactive Builder using their browser and the URL of the docker container. For example, http://<docker machine IP or URL>/ NGCHM-web-builder.

The other option for deploying the software is to install it on an existing web server like tomcat. To do this, first clone the git repository and then use the ant script, ant_buildfile.xml in the NG-CHM_GUI_BUILDER folder to create a .war file. Then simply copy the .war file to the webapps directory of the web server. The application should then be available at http://<server URL>/ NGCHM-web-builder.

Select matrix

The Open Matrix File button on the first page of the builder (Figure 2) is used to upload the data matrix. A name and optional description to be associated with the heat map are entered. When the data have been loaded, the Select Matrix page will show the first few rows and columns of the matrix. It is important that the builder correctly identify the row labels, column labels, and matrix data; the backgrounds of labels and matrix data should be blue and green, respectively. If the input file has extra rows or columns, you may need to correct the identification of labels and matrix data by selecting the appropriate radio button and then clicking on the correct location in the matrix displayed.

Figure 2. Heat map creation starts with importing a text matrix file (e.g., *.txt, *.csv or Excel *.xlsx file) and identifying the row labels, column labels and numerical data values.

Note that several screens in the builder include advanced features that are hidden by default to simplify the process for first-time users. The use-case example here does not require advanced features, but be aware that additional capabilities can be accessed using the Advanced Features checkbox.

Transform/filter the data

Creating a good heat map depends on proper data preparation. The second step in the build process is the Data Transform page (Figure 3), which provides three primary categories of matrix transformations: functions that identify and replace missing/invalid values, filters to remove rows or columns, and transforms to perform mathematical operations on data values. There are additional choices in advanced mode for transposing the matrix and calculating correlations.

Figure 3. The data transform page makes it easy to perform operations on the matrix like log transformation or filtering to reduce and normalize data.

The right-hand panel of the Transform page provides summary statistics about the data matrix, including the number of rows and columns, a histogram of the data distribution, and an indication of the number of invalid cells in the matrix. The top of the page also provides suggestions about transformations that can be performed and flags any problems with the data. The use-case matrix is too large for the Iterative Builder to use in creating a heat map, so a message in red indicates that. Currently, the website limits the heat map to no more than 4,000 total rows and columns and no more than 3,500 elements on either axis.

For this use case, we apply the following transforms:

After applying the transformations, the matrix contains no errors and should be suitable for heat map generation (Figure 4). Note that the left-hand panel shows the history of transformations performed on the matrix, and one can ‘undo’ back to any previous state of the matrix (including the original version) by clicking the desired previous state and hitting reset. More generally, the entire process of creating a heat map is iterative; the Next and Previous buttons can be used to return to previous steps to try different options. If, after generating the heat map, it appears that there should be more or fewer rows or different transforms, one can return to the pertinent screen and use the history and Reset option to adjust the data matrix. Finally, as an added feature, the Transform screen enables the user to download the filtered, transformed matrix for use in other analyses.

Figure 4. The transformed dataset has a better distribution and size for heat map generation than did the original.

The history of transformations in the left-hand panel can be used to undo changes and revert to previous matrix states.

Clustering

The next step is clustering (Figure 5). The row order and column order drop-down menus can be used to select the clustering algorithm and distance measure to be applied to the rows and/or columns. Ward’s algorithm with Euclidean distance metric is one common choice, but the menus include many other possibilities, appropriate for different purposes and data characteristics. For the sample case, the Ward/Euclidean options provide strong separation in the dendrogram and interesting groups of samples. The menus also allow the rows and columns to be left in original order or randomized. Additional options will be provided in the future.

Figure 5. The clustering step supports many different clustering methods and distance measures.

The Apply button performs clustering and displays the resulting dendrograms.

Please be aware that clustering of larger matrices may take a few minutes to complete. (The time it takes to cluster data increases approximately as the square of the number of rows or number of columns, whichever is larger.)

Covariate bars

The next page allows covariate (classification) bars to be added to the heat map (Figure 6). Covariate bars add descriptive information about the rows or columns of the heat map. A covariate bar file has the same labels as the rows or columns in the matrix and an annotation value. In this use-case we will use TCGA clinical data to add age, smoking status, gender, and tumor stage to the heat map. The covariate file contains sample ids and clinical values – one value per line. When a covariate file is added, one must identify it as a row or column covariate and specify whether it contains discrete (categorical) data or continuous values. In this case smoker status, gender, and stage are discrete column covariates, and age is a continuous column covariate.

Figure 6. The covariate screen allows for the addition of supplemental data that describes the rows or columns of the data.

This screen is also used to change the color of values and ordering of the covariate bars.

After covariate bars have been added, the colors associated with the covariate values can be changed. If the color scheme might be useful for other maps, the palette can be saved to the server using the See Palettes button. Covariates can be reordered on the same screen.

An advanced feature, accessed on the cluster page, is the ability to generate a covariate bar based on the clustering dendrogram. If, for example there are four distinct clusters in the data and one wants to emphasize them in discussion of the heat map, a covariate that identifies the four top clusters based on the four top branches of the dendrogram can be generated.

Another notable advanced feature is the ability to include classification data in the original matrix uploaded in the first step, rather than providing individual covariate files on the covariate page. Choosing advanced features on the first page enables the user to identify covariates as well as labels and data in the uploaded matrix.

Format heat map

The format screen (Figure 7) supports the final step in generation of a heat map, adjustments of its appearance:

Figure 7. The format step is used to make changes to the appearance of the heat map, for example, changing the color scheme or altering the breakpoints associated with the colors.

Many appearance change options are available.

For this use case, several changes were made: (i) a slight adjustment to the break points to emphasize high and low values in the matrix, (ii) identification of row labels as gene symbols, and (iii) identification of column labels as TCGA sample identifiers. Associating the labels with known data types activates available type-specific link-outs to external data resources.

Interesting advanced features on the same page include the addition of ‘top items’ that will be displayed in the global (i.e., full) heat map view. For example, to show the positions of a few key genes, they can be entered on the page and will show on the global heat map display. Another powerful advanced feature is the ability to add gaps to emphasize sub-groups in the heat map.

Heat map – view and download

The heat map is now complete, but the Prev button can still be used to go back to previous build steps to try different options. On this final page of the Interactive Builder (Figure 8), the map can be explored dynamically and downloaded. The Get Heat Map PDF button downloads a PDF of the summary and/or detail views as they appear on the screen – including a version of the detailed view zoomed as desired. The legends and other metadata are shown on a separate page of the pdf. The final screen can also be used to explore the dynamic heat map by zooming, panning, searching, dendrogram selection, and link outs. Clicking the Expand Map button devotes the whole browser window to the map.

Figure 8. The heat map review and download screen shows the completed heat map, allows for dynamic exploration of the map, and provides download options for a PDF, an NG-CHM, and/or the construction history.

Heat maps constructed on the Iterative Builder website are not saved. However, NG-CHMs can be downloaded to save and explore dynamically on your own computer. Select the Get NG-CHM file to obtain a map and then select the Get Heat Map Viewer to get a stand-alone NG-CHM viewer to run on your computer. See our NG-CHM site for more details on the capabilities of dynamic heat maps, additional builders to generate NG-CHMs (Galaxy and R)², and instructions on how to embed dynamic heat maps in your websites - https://www.ngchm.net/. Also see our YouTube channel for tutorials on NG-CHM features.