Insights into the Datasets, Tools, and Training Needs of the AnVIL Community: 2024

Design of the poll

Given that AnVIL is a newer platform that has never polled users before, this community poll was designed to collect baseline information specifically with regards to the aforementioned study aims ( Table 1). Questions were sourced from the AnVIL team as a whole – a combined effort across Johns Hopkins University, the Broad Institute, Fred Hutch Cancer Center, Vanderbilt University Medical Center, and other institutions within the AnVIL team.

The AnVIL Community Poll was constructed using Google Forms. The poll had 6 parts ( Figure 1, Table 1). In the first part, using a multiple choice question, respondents were asked to select the best description of their current usage of the AnVIL platform. We used the answers to categorize respondents as returning or potential users of the AnVIL platform. Going forward, we discuss usertypes as either returning or potential users of the AnVIL. We define returning users as those who selected any of the following: “For ongoing projects (e.g., consistent project development and/or work)”, “For short-term projects (e.g., short, intense bursts separated by a few months)”, or “For completed/long-term projects (e.g., occasional updates/maintenance as needed)”; and potential users as those who selected any of the following: “I do not currently use the AnVIL, but have in the past”, “I have never used the AnVIL, but have heard of it”, or “I have never heard of the AnVIL”. Depending on the answer to this question, respondents were directed to the second part as either a returning or potential user of the AnVIL platform. Part 2 of the poll had a single question for potential users or 7 questions for returning users. The 6 additional questions asked about their experience with, needs, and recommendation likelihood for using the AnVIL while both categories of respondents were asked to rank features of the AnVIL according to their importance for their continued (returning user) or potential (potential user) use of the platform. Every respondent was given the same questions for the remaining parts of the poll (parts 3-6), with 5, 5, 2, and 6 questions respectively for each section. If respondents assented that they were willing to be contacted again to give input in the future, then they were directed to provide an email address. Finally, all users were provided with a link to a separate google form in case they wanted to be entered in a potential prize raffle. Due to there being a total of 20-26 questions, we estimated the poll would take 10-15 minutes to complete (~30 seconds/question).

Figure 1. State of the AnVIL 2024 Community Poll Design.

The State of the AnVIL 2024 Community Poll was designed such that there were 6 parts. Every respondent was asked the same questions for Parts 1, 3, 4, 5, and 6 while respondents were provided with user type specific questions in Part 2, depending upon their answer to the first question of the poll. Part 1 was used to classify a respondent as a returning or potential user of the AnVIL; Part 2 asked questions specific to each user type (returning or potential); Parts 3-6 contained demographics, experience, awareness, and preferences questions respectively. Brief descriptions of questions within each part are provided.

The outline of the poll is as follows ( Figure 1):

For a pdf version of the full poll, see the Data and Software Availability.

Recruitment for/Dissemination of the poll

The AnVIL Community Poll was disseminated through multiple channels including posting on social media platforms like X (formerly Twitter) and LinkedIn, emailing applicable mailing lists (registered AnVIL users through Terra, AnVIL Demo attendees, AnVIL mailing list, etc.), and posting within Slack workspaces (AnVIL, nf-co.re, Community Bioconductor, Fred Hutch Cancer Center, and the Johns Hopkins University Genomics and Biostat communities). We also posted an advertisement for the survey as a news item on the anvilproject.org portal and the AnVIL Support Forum (help.anvilproject.org). Finally, the poll was sent directly to some select individuals and consortia who were asked to further circulate it. All advertisements mentioned the possibility of a prize raffle. The poll was conducted in Spring 2024. It was administered using Google Forms and open for responses from February 15th to March 25th.

Because of the nature of dissemination (through social media), we cannot confidently calculate a response rate. However, we consider this to be a data gathering exercise where we can report back the findings to the AnVIL Community and see if and how they resonate.

Ethical considerations

This work has been reviewed by The Johns Hopkins University Homewood IRB and was determined to not-qualify as human subject research (#HIRB00020632). Participating in the poll was completely voluntary, and the results were de-identified.

Analysis of the poll

Following the closing of the poll, the Google sheet with form responses was locked so that edits could not be made and the results were imported from there for analysis using R. Respondent email (if provided) was used to de-duplicate responses. Identifying information or information that was highly specific and potentially identifiable like specific institution names was removed (email addresses) or annotated and grouped into categories (type of institution, e.g., an R1 University, R2 University, Community College, etc.) before removal.

Respondents were asked about their level of experience with human genomic, non-human genomic, and clinical research. Possible responses included “Not at all experienced”, “Slightly experienced”, “Somewhat experienced”, “Moderately experienced”, and “Extremely experienced” (a 5 point Likert scale). If a respondent selected “Moderately” or “Extremely” experienced for any of these categories, they were assigned as someone with experience in that research category. This method of assigning experience is an example of a Top 2 Box simplification.²¹

Poll takers were also asked to select the type(s) of work they perform regularly Possible choices for type(s) of work were Computational work, Engineering work, Wet lab work, Clinical work, Computational education, Wet lab education, Project leadership, Project management, Program administration, or Other (with free text entry if Other was selected). For analysis purposes, we grouped users into 5 possible personas: Clinician, Analyst, Educator, Leadership (PI), or Admin. These groups were created based on participant answers to the “type of work” question. A clinician persona was assigned if “Clinical work” was selected; An analyst persona was assigned if only “Computational work” was selected; An educator persona was assigned if “education” appeared in a response; A leadership persona was assigned if project leadership, management, or administration co-occurred with “Computational work”; and an Admin persona was assigned if any of project leadership, management or administration was selected without any other kinds of work selected. With this assignment procedure, some responses were not assigned personas and two were assigned multiple personas. To simplify those assigned multiple personas, the selections were individually inspected and categorized. We also performed principal components analysis, and did not observe any conspicuous clustering of respondents (see https://github.com/fhdsl/SOTA2024_ReportOut).

When respondents were asked to rank preferences or level of importance ( Figure 5), responses were translated into rank values. If n choices were given, “Most preferred/important” was assigned a value of 1, with each subsequent value denoting decreasing preference or importance. The “Least important/preferred” was assigned the largest possible value. An overall or average rank choice was assigned by summing the rank values and dividing by the total number of respondents. For an average knowledge or comfort score ( Figure 7A), a similar procedure was followed; however, for these questions the higher value was associated with greater knowledge or comfort and a score of 0 was used for responses of not knowing at all. Rank and comfort scores were averaged within user cohorts (e.g., persona or user type) rather than considering all respondents together.

Figure 2. Background of users in our sample.

A: We asked all respondents “How would you describe your current usage of the AnVIL platform?” Respondents were identified as returning or potential AnVIL users based on their responses. Most of the 22 returning users leverage AnVIL for ongoing projects. The 28 potential users were evenly split between those who have never used the AnVIL (but have heard of it) and those who have used the AnVIL previously, but don’t currently. B: We asked “What is the highest degree you have attained?” Most of the respondents have a PhD or are currently working on a PhD, though a range of career stages were represented. C: We asked all respondents “What institution are you affiliated with?” Most of the respondents, but also the majority of returning individuals using the AnVIL, reported being affiliated with a research intensive institution.

Figure 3. Background of users and researcher’s technological comfort.

A: We asked respondents “How much experience do you have analyzing the following data categories?” 21 respondents report that they are “extremely experienced” in analyzing human genomic data, while only 6 respondents report that they are “not at all experienced” in analyzing human genomic data. However, more respondents report being “not at all experienced” in analyzing human clinical data and non-human genomic data. B: Venn diagram showing the overlap for respondents who reported being “moderately” or “extremely experienced” for these various research categories (n = 37). 32% reported such levels of experience for all 3 research categories; the next highest percentage (27%) reported such levels of experience for human genomic data only with no overlap with other research categories.

Figure 4. Current work of users related to appropriate personas.

We asked all respondents “What kind of work do you do?” Possible selections (computational work, computational education, project management, etc.) are shown on the y-axis and cells are colored for each choice a respondent (x-axis) selected. Based on selections, respondents were clustered into Admin, Analyst, Clinician, Educator, PI, and not assigned (NA) personas. These assignments are shown in grayscale along the x-axis with a corresponding pie chart showing the relative abundance of these assignments. 2 potential users were assigned “Clinician” personas and 4 were assigned “Educator” personas, compared to 0 and 1 respectively for returning users. The other personas show similar abundances between potential and returning users.

Figure 5. Barriers to platform adoption and user preferences for training and support in our sample.

A: We asked all users to "Rank the following features according to their importance to you as a potential user or for your continued use of the AnVIL." Responses were averaged within potential and returning user cohorts to find an average rank. All respondents rated having specific tools or datasets supported/available as an important feature for using AnVIL. Compared to returning users, potential users rated having a free-version with limited compute or storage as the most important feature for their potential use of the AnVIL. B: We asked all respondents "Rank how/where you would prefer to attend AnVIL training workshops." Responses were averaged within potential and returning user cohorts to find an average rank. Both returning and potential users preferred virtual training workshops over other modalities.

Figure 6. Awareness and utilization of training and support.

A & B: We asked all respondents “Have you attended a monthly AnVIL Demo?” A: Most respondents had not attended an AnVIL Demo. However, returning users were more represented among AnVIL Demo attendees. B: All responses to (A) except “No, did not know of” were aggregated, showing that the majority of respondents are aware of AnVIL Demos. C & D: We asked all respondents “Have you ever read or posted in our AnVIL Support Forum?” C: Raw responses are shown (users could select more than one). Most respondents have not used the AnVIL support forum, but utilization in some form is reported by 24% of respondents; reading through others’ posts is the most common way of utilizing the support forum within this sample. D: Each set of user responses are recoded and aggregated to examine whether users are or are not aware of the AnVIL Support Forum. We observe that there is awareness of the support forum across potential and returning users.

Figure 7. Technological comfort with cloud-based genomic analysis tools.

A: We asked respondents “How would you rate your knowledge of or comfort with these technologies or data features?” Except for Galaxy, potential users tended to report lower comfort levels for the various tools and technologies when compared to returning users. Overall, there was less comfort with containers or workflows than using various programming languages and integrated development environments (IDEs). B: We asked all respondents “Where do you currently run analyses?” Institutional HPC and locally run (personal computers) were the most common responses. Google Cloud Platform (GCP) was reported as used more than other cloud providers within this sample. We also saw that potential users reported using Galaxy (a free option) more than returning users do.

In Part 5 of the poll, respondents were asked about their experiences with user support options. Respondents could select “No, didn’t know of”, “No, but aware of”, or some specific description related to how they used the support option (e.g., attending a demo or answering someone’s post on the support forum). We broadly translated these responses into two binary reporters: one for awareness and the other for active use. To understand awareness, we combined responses in the following way: Anything other than “No, didn’t know of” was combined to represent awareness. To understand use, we combined responses such that anything other than “No, didn’t know of” and “No, but aware of” were examples of use or future use. If looking at past use (ignoring future use), “Not yet, but registered to” was also not included in answers representing use. (See Data and Software Availability for a table representing these categorizations.)

All code for reproducing the stats and figures in this analysis are available on GitHub: https://github.com/fhdsl/SOTA2024_ReportOut. A de-identified dataset is also available.

Responses

This AnVIL user poll was conducted in Spring 2024, receiving 52 total responses. Two responses were determined to be submitted by duplicate users, leaving a total of 50 user responses used in the analysis. Given how the AnVIL Community Poll was advertised through general, often public channels rather than enumerated closed lists, it was not possible to compute an exact response rate. Considering the distribution of daily responses compared to recruitment efforts for the poll, responses were received on days without advertisement, but days with advertisement (at least 7 unique days) aligned with or directly preceded receiving responses. The two days with the highest response count (7 and 8 responses) aligned with a user listserv reminder email (2024-March-13) and an X (formerly Twitter) post (2024-March-14). The average number of responses in a day was about 1, though the most common number of responses was 0 (21 different days), followed by 1 (8 different days) if restricted to non-zero responses. When considering the institutional affiliation of respondents, there were no higher than 2 responses from users from the same institution on any day. Both potential and returning users responded throughout the time period with no discernible pattern related to recruitment method.

User backgrounds and current work

Of the 50 de-duplicated responses, 44% (n = 22) were from returning users and 56% (n = 28) were from potential users. The majority of returning user responses (73%, n = 16) belonged to the group who use the AnVIL for ongoing projects, with consistent work on the platform. The majority of potential users were evenly split between two groups (46%, n = 13 each), those who have never used the AnVIL (but have heard of it) and those who have used the AnVIL previously, but don’t currently ( Figure 2A).

Most respondents in our sample had obtained or were in the process of obtaining a PhD (62%, n = 31) though a range of career stages were represented ( Figure 2B) including those with little to no post-secondary education and those with multiple advanced degrees. Clinical advanced degrees (MDs) (6%, n = 3) were less represented than either research degrees (PhD) or Master’s degrees (18%, n = 9).

Most of the respondents using the AnVIL (n = 21) reported being affiliated with a research intensive institution (e.g., an R1 University, a medical campus, a research center, or the NIH), while one reported being affiliated with an education focused institution (e.g., R2 University or community college). Only potential users of the platform reported being affiliated with an industry-based institution ( Figure 2C). In our sample, we observed very few industry partners.

Of the 50 responses, 21 provided consortia affiliations across 23 unique affiliations (respondents could report more than one consortium). Of the 21 responses providing consortia affiliations, 13 were from returning users. Genomics Research to Elucidate the Genetics of Rare Diseases (GREGoR),^22,23 Polygenic Risk Methods in Diverse Populations (PRIMED),^24,25 Electronic Medical Records and Genomics (eMERGE),²⁶ Centers for Common Disease Genomics (CCDG)²⁷ and The Genotype-Tissue Expression Project (GTEx)²⁸ were the top 5 most represented with n=3 each for GREGoR, PRIMED, and eMERGE and n=2 each for CCDG and GTEx. Additionally, all 5 of these consortia were represented in the subset of consortia affiliations reported by returning users.

When asked about experience with analyzing human genomic, human clinical, and non-human genomic data, 21 respondents report that they are extremely experienced in analyzing human genomic data, while only 6 respondents report that they are not at all experienced in analyzing human genomic data. However, for human clinical data and non-human genomic data, more respondents report being not at all experienced in analyzing those data than report being extremely experienced ( Figure 3A). Returning and potential users showed a similar distribution on experience levels across these 3 research categories. If we consider the Top 2 Box simplification, lumping together “Moderately” and “Extremely” experienced responses (the highest 2 of the possible rankings), returning users show slightly more experience across all 3 research categories.

When limiting the responses we consider to include only respondents who reported being “moderately” or “extremely” experienced for at least one of the categories, we are left with 37 responses (17 from returning users). Considering the overlap among experience within these categories for these responses, 32% (n = 12) reported these levels of experience for all 3 research categories and 27% (n = 10) reported this level of experience for only human genomic research (but no other research categories) ( Figure 3B). 16% (n = 6) reported this level of experience for both human clinical and human genomic research (but not non-human genomic research) and another 16% (n = 6) reported this level of experience in analyzing both non-human and human genomic research.

When asked to select the kind of work that they performed, 44% (n = 22) selected only 1 description, 36% (n = 18) selected 2 descriptions, 16% (n = 8) selected 3 descriptions, and 4% (n=2) selected 5 descriptions. “Computational work” was the most frequently selected response, reported by 68% of respondents (n = 34). “Project management” and “Project leadership” followed at 36% (n = 18) and 24% (n = 12) respectively. Of the 22 respondents who selected only 1 description, “Computational work” was the most frequently selected (n = 13, 59%). Of the responses who selected more than one description, “Computational work” was paired most frequently with “Project management” or “Project leadership”. (n = 11 and 7 respectively). Two respondents utilized the “Other” option, each supplying one work description. These included “Cloud Architect” and “Software Development”. The distribution of work descriptions selected were similar when comparing between potential and returning users ( Figure 4).

All of the results summarized so far describe the varied backgrounds and current work of respondents in our sample. Despite this variety, we expect several overarching user groups: Admins, Analysts, Clinicians, PIs, and Educators. Given that the kind of work question focused on work activities and that these expected user groups have differential work activities, we assigned personas based on the responses to the kind of work question ( Figure 4). 10 respondents were left uncategorized, but the majority of respondents were assigned to either the Leadership (PI) persona (n = 13, showing evidence of computational work together with some form of project management, leadership, or administration) or the Analyst persona (n = 13, only reporting computational work). 7 respondents were categorized as an Admin persona. The Leadership (PI), Analyst, and Admin personas showed a similar split of assignment across potential and returning users. The Clinician and Educator personas were differentially represented in this sample: 2 potential users were assigned a Clinician persona because of selecting clinical work (this option was not selected by returning users) and 4 potential users were assigned an Educator persona while only 1 returning user was.

Barriers & user preferences

The kind of work that researchers perform or different hats that respondents wear are likely to influence their preferences and barriers to adoption. Respondents were asked about features that are most important for their continued or potential use of the AnVIL. Of the features listed, currently the AnVIL utilizes use-based billing and does not offer a free version. When looking at responses specific to the assigned personas, we observe that the average rank choice for easy billing setup is highest for Admins compared to other personas; Educators and Clinicians rank both a free version with limited compute or storage and greater adoption by the scientific community more highly than other personas; and PIs rank having specific tools or datasets available/supported as their most important feature of AnVIL. Specific tools or datasets being available and supported ranks as the most important feature for both returning and potential users of the AnVIL ( Figure 5A). Potential users rank having a free version of the AnVIL with limited compute or storage more highly than returning users do. Easy billing setup and on demand support and documentation are ranked in the middle with similar preferences observed across user types. Flat-rate billing rather than use-based billing is ranked the lowest across this set of features within this sample.

As for respondent preferences regarding training modalities/locations, we see that the vast majority of responses in our sample rank virtual options above all other modalities ( Figure 5B). Respondents viewed conferences, on-site at their institution, or AnVIL specific events with similar preference ranks. These observations were consistent across user types. At the time of this poll, the AnVIL had offered training through every represented modality option except for an AnVIL-specific event (e.g, the AnVIL Community Conference whose inaugural event occurred after the poll closed).

The AnVIL already supports several virtual training opportunities including monthly AnVIL Demos and a 24/7 online support forum (help.anvilproject.org). The monthly AnVIL Demos are virtual meetings hosted over zoom where typically the first 30 minutes are used for a demonstration highlighting what is possible on the AnVIL and the last 30 minutes are reserved for questions and community discussion. The AnVIL Demos are advertised through the AnVIL mailing list, on the events tab of the AnVIL portal, and described on the support forum. Recordings are posted to YouTube and listed as part of the AnVIL Collection. The 24/7 online support forum provides a place for users to submit questions or read through/reply to threads of questions regarding the platform. The AnVIL outreach team responds to questions and sorts threads into relevant categories (e.g., data access, feature requests, etc.). When asked about their utilization of these virtual training opportunities, we found that more respondents were aware of these offerings than were not aware. Potential users were split at 46% (n = 13) knowing about monthly AnVIL Demos or the support forum and 54% (n = 15) not being aware of the support opportunities ( Figure 6B,D). Interestingly, only 71% (n = 20) of potential users had matching aware/not aware responses for both AnVIL Demos and the support forum. Of the 15 not aware respondents, only 73% (n = 11) were not aware of both AnVIL Demos and the support forum. Similarly, 86% (n = 19) of returning users had matching aware/not aware responses for both AnVIL Demos and the support forum. Only 4 returning user responses were not aware of both the AnVIL Demos or the support forum. Returning users showed a higher percentage of respondents who were aware of the support opportunities (n = 16, 73% and n = 17, 77%) than potential users ( Figure 6B,D). Returning users are also more represented in the group of respondents who have attended AnVIL Demos (n = 12 or 71% of attenders) ( Figure 6A) or utilized the support forum (n = 8 or 67% of those who have read through, posted, or replied) in this sample. In general, most poll respondents have not attended AnVIL Demos with only 34% (n = 17) reporting attendance ( Figure 6A). For those who have attended an AnVIL Demo within this sample, attending more than one demo or just one demo occurs at similar rates ( Figure 6A). Most respondents have not utilized the support forum with only 24% (n = 12) in some way reporting use. Reading through others’ posts is the most common form of utilization ( Figure 6C).

User technological comfort

The poll asked about comfort with technology in two different sections: within Part 2 (the returning user exclusive questions) and within Part 4 (Experience). Within Part 4, all respondents were asked to rank their knowledge or comfort with certain technologies separate from the AnVIL. Only respondents identified as returning users were asked about these same technologies (excluding specific programming languages) on the AnVIL. Returning AnVIL users were the only respondents asked about knowledge of or comfort with AnVIL specific features such as Data Use Oversight System (DUOS), Terra Data Repository (TDS), Terra Workspaces, or generally accessing controlled access data. Returning users reported higher comfort levels for the various tools and technologies when compared to potential users. The one deviation from this trend is a slight increase in the comfort with Galaxy reported by potential users. Overall, there is less comfort with containers or workflows than various programming languages and integrated development environments (IDEs). While respondents report fairly high comfort with Jupyter Notebooks and RStudio for interactive analysis on the AnVIL, Galaxy on AnVIL is among the technologies with the lowest average knowledge or comfort. Returning users report low levels of knowledge or comfort with TDR and DUOS as well. They report intermediate levels of comfort with accessing controlled access data in general and the highest level of comfort with workspaces.

All respondents in our sample were asked where they currently run analyses. Respondents could select multiple choices. Institutional High Performance Computers (HPCs) (70%, n = 35; 64% of returning users and 75% of potential users) and locally on personal computers (68%, n = 34; 55% of returning users and 79% of potential users) were reported at the highest rates in this sample. For provided cloud options, Google Cloud Platform (GCP)²⁹ was reported as used the most (n = 21) split across user types (55% of returning users and 32% of potential users) of the AnVIL, compared to 22 returning AnVIL users in our sample). We also observed that potential users report using Galaxy (a free option)³⁰ (18%, n = 5), more than returning users do (5%, n = 1). Potential users also submitted All of Us,³¹ sciserver.org, and UKBB RAP³² as other platforms they use to perform analyses.

User resource and data needs

Returning users (n = 22) were asked about their foreseeable computational needs. The most common response was needing large amounts of storage (e.g., Terabytes), selected by 50% (n = 11); 32% (n = 7) only selected this option. Many nodes, GPUs, and large memory (>192 GB RAM) were also reported as foreseeable needs, but to a lesser extent (selected by 27%, 27%, and 18% respectively). Only 1 response selected all 4 computational needs. The most common combination of selections was GPUs and many nodes (n = 2). All other combinations were only selected once or not at all (though this could be an artifact of a small sample size and 11 possible combinations of multiple options).

46% (n = 23) of all respondents flagged the AnVIL as a repository that they use or are considering use of to share data to comply with the NIH DMS policy. Of those 23, 65% (n = 15) were returning users of the AnVIL (68% of the returning user sample of this community poll).

Over half of all respondents (n = 29) reported that they are “extremely interested” in working with controlled access datasets. When provided choices of specific controlled access datasets that respondents have accessed or are particularly interested in accessing with AnVIL, All of Us³³ and UK Biobank³⁴ were the most selected (n = 34 each). GTEx was selected by 64% (n = 32) of respondents and CCDG was selected by 40% (n = 20). All of Us, UK Biobank, and GTEx remained the top three when we subset the data to consider (1) only responses from those moderately or extremely experienced with human clinical data, (2) only responses from those moderately or extremely experienced with human genomic data, and (3) only responses from those moderately or extremely experienced with non-human genomic data. CCDG remained the 4th rank selection for those moderately or extremely experienced with non-human genomic data, but was surpassed by eMERGE and HPRC³⁵ when considering the subsets of researchers experienced with human clinical and human genomic data. Note that poll respondents were informed that All of Us and UK Biobank were not currently available on the AnVIL due to policy restrictions. (Note: Since the results of this poll were analyzed, All of Us is now more accessible to researchers.³⁶)

When asked about interest in different types of data, respondents selected Genomes/exomes (88%, n = 44) and Transcriptomes (62%, n = 31) as the major types of data they would be interested in analyzing with the AnVIL. A variety of other options (e.g., Phenotypic, Single Cell, Electronic Health Record, Epigenomes, Metabolomes, Proteomes, Imaging, etc.) were selected at a rate of at least 10% (n = 5), but no more than 40% (n = 20). Survey, structural, and variant calling were the only options selected or mentioned by fewer than 10% of respondents.