<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Publishing DTD v1.2 20190208//EN" "http://jats.nlm.nih.gov/publishing/1.2/JATS-journalpublishing1.dtd"><article xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" article-type="other" dtd-version="1.2" xml:lang="en">
    <front>
        <journal-meta>
            <journal-id journal-id-type="pmc">F1000Research</journal-id>
            <journal-title-group>
                <journal-title>F1000Research</journal-title>
            </journal-title-group>
            <issn pub-type="epub">2046-1402</issn>
            <publisher>
                <publisher-name>F1000 Research Limited</publisher-name>
                <publisher-loc>London, UK</publisher-loc>
            </publisher>
        </journal-meta>
        <article-meta>
            <article-id pub-id-type="doi">10.12688/f1000research.10964.1</article-id>
            <article-categories>
                <subj-group subj-group-type="heading">
                    <subject>Opinion Article</subject>
                </subj-group>
                <subj-group>
                    <subject>Articles</subject>
                    <subj-group>
                        <subject>Bioinformatics</subject>
                    </subj-group>
                </subj-group>
            </article-categories>
            <title-group>
                <article-title>
                    <italic>hackseq</italic>: Catalyzing collaboration between biological and computational scientists via hackathon</article-title>
                <fn-group content-type="pub-status">
                    <fn>
                        <p>[version 1; peer review: 2 approved]</p>
                    </fn>
                </fn-group>
            </title-group>
            <contrib-group>
                <contrib contrib-type="author" corresp="no">
                    <collab>hackseq Organizing Committee 2016</collab>
                </contrib>
            </contrib-group>
            <author-notes>
                <fn fn-type="con">
                    <p>All members of the 
                        <italic toggle="yes">hackseq</italic> Organizing Committee 2016 contributed equally to 
                        <italic toggle="yes">hackseq</italic> and participated in the discussions expressed in this manuscript:</p>
                    <p>Artem Babaian
                        <italic toggle="yes">, Terry Fox Laboratory, BC Cancer Agency, Vancouver, BC, Canada</italic>
</p>
                    <p>Britt Dr&#x00f6;gem&#x00f6;ller, 
                        <italic toggle="yes">Faculty of Pharmaceutical Sciences, University of British Columbia</italic>
</p>
                    <p>Bruno M Grande, 
                        <italic toggle="yes">Department of Molecular Biology and Biochemistry, University of British Columbia</italic>
</p>
                    <p>Shaun D Jackman, 
                        <italic toggle="yes">BC Cancer Agency Genome Sciences Centre</italic>
</p>
                    <p>Amy Huei-Yi Lee, 
                        <italic toggle="yes">Department of Microbiology and Immunology, University of British Columbia</italic>
</p>
                    <p>Santina Lin, 
                        <italic toggle="yes">Bioinformatics Training Program, University of British Columbia</italic>
</p>
                    <p>Catrina Loucks, 
                        <italic toggle="yes">Department of Molecular Biology and Biochemistry, Simon Fraser University</italic>
</p>
                    <p>Adriana Suarez-Gonzalez, 
                        <italic toggle="yes">Department of Botany, University of British Columbia</italic>
</p>
                    <p>Tiffany Timbers, 
                        <italic toggle="yes">Masters of Data Science &amp; Department of Statistics, University of British Columbia</italic>
</p>
                    <p>Galen Wright, 
                        <italic toggle="yes">Centre for Molecular Medicine and Therapeutics, BC Children's Hospital Research Institute, University of British Columbia</italic>
</p>
                    <p>AB, BD, BG, AL, SL, ASG and GW wrote the first draft of the manuscript. All authors were involved in the revision of the manuscript and agreed to the final version.</p>
                </fn>
                <fn fn-type="conflict">
                    <p>
                        <bold>Competing interests: </bold>No competing interests were disclosed.</p>
                </fn>
            </author-notes>
            <pub-date pub-type="epub">
                <day>28</day>
                <month>2</month>
                <year>2017</year>
            </pub-date>
            <pub-date pub-type="collection">
                <year>2017</year>
            </pub-date>
            <volume>6</volume>
            <elocation-id>197</elocation-id>
            <history>
                <date date-type="accepted">
                    <day>24</day>
                    <month>2</month>
                    <year>2017</year>
                </date>
            </history>
            <permissions>
                <copyright-statement>Copyright: &#x00a9; 2017 hackseq Organizing Committee 2016</copyright-statement>
                <copyright-year>2017</copyright-year>
                <license xlink:href="https://creativecommons.org/licenses/by/4.0/">
                    <license-p>This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.</license-p>
                </license>
            </permissions>
            <self-uri content-type="pdf" xlink:href="https://f1000research.com/articles/6-197/pdf"/>
            <abstract>
                <p>The first instance of 
                    <italic toggle="yes">hackseq</italic> (
                    <ext-link ext-link-type="uri" xlink:href="http://www.hackseq.com">http://www.hackseq.com</ext-link>) was a genomics hackathon with the aim of bringing together a diverse set of biological and computational scientists to work on collaborative bioinformatics projects. In October 2016, 66 participants from nine nations came together for three days for 
                    <italic toggle="yes">hackseq</italic> and collaborated on nine projects ranging from data visualization to algorithm development. The response from participants was overwhelmingly positive with 100% (n = 54) of survey respondents saying they would like to participate in future hackathons. We detail key steps for others interested in organizing a successful hackathon and report excerpts from each project.</p>
            </abstract>
            <kwd-group kwd-group-type="author">
                <kwd>Hackathon</kwd>
                <kwd>Genomics</kwd>
                <kwd>Bioinformatics</kwd>
                <kwd>Open Science</kwd>
                <kwd>Diversity in Science</kwd>
            </kwd-group>
            <funding-group>
                <funding-statement>The author(s) declared that no grants were involved in supporting this work.</funding-statement>
            </funding-group>
        </article-meta>
    </front>
    <body>
        <sec sec-type="intro">
            <title>Introduction</title>
            <p>Technological advances in the biological sciences have led to an increasing availability of so-called &#x2018;-omic&#x2019; datasets, allowing fundamental questions in biology to be answered at an unprecedented rate
                <sup>
                    <xref ref-type="bibr" rid="ref-1">1</xref>
                </sup>. However, these datasets are complex, requiring novel and specialized informatics tools for proper analysis and to overcome the computational bottleneck in research. Open-source bioinformatics tools and pipeline development accelerates the rate of research by allowing the community to both reuse and thoroughly assess such methods. Thus, by solving biological problems in an open and collaborative manner, the field can progress at a faster rate than if code remains unavailable to the larger community
                <sup>
                    <xref ref-type="bibr" rid="ref-2">2</xref>
                </sup>.</p>
            <p>Hackathons offer a solution to catalyze tool and pipeline development for biological data science, as well as foster interdisciplinary collaborations
                <sup>
                    <xref ref-type="bibr" rid="ref-3">3</xref>
                </sup>. These events aim to solve defined computational problems over a period of days by bringing together small teams of individuals with different and diverse skillsets. Although frequently valuable for the outputs they generate, hackathons have faced criticism due to low levels of diversity amongst participants
                <sup>
                    <xref ref-type="bibr" rid="ref-4">4</xref>
                </sup>. We therefore established 
                <italic toggle="yes">hackseq</italic>, a genomics hackathon collective (
                <ext-link ext-link-type="uri" xlink:href="http://www.hackseq.com">http://www.hackseq.com</ext-link>) that aims to promote open science, collaboration and diversity. We placed special emphasis in promoting leadership amongst women, minorities and early-career scientists. The inaugural 
                <italic toggle="yes">hackseq</italic> event took place over three days in October 2016 in Vancouver (British Columbia, Canada) and was a satellite event to the annual American Society of Human Genetics (ASHG) meeting. Here we report a summary of this hackathon in the hopes of promoting similar events in the future.</p>
        </sec>
        <sec>
            <title>

                <italic toggle="yes">hackseq</italic> format</title>
            <p>

                <italic toggle="yes">hackseq</italic> was the first genomics hackathon in Vancouver and was based on the NCBI hackathon format
                <sup>
                    <xref ref-type="bibr" rid="ref-3">3</xref>
                </sup>. Some hackathons can be perceived as high-pressure events exclusive to technically inclined and experienced individuals. We thus took measures to ensure that people of all skill levels and backgrounds were encouraged to apply. We structured 
                <italic toggle="yes">hackseq</italic> as a three-day event that runs primarily from 8AM &#x2013; 5PM on the Saturday/Sunday/Monday prior to the 2016 ASHG meeting. The 
                <italic toggle="yes">hackseq</italic> itinerary is accessible on the 
                <italic toggle="yes">hackseq</italic> github repository (
                <ext-link ext-link-type="uri" xlink:href="https://github.com/hackseq/October_2016/blob/master/hackseq_2016_schedule.md">https://github.com/hackseq/October_2016/blob/master/hackseq_2016_schedule.md</ext-link>).</p>
            <p>First, we opened a call for &#x2018;team leaders&#x2019; to propose a project and lead a small team at 
                <italic toggle="yes">hackseq</italic> through social media, such as Twitter, making announcements at the Vancouver Bioinformatics User Group (VanBUG) and 
                <ext-link ext-link-type="uri" xlink:href="https://bioinformatics.ca">bioinformatics.ca</ext-link>, as well as direct email contact with potential leaders. We screened the projects to confirm that their aims and scope would be appropriate for a 72 hour hackathon. For the ten accepted projects we used GitHub as a discussion board, creating issue threads (
                <ext-link ext-link-type="uri" xlink:href="https://github.com/hackseq/hackseq_projects_2016/issues">https://github.com/hackseq/hackseq_projects_2016/issues</ext-link>) for each project, allowing prospective participants to view and discuss the details about each project before applying to join a particular team.</p>
            <p>Once we established the ten 
                <italic toggle="yes">hackseq</italic> projects, we opened the call for participants. Our main goal in recruiting participants was to reach out to a diverse group of individuals and to promote participation of women, minorities and early-career scientists. To this end, we partnered with organizations, such as Society for Canadian Women in Science and Technology (SCWIST) and VanBUG, to attract local participants. To encourage early-career scientist involvement, we contacted undergraduate and graduate-level computational sciences and bioinformatics programs at regional universities. To reach the global scientific audience, we contacted several human genetics societies around the world asking them to email participant application information to their respective mailing lists.</p>
            <p>To promote economic diversity and lower the barrier to entry for international participants, we partnered with ASHG to create travel awards based on financial need and/or minority status. 
                <italic toggle="yes">hackseq</italic> had no registration fee. Lastly, we made announcements on Twitter, Galaxy Project&#x2019;s events calendar, and various international conferences, such as Bioinformatics Open Source Conference 2016 and the 13
                <sup>th</sup> International Congress of Human Genetics, leading up to the hackathon.</p>
            <p>In the participant application form, prospective participants ranked the top three projects on which they would like to work. Participants could apply for travel awards by ASHG and request child care, covered by our budget, to promote participation amongst parents. The organizing committee and the team leaders evaluated the applications based on not only their skill levels, but also their interests and passion for genomics. To ensure well-rounded teams, we considered both the project preferences and skill levels during the team assignment phase, ensuring a balance between novices and expert coders, and biological and computational expertise. All forms developed for 
                <italic toggle="yes">hackseq</italic> are available online (
                <ext-link ext-link-type="uri" xlink:href="https://github.com/hackseq/October_2016/blob/master/Forms.md">https://github.com/hackseq/October_2016/blob/master/Forms.md</ext-link>).</p>
            <p>By defining the projects and teams beforehand, participants got to know their team members and prepare technical infrastructure. Teams hit the ground running on the first day, beginning work unprompted by the organizers at 8AM of the first day.</p>
            <p>

                <italic toggle="yes">hackseq</italic> had 66 participants in attendance from nine nations, divided into nine teams ranging from 3 to 10 individuals. Of the accepted ten projects, two team leaders withdrew prior to the hackathon for personal reasons, and one popular project split into two teams, resulting in nine teams. The mode age-category was 30&#x2013;34 years old (62.5%) for team leaders, and 25&#x2013;29 years old (58%) for participants (
                <xref ref-type="fig" rid="f1">Figure 1A</xref>). Graduate students made up the largest fraction of participants with 48.2%, followed by academic staff (15.5%), industrial scientists (13.8%), undergraduates (10.3%), postdoctoral fellows (6.9%) and academic faculty (5.2%). Notably, the team leaders were more likely to be industry scientists (44.4%) or young faculty (22.2%) (
                <xref ref-type="fig" rid="f1">Figure 1B</xref>). In total, 22 of 62 (35.5%) participants identified as female and 40 as male. A total of 41% self-identified as Caucasian, 40% as Asian or Pacific Islander, and 19% as Arab, Latin American or unspecified (
                <xref ref-type="fig" rid="f1">Figures 1C and D</xref>).</p>
            <fig fig-type="figure" id="f1" orientation="portrait" position="float">
                <label>Figure 1. </label>
                <caption>
                    <title>Participant diversity at hackseq 16.</title>
                    <p>To measure diversity of hackseq participants, we asked team leaders and participants to self-report their (
                        <bold>A</bold>) age, (
                        <bold>B</bold>) current occupation, (
                        <bold>C</bold>) ethnicity and (
                        <bold>D</bold>) gender. Data is shown for team leaders (yellow) and participants (blue).</p>
                </caption>
                <graphic orientation="portrait" position="float" xlink:href="https://f1000research-files.f1000.com/manuscripts/11818/29953f30-c91d-4b93-a92e-a69306c3661e_figure1.gif"/>
            </fig>
            <supplementary-material id="DS0" orientation="portrait" position="float" xlink:href="https://f1000researchdata.s3.amazonaws.com/datasets/10964/4eef30a8-2980-4988-b149-c0e7c719f508_Dataset_1.csv">
                <label>
                    <italic toggle="yes">hackseq</italic> demographics</label>
                <caption>
                    <p>De-identified demographic data from 
                        <italic toggle="yes">hackseq</italic> participants in the pre-meeting survey/confirmation of attendance.</p>
                </caption>
            </supplementary-material>
            <supplementary-material id="DS1" orientation="portrait" position="float" xlink:href="https://f1000researchdata.s3.amazonaws.com/datasets/10964/0701d333-624c-476a-9fdf-0c35961ecee3_Dataset_2.csv">
                <label>Post-
                    <italic toggle="yes">hackseq</italic> survey responses</label>
                <caption>
                    <p>De-identified post-
                        <italic toggle="yes">hackseq</italic> survey response data for the figures.</p>
                </caption>
            </supplementary-material>
        </sec>
        <sec>
            <title>Technical and logistical requisites</title>
            <p>Hackathons have little essential resource requirements. In this section, we outline the core logistics and technical infrastructure we employed. While these requisites could be stripped down, our experience was that attention and planning for these details maximized the efficiency of our teams to focus on coding and development.</p>
            <sec>
                <title>Core logistics</title>
                <p>To encourage participation, 
                    <italic toggle="yes">hackseq</italic> had no entry-cost for participants. To ensure teams could focus on the hackathon and not technical or logistical issues, we secured funding for the venue, technical infrastructure, food, transportation and stationary by partnering with different organizations.</p>
                <p>A sponsorship package was created to approach different academic, non -profit and industry organizations. Besides asking for financial support, we also made communication and marketing requests given that one of 
                    <italic toggle="yes">hackseq&#x2019;s</italic> goals was to recruit a diverse pool of participants. A strong emphasis was placed on women&#x2019;s groups in science and technology.</p>
                <p>In November 2015, we contacted ASHG to ask if we could be a satellite event for their meeting. Given that ASHG 2016 conference was planned to be held in Vancouver, 
                    <italic toggle="yes">hackseq</italic> gained exposure from the ASHG's communication strategy. The ASHG also provided three travel grants to participants based on financial need and diversity.</p>
                <p>These partnerships allowed 
                    <italic toggle="yes">hackseq</italic> to take place in a large, bright atrium at the University of British Columbia. This allowed all the teams to be in a single-space and interact with one another. Food was provided to minimize distraction and two social events were hosted, one the first night and one on the last night to encourage collaboration and networking amongst participants.</p>
            </sec>
            <sec>
                <title>Technical infrastructure</title>
                <p>Reliable technical infrastructure is necessary for organizing a successful hackathon; primarily, electrical power, Internet access and computing resources. We ensured the venue had adequate electrical outlets for the participants&#x2019; laptops and organized a dedicated Wi-Fi network connection be established for the event through the university's information technology office.</p>
                <p>Unlike many hackathons, 
                    <italic toggle="yes">hackseq</italic> was not restricted to coding. It also included genomic data analyses, which required additional computational resources. To promote reproducibility and collaboration, all the projects were based on pre-organized GitHub teams and repositories (see 
                    <italic toggle="yes">hackseq</italic> organization on Github; 
                    <ext-link ext-link-type="uri" xlink:href="https://github.com/hackseq">https://github.com/hackseq</ext-link>). To provide teams with reliable and powerful computation, we secured in-kind donation of cloud computing from Amazon Web Services Elastic Compute Cloud (AWS-EC2), and Canada&#x2019;s Michael Smith Genome Science Centre genOmics Research Container Architecture (ORCA). We used Linuxbrew, a cross-platform package management tool, to install bioinformatics software on ORCA
                    <sup>
                        <xref ref-type="bibr" rid="ref-5">5</xref>
                    </sup>.</p>
                <p>There was an equal usage of AWS-EC2 and ORCA amongst the participants (43%, not mutually exclusive) and an additional 12% using high-performance computing resources from their resident institutions. Users showed a preference for resources they were previously experienced with, and reported that it was not feasible to learn to use a new computing resource in the given time. Allowing team leaders and participants access to computing resources ahead of time in the future to &#x2018;experiment&#x2019; and familiarize themselves with the different resources is advisable.</p>
                <p>Each team chose which programming language and software they used. The majority of participants relied on Python (82.6%) and R (53.8%) programming languages and also used specialized software that related to their particular projects (
                    <xref ref-type="fig" rid="f2">Figure 2</xref>).</p>
                <fig fig-type="figure" id="f2" orientation="portrait" position="float">
                    <label>Figure 2. </label>
                    <caption>
                        <title>Software usage during hackseq 16.</title>
                        <p>At the conclusion of hackseq, we asked participants to complete a survey on their experience at hackseq. There were 52 responses to the question, &#x201c;Which programming languages and tools did you and your team use during the course of hackseq? (Comma delimited please).&#x201d; These responses were parsed and the number of unique instances is reported. Languages or software listed &lt;2 is reported as &#x2018;Other&#x2019;.</p>
                    </caption>
                    <graphic orientation="portrait" position="float" xlink:href="https://f1000research-files.f1000.com/manuscripts/11818/29953f30-c91d-4b93-a92e-a69306c3661e_figure2.gif"/>
                </fig>
                <p>In summary, the infrastructure requisites for running a successful hackathon are minimal and many can be acquired as in-kind donations from related organizations. In highlighting the essentials and key lessons, we hope to encourage the motivated reader to run a local scientific hackathon.</p>
            </sec>
        </sec>
        <sec>
            <title>Research project summaries</title>
            <p>The projects undertaken during 
                <italic toggle="yes">hackseq</italic> were from diverse fields within bioinformatics, ranging from human genomic variation analysis, microbial ecology and transcriptomics, to bioinformatic algorithm development. The projects were proposed by the team leaders, who defined the scope of the work, with the idea that at the end of the 72 hours there will have developed a working prototype. Here we provide a brief summaries from the projects. Scientific abstracts, videos of final presentations and updated information on each project can be found at 
                <ext-link ext-link-type="uri" xlink:href="https://www.hackseq.com/projects16">www.hackseq.com/projects16</ext-link>.</p>
            <sec>
                <title>VASCO: Visualization app for single cell exploration (led by Grace X.Y. Zheng)</title>
                <p>Modern transcriptomics analysis tools have limited capacity for analyzing thousands of single-cell RNA-sequencing data (scRNA-seq). VASCO is an intuitive user-interface to visualize gene-cell expression and cell clustering data to explore the relationship between populations of cells and gene expression, including cell cluster of differentiation markers (CD-markers). This project was awarded the &#x201c;People&#x2019;s Choice&#x201d; for the most outstanding project developed at 
                    <italic toggle="yes">hackseq</italic>.</p>
            </sec>
            <sec>
                <title>XYalign: Hacking sex chromosome variation (led by Melissa A. Wilson Sayres)</title>
                <p>Human sex chromosomes violate typical ploidy assumptions made for NGS autosome copy number and variant measurement, which is further confounded by mis-alignment between the X and Y chromosomes. XYalign was developed to measure sex chromosome ploidy and remap reads based on the inferred sex for downstream analysis.</p>
            </sec>
            <sec>
                <title>ParetoParrot: A tool to optimize the parameters of command line software (led by Shaun Jackman)</title>
                <p>Many bioinformatics software, such as genome sequence alignment and assembly, requires optimization of several input parameters to maximize a target metric. ParetoParrot measured the performance of several &#x2018;black-box&#x2019; optimization algorithms to improve the performance of genome sequence assembly software.</p>
            </sec>
            <sec>
                <title>BaklavaWGS: Pseudo-WGS variant calling for common cell types aggregating ChIP-seq, RNA-seq and DHS from ENCODE and Roadmap Epigenomics data (led by Luca Pinello)</title>
                <p>There is a wealth of sequencing datasets for cell types that have helped to understand and prioritize non-coding variants. Unfortunately, for many of those cell types we still don't have complete genotype information. BaklavaWGS recovers genotype data from cell lines aggregating sequencing data to aid downstream allele specific analysis. A preliminary analysis is available at 
                    <ext-link ext-link-type="uri" xlink:href="http://www.baklavawgs.com/">http://www.baklavawgs.com/</ext-link>.</p>
            </sec>
            <sec>
                <title>Evaluating epigenetic modifications in ChIP-seq and methylation data across cell types and states (led by Manuel Belmadani)</title>
                <p>A variety of datasets and approaches were investigated for analyzing cell type and state-specific genome regulation. The outcome of the experimental work in exploring differentially methylated regions from different epigenomic data and public databases, such as ENCODE ChIP-seq, IHEC and JASPAR, is presented.</p>
            </sec>
            <sec>
                <title>Selection of tag SNPs for an African SNP array by LD and haplotype based methods (led by Tommy Cartensen)</title>
                <p>Commercial SNP arrays fail to capture the diversity of African populations and limit the capacity to conduct large-scale medical genetic studies. Using African whole genome sequencing (WGS) data, an algorithm was developed to quickly identify SNP tags for this population. This will be used to improve upon SNP arrays for this richly diverse continent.</p>
            </sec>
            <sec>
                <title>Somatic mutation from separated haplotypes (SMUSH) (led by Patrick Marks)</title>
                <p>Calling somatic mutations relies on matched tumour and normal DNA sequencing, but a matched normal sample is often not available. The SMUSH algorithm was developed to differentiate wild type, germline and somatic mutations from linked-read DNA sequencing libraries.</p>
            </sec>
            <sec>
                <title>MetaGenius (led by Michael Schnall-Levin)</title>
                <p>Analysis of shotgun metagenomic sequencing data is limited in its capacity to assemble over homologous sequences. MetaGenius uses linked-read DNA sequencing to improve the assembly of a mixture of five bacterial species.</p>
            </sec>
            <sec>
                <title>mICP: Metagenomic indicator contig predictor (led by Ben Busby)</title>
                <p>Metagenomic sequencing has largely focused on 16S rRNA amplicons. This mICP strategy uses a mixture of long PacBio and short Illumina reads to identify contigs from environmental sequencing samples, which predict the environmental state from which they were found.</p>
            </sec>
        </sec>
        <sec sec-type="discussion">
            <title>Discussion</title>
            <p>The overarching themes of 
                <italic toggle="yes">hackseq</italic> were inclusivity, open science and collaboration. To gauge the extent to which we were successful in delivering on these themes, we performed a final survey at the conclusion of 
                <italic toggle="yes">hackseq</italic>. Participants overwhelmingly described their experience as positive (
                <xref ref-type="fig" rid="f3">Figure 3</xref>), with 100% (n = 54) of the survey respondents indicating that they would participate in an event like 
                <italic toggle="yes">hackseq</italic> again and a further 80% indicating that they would like to take on an organizational/leadership role in future 
                <italic toggle="yes">hackseq</italic> events. Participants specifically highlighted that 
                <italic toggle="yes">hackseq</italic> created ample recruitment, employment and collaborative opportunities, while also exposing participants to different datasets and analyses. We believe this reflects the underlying desire amongst young scientists to share, collaborate and learn from one another. They only need be given the opportunity to do so.</p>
            <fig fig-type="figure" id="f3" orientation="portrait" position="float">
                <label>Figure 3. </label>
                <caption>
                    <title>Quantification of subjective experience.</title>
                    <p>To measure the quality of the experience 
                        <italic toggle="yes">hackseq</italic> participants had after the event, we asked (
                        <bold>A</bold>) &#x201c;Please write three single word adjectives to describe your experience at hackseq?&#x201d; Responses were parsed and used to make a word-cloud (
                        <ext-link ext-link-type="uri" xlink:href="https://www.wordle.net">www.wordle.net</ext-link>), where the size of the word is proportional to the number of occurrences of that word in the survey responses. For scale, in 50 responses: &#x2018;fun&#x2019; was mentioned 26 times; &#x2018;exciting&#x2019; 6 times; and &#x2018;supercalifragilisticexpialidocious&#x2019; once. (
                        <bold>B</bold>) Additionally, we asked participants to rate four dimensions of their experience on a linear scale from 1 to 5. The kernel density of responses for these dimensions are shown, with a red dotted line showing the mean value of the responses.</p>
                </caption>
                <graphic orientation="portrait" position="float" xlink:href="https://f1000research-files.f1000.com/manuscripts/11818/29953f30-c91d-4b93-a92e-a69306c3661e_figure3.gif"/>
            </fig>
            <p>By organizing 
                <italic toggle="yes">hackseq</italic> as a satellite meeting of an international conference like ASHG, we were able to attract team leaders and participants from around the world, including a large proportion of young investigators and female participants (
                <xref ref-type="fig" rid="f1">Figure 1</xref>). There was a higher proportion of females at 
                <italic toggle="yes">hackseq</italic> (35.5%), than reported ratios at hackathons for which data is available, 20% at NASA&#x2019;s Space Apps Challenge (
                <ext-link ext-link-type="uri" xlink:href="https://www.fastcompany.com/3059036/most-creative-people/what-do-women-want-at-hackathons-nasa-has-a-list">https://www.fastcompany.com/3059036/most-creative-people/what-do-women-want-at-hackathons-nasa-has-a-list</ext-link>) or 15% at Spotify-organized hackathons (
                <ext-link ext-link-type="uri" xlink:href="https://labs.spotify.com/2015/01/13/diversify-how-we-created-a-hackathon-with-50-50-female-male-participants/">https://labs.spotify.com/2015/01/13/diversify-how-we-created-a-hackathon-with-50-50-female-male-participants/</ext-link>), which we believe to be a consequence of starting with a representative organizing committee and specifically encouraging female participation during recruitment.</p>
            <p>To further increase global representation at future 
                <italic toggle="yes">hackseq</italic> events, we recommend providing additional targeted travel awards or remote participation options to reduce proximity/cost restrictions. Further improvements could include educational resources to address common technical issues, the provision of an overnight area for participants who would like to continue to work after hours and additional activities to encourage interaction with members from different teams.</p>
        </sec>
        <sec sec-type="conclusions">
            <title>Conclusion</title>
            <p>The nature of biological sciences has shifted to an increasing emphasis on computational analysis. Collaborative events, such as 
                <italic toggle="yes">hackseq,</italic> offer an exciting platform to bring together a wide spectrum of scientists to work together and innovate. We present demographic information about the first 
                <italic toggle="yes">hackseq</italic> hackathon and encourage future organizers to do likewise, to quantify social inequalities that may be present in such events, and strive to achieve equal representation in the sciences. It&#x2019;s our hope that the information presented here will aid and encourage others in organizing genomics hackathons.</p>
        </sec>
        <sec>
            <title>Data availability</title>
            <p>The data referenced by this article are under copyright with the following copyright statement: Copyright: &#x00ef;&#x00bf;&#x00bd; 2017 hackseq Organizing Committee 2016</p>
            <p>Data associated with the article are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).
                <ext-link ext-link-type="uri" xlink:href="http://creativecommons.org/publicdomain/zero/1.0/"/>
            </p>
            <p>

                <bold>Dataset 1: 
                    <italic toggle="yes">hackseq</italic> demographics:</bold> De-identified demographic data from 
                <italic toggle="yes">hackseq</italic> participants in the pre-meeting survey/confirmation of attendance. doi, 
                <ext-link ext-link-type="uri" xlink:href="http://dx.doi.org/10.5256/f1000research.10964.d152802">10.5256/f1000research.10964.d152802</ext-link>
                <sup>
                    <xref ref-type="bibr" rid="ref-6">6</xref>
                </sup>
            </p>
            <p>

                <bold>Dataset 2: Post-
                    <italic toggle="yes">hackseq</italic> survey responses:</bold> De-identified post-
                <italic toggle="yes">hackseq</italic> survey response data for the figures. doi, 
                <ext-link ext-link-type="uri" xlink:href="http://dx.doi.org/10.5256/f1000research.10964.d152803">10.5256/f1000research.10964.d152803</ext-link>
                <sup>
                    <xref ref-type="bibr" rid="ref-7">7</xref>
                </sup>
            </p>
        </sec>
    </body>
    <back>
        <ack>
            <title>Acknowledgements</title>
            <p>We would first and foremost thank the 
                <italic toggle="yes">hackseq</italic> participants without which this event would not have happened. By team we&#x2019;d like to thank, Jean-Christophe Berube, Ogan Mancarci, Erin Marshall, Edward Mason, Celia Siu, Ben Weisburd, Shing Hei Zhan, 
                <bold>Grace X.Y. Zheng</bold>; Madeline Couse, Bruno Grande, Eric Karlins, Tanya Phung, Phillip Richmond, Timothy H. Webster, Whitney Whitford, 
                <bold>Melissa A. Wilson Sayres</bold>; Craig Glastonbury, Daisie Huang, Hamid Younesy, Jasleen Grewal, Laura Gutierrez Funderburk, Lisa Bang, 
                <bold>Shaun Jackman</bold>, Veera Manikandan Rajagopal, Y. Brian Lee; Carolyn Ch'ng, David Brazel, Karthigayini Sivaprakasam, Jill Moore, Shobhana Sekar, Stephen Kan, Jing Yun Alice Zhu, Ka-Kyung Kim, 
                <bold>Luca Pinello</bold>; Fotis Tsetsos, Kieran O'Neill, Shreejoy Tripathy, 
                <bold>Manuel Belmadani</bold>; Ayton Meintjes, Scott Hazelhurst, Vincent Montoya, Marcia MacDonald, Jocelyn Lee, Dan Fornika, Brian Lee, Austin Reynolds, 
                <bold>Tommy Carstensen</bold>; Amanjeev Sethi, Eric Zhao, Hua Ling, 
                <bold>Patrick Marks</bold>, Peng Zhang, Samantha Kohli; Erik Gafni, Dan Kvitek, Jake Lever and 
                <bold>Michael Schnall-Levin</bold>; 
                <bold>Ben Busby</bold>, Justin Chu, Jessica Hardwicke, Sean La and Feng Xu.</p>
            <p>We would like to thank our sponsorship partners 10X Genomics, ECOSCOPE, Amazon AWS, American Society of Human Genetics, Vancouver Tourism, Genome British Columbia, Association for Computing Machinery &#x2013; Women, Affymetrix, bioinformatics.ca, and GitHub. We also partnered with local organizations for logistical support: Society for Canadian Women in Science and Technology, BC Cancer Agency Graduate Student and Post-Doctoral Fellow Society, National Center for Biotechnology Information and the Vancouver Bioinformatics User Group. Sponsorship partners had no role in data collection and analaysis or preperation of the manuscript.</p>
        </ack>
        <ref-list>
            <ref id="ref-1">
                <label>1</label>
                <mixed-citation publication-type="journal">
                    <person-group person-group-type="author">

                        <name name-style="western">
                            <surname>Stephens</surname>
                            <given-names>ZD</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Lee</surname>
                            <given-names>SY</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Faghri</surname>
                            <given-names>F</given-names>
                        </name>

                        <etal/>
</person-group>:
                    <article-title>Big Data: Astronomical or Genomical?</article-title>
                    <source>

                        <italic toggle="yes">PLoS Biol.</italic>
</source>Public Library of Science;<year>2015</year>;<volume>13</volume>(<issue>7</issue>):<fpage>e1002195</fpage>.
                    <pub-id pub-id-type="pmid">26151137</pub-id>
                    <pub-id pub-id-type="doi">10.1371/journal.pbio.1002195</pub-id>
                    <pub-id pub-id-type="pmcid">4494865</pub-id>
                </mixed-citation>
            </ref>
            <ref id="ref-2">
                <label>2</label>
                <mixed-citation publication-type="journal">
                    <person-group person-group-type="author">

                        <name name-style="western">
                            <surname>Prins</surname>
                            <given-names>P</given-names>
                        </name>

                        <name name-style="western">
                            <surname>de Ligt</surname>
                            <given-names>J</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Tarasov</surname>
                            <given-names>A</given-names>
                        </name>

                        <etal/>
</person-group>:
                    <article-title>Toward effective software solutions for big biology.</article-title>
                    <source>

                        <italic toggle="yes">Nat Biotechnol.</italic>
</source>
                    <year>2015</year>;<volume>33</volume>(<issue>7</issue>):<fpage>686</fpage>&#x2013;<lpage>687</lpage>.
                    <pub-id pub-id-type="pmid">26154002</pub-id>
                    <pub-id pub-id-type="doi">10.1038/nbt.3240</pub-id>
                </mixed-citation>
            </ref>
            <ref id="ref-3">
                <label>3</label>
                <mixed-citation publication-type="journal">
                    <person-group person-group-type="author">

                        <name name-style="western">
                            <surname>Busby</surname>
                            <given-names>B</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Lesko</surname>
                            <given-names>M</given-names>
                        </name>

                        <collab>August 2015 and January 2016 Hackathon participants,</collab> 

                        <etal/>
</person-group>:
                    <article-title>Closing gaps between open software and public data in a hackathon setting: User-centered software prototyping [version 2; referees: not peer reviewed].</article-title>
                    <source>

                        <italic toggle="yes">F1000Res.</italic>
</source>
                    <year>2016</year>;<volume>5</volume>:<fpage>672</fpage>.
                    <pub-id pub-id-type="pmid">27134733</pub-id>
                    <pub-id pub-id-type="doi">10.12688/f1000research.8382.2</pub-id>
                    <pub-id pub-id-type="pmcid">4837979</pub-id>
                </mixed-citation>
            </ref>
            <ref id="ref-4">
                <label>4</label>
                <mixed-citation publication-type="journal">
                    <person-group person-group-type="author">

                        <name name-style="western">
                            <surname>Richard</surname>
                            <given-names>GT</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Kafai</surname>
                            <given-names>YB</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Adleberg</surname>
                            <given-names>B</given-names>
                        </name>

                        <etal/>
</person-group>:
                    <article-title>StitchFest: Diversifying a College Hackathon to Broaden Participation and Perceptions in Computing.</article-title>
                    <source>

                        <italic toggle="yes">Proceedings of the 46th ACM Technical Symposium on Computer Science Education - SIGCSE &#x2019; 15</italic>.</source>New York, USA: ACM Press;<year>2015</year>;<fpage>114</fpage>&#x2013;<lpage>119</lpage>.
                    <pub-id pub-id-type="doi">10.1145/2676723.2677310</pub-id>
                </mixed-citation>
            </ref>
            <ref id="ref-5">
                <label>5</label>
                <mixed-citation publication-type="journal">
                    <person-group person-group-type="author">

                        <name name-style="western">
                            <surname>Jackman</surname>
                            <given-names>S</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Birol</surname>
                            <given-names>I</given-names>
                        </name>						
</person-group>:
                    <article-title>Linuxbrew and Homebrew for cross-platform package management [version 1; not peer reviewed].</article-title>
                    <source>

                        <italic toggle="yes">F1000Res.</italic>
</source>
                    <year>2016</year>;<volume>5</volume>(<issue>ISCB Comm J</issue>):<fpage>1795 (poster)</fpage>.
                    <pub-id pub-id-type="doi">10.7490/f1000research.1112681.1</pub-id>
                </mixed-citation>
            </ref>
            <ref id="ref-6">
                <label>6</label>
                <mixed-citation publication-type="journal">
                    <collab>
                        <italic toggle="yes">hackseq</italic> Organising Committee (2016): </collab>
                    <article-title>Dataset 1 in: 
                        <italic toggle="yes">hackseq</italic>: Catalyzing collaboration between biological and computational scientists via hackathon.</article-title>
                    <source>

                        <italic toggle="yes">F1000Research.</italic>
</source>
                    <year>2017</year>.
                    <ext-link ext-link-type="uri" xlink:href="http://dx.doi.org/10.5256/f1000research.10964.d152802">Data Source</ext-link>
                </mixed-citation>
            </ref>
            <ref id="ref-7">
                <label>7</label>
                <mixed-citation publication-type="journal">
                    <collab>
                        <italic toggle="yes">hackseq</italic> Organising Committee (2016): </collab>
                    <article-title>Dataset 2 in: 
                        <italic toggle="yes">hackseq</italic>: Catalyzing collaboration between biological and computational scientists via hackathon.</article-title>
                    <source>

                        <italic toggle="yes">F1000Research.</italic>
</source>
                    <year>2017</year>.
                    <ext-link ext-link-type="uri" xlink:href="http://dx.doi.org/10.5256/f1000research.10964.d152803">Data Source</ext-link>
                </mixed-citation>
            </ref>
        </ref-list>
    </back>
    <sub-article article-type="reviewer-report" id="report20964">
        <front-stub>
            <article-id pub-id-type="doi">10.5256/f1000research.11818.r20964</article-id>
            <title-group>
                <article-title>Reviewer response for version 1</article-title>
            </title-group>
            <contrib-group>
                <contrib contrib-type="author">
                    <name>
                        <surname>Guo</surname>
                        <given-names>Jiarong</given-names>
                    </name>
                    <xref ref-type="aff" rid="r20964a1">1</xref>
                    <role>Referee</role>
                </contrib>
                <aff id="r20964a1">
                    <label>1</label>Center for Microbial Ecology, Michigan State University, East Lansing, MI, USA</aff>
            </contrib-group>
            <author-notes>
                <fn fn-type="conflict">
                    <p>
                        <bold>Competing interests: </bold>No competing interests were disclosed.</p>
                </fn>
            </author-notes>
            <pub-date pub-type="epub">
                <day>31</day>
                <month>3</month>
                <year>2017</year>
            </pub-date>
            <permissions>
                <copyright-statement>Copyright: &#x00a9; 2017 Guo J</copyright-statement>
                <copyright-year>2017</copyright-year>
                <license xlink:href="https://creativecommons.org/licenses/by/4.0/">
                    <license-p>This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.</license-p>
                </license>
            </permissions>
            <related-article ext-link-type="doi" id="relatedArticleReport20964" related-article-type="peer-reviewed-article" xlink:href="10.12688/f1000research.10964.1"/>
            <custom-meta-group>
                <custom-meta>
                    <meta-name>recommendation</meta-name>
                    <meta-value>approve</meta-value>
                </custom-meta>
            </custom-meta-group>
        </front-stub>
        <body>
            <p>The authors reported a detailed summary of their genomic hackathon, hackseq, to help those interested in organizing similar hackathons in future. The hackseq brought together 66 biological and computational scientists with diverse demographic background to collaborate on nine projects on genomics ranging from data visualization to algorithm development. All the participants had positive responses in post assessment and showed interests in future hackathons.</p>
            <p> </p>
            <p> The background is clearly articulated. There are detailed descriptions on hackseq format and technical and logistical requisites, which are useful for future hackathons. Brief research project summaries are also described with more information available on GitHub. The data for reproducing the figures are made available on F1000 and schedules and application forms are available on GitHub.</p>
            <p> </p>
            <p> 
                <bold>Major comments:</bold>
            </p>
            <p> Strengths:</p>
            <p> Overall, sharing details and experiences of the hackseq such as recruiting project leaders and participants, assigning teams, logistical and technical requisites, and post assessment is valuable for the open science community to organize future hackathons. 
                <list list-type="order">
                    <list-item>
                        <p>It is a great idea to organize&#x00a0;the hackathon as a satellite event of bigger events. The bigger events can help on travel cost of participants and more importantly promote hackathon participations.</p>
                    </list-item>
                    <list-item>
                        <p>This hackseq is very successful at recruiting diverse participants, because it has a representative committee that encourages female, minority and early career scientist participations and also good promotion strategies that it partners with organizations such as Society for Canadian Women in Science.</p>
                    </list-item>
                </list> Weakness:</p>
            <p> The team leaders seem to have a critical role in each project, but their roles and responsibilities during the hackathon are not clearly mentioned in the manuscripts.</p>
            <p> </p>
            <p> 
                <bold>Minor comments:&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0;&#x00a0; </bold>
            </p>
            <p> Last paragraph in &#x201c;Hackseq format&#x201d; section: it states that participants ranked top three projects on which they would like to work, but there are four projects of choice in the application form.</p>
            <p> First paragraph in &#x201c;Core logistics&#x201d; section: stationary -&gt; stationery.</p>
            <p> Second paragraph in &#x201c;Discussion&#x201d;: female percentage in biology is significantly higher than in engineering.&#x00a0;Thus the direct comparison of female participation rate of hackseq to engineering (NASA and Spotify) hackathons is not meaningful.</p>
            <p> Discussion: assessment is difficult with participants from diverse background. Some discussion on the current assessment and possible improvement would be useful.</p>
            <p>Reviewer Expertise:</p>
            <p>NA</p>
            <p>I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.</p>
        </body>
    </sub-article>
    <sub-article article-type="reviewer-report" id="report20626">
        <front-stub>
            <article-id pub-id-type="doi">10.5256/f1000research.11818.r20626</article-id>
            <title-group>
                <article-title>Reviewer response for version 1</article-title>
            </title-group>
            <contrib-group>
                <contrib contrib-type="author">
                    <name>
                        <surname>Hertweck</surname>
                        <given-names>Kate L.</given-names>
                    </name>
                    <xref ref-type="aff" rid="r20626a1">1</xref>
                    <role>Referee</role>
                    <uri content-type="orcid">https://orcid.org/0000-0002-4026-4612</uri>
                </contrib>
                <aff id="r20626a1">
                    <label>1</label>Department of Biology, The University of Texas at Tyler, Tyler, TX, USA</aff>
            </contrib-group>
            <author-notes>
                <fn fn-type="conflict">
                    <p>
                        <bold>Competing interests: </bold>No competing interests were disclosed.</p>
                </fn>
            </author-notes>
            <pub-date pub-type="epub">
                <day>31</day>
                <month>3</month>
                <year>2017</year>
            </pub-date>
            <permissions>
                <copyright-statement>Copyright: &#x00a9; 2017 Hertweck KL</copyright-statement>
                <copyright-year>2017</copyright-year>
                <license xlink:href="https://creativecommons.org/licenses/by/4.0/">
                    <license-p>This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.</license-p>
                </license>
            </permissions>
            <related-article ext-link-type="doi" id="relatedArticleReport20626" related-article-type="peer-reviewed-article" xlink:href="10.12688/f1000research.10964.1"/>
            <custom-meta-group>
                <custom-meta>
                    <meta-name>recommendation</meta-name>
                    <meta-value>approve</meta-value>
                </custom-meta>
            </custom-meta-group>
        </front-stub>
        <body>
            <p>Thank you to the authors for writing a summary of what seems to be a very successful collaborative coding event, with this manuscript in particular focused on preparation for the event, managing logistic concerns during the event, and an overview of the projects supported. The manuscript is quite well written, and I have no concerns about the content presented therein.</p>
            <p> </p>
            <p> I especially appreciate the recommendations for how to solicit diverse leaders/participants, engage with partner organizations, and carefully craft a sense of community among attendees. Moreover, the authors include suggestions on how to improve similar events in the future. The data reported here provide an important context for comparison for events which continue to encourage participation from underrepresented groups.&#x00a0;</p>
            <p> </p>
            <p> Although not highlighted in the paper, the itinerary for the three-day meeting described here includes a number of additional details which would be useful to other coding event organizers. For example, while the majority of meeting time was dedicated to team work, extra workshops and talks were offerred (e.g., introduction to git) that would be encourage skills development for students or other participants new to the field. I'll be interested to see whether this model for hosting a hackathon continues at ASHG or other meetings.</p>
            <p>Reviewer Expertise:</p>
            <p>NA</p>
            <p>I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.</p>
        </body>
    </sub-article>
</article>
