<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Publishing DTD v1.2 20190208//EN" "http://jats.nlm.nih.gov/publishing/1.2/JATS-journalpublishing1.dtd"><article xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" article-type="other" dtd-version="1.2" xml:lang="en">
    <front>
        <journal-meta>
            <journal-id journal-id-type="pmc">F1000Research</journal-id>
            <journal-title-group>
                <journal-title>F1000Research</journal-title>
            </journal-title-group>
            <issn pub-type="epub">2046-1402</issn>
            <publisher>
                <publisher-name>F1000 Research Limited</publisher-name>
                <publisher-loc>London, UK</publisher-loc>
            </publisher>
        </journal-meta>
        <article-meta>
            <article-id pub-id-type="doi">10.12688/f1000research.12914.1</article-id>
            <article-categories>
                <subj-group subj-group-type="heading">
                    <subject>Research Note</subject>
                </subj-group>
                <subj-group>
                    <subject>Articles</subject>
                    <subj-group>
                        <subject>Toxicology</subject>
                    </subj-group>
                    <subj-group>
                        <subject>Web and Social Media</subject>
                    </subj-group>
                </subj-group>
            </article-categories>
            <title-group>
                <article-title>Detection of illicit online sales of fentanyls via Twitter</article-title>
                <fn-group content-type="pub-status">
                    <fn>
                        <p>[version 1; peer review: 3 approved]</p>
                    </fn>
                </fn-group>
            </title-group>
            <contrib-group>
                <contrib contrib-type="author" corresp="yes">
                    <name>
                        <surname>Mackey</surname>
                        <given-names>Tim K.</given-names>
                    </name>
                    <role content-type="http://credit.niso.org/">Conceptualization</role>
                    <role content-type="http://credit.niso.org/">Data Curation</role>
                    <role content-type="http://credit.niso.org/">Formal Analysis</role>
                    <role content-type="http://credit.niso.org/">Funding Acquisition</role>
                    <role content-type="http://credit.niso.org/">Investigation</role>
                    <role content-type="http://credit.niso.org/">Methodology</role>
                    <role content-type="http://credit.niso.org/">Project Administration</role>
                    <role content-type="http://credit.niso.org/">Resources</role>
                    <role content-type="http://credit.niso.org/">Software</role>
                    <role content-type="http://credit.niso.org/">Supervision</role>
                    <role content-type="http://credit.niso.org/">Validation</role>
                    <role content-type="http://credit.niso.org/">Visualization</role>
                    <role content-type="http://credit.niso.org/">Writing &#x2013; Original Draft Preparation</role>
                    <role content-type="http://credit.niso.org/">Writing &#x2013; Review &amp; Editing</role>
                    <uri content-type="orcid">https://orcid.org/0000-0002-2191-7833</uri>
                    <xref ref-type="corresp" rid="c1">a</xref>
                    <xref ref-type="aff" rid="a1">1</xref>
                    <xref ref-type="aff" rid="a2">2</xref>
                    <xref ref-type="aff" rid="a3">3</xref>
                </contrib>
                <contrib contrib-type="author" corresp="no">
                    <name>
                        <surname>Kalyanam</surname>
                        <given-names>Janani</given-names>
                    </name>
                    <role content-type="http://credit.niso.org/">Conceptualization</role>
                    <role content-type="http://credit.niso.org/">Data Curation</role>
                    <role content-type="http://credit.niso.org/">Formal Analysis</role>
                    <role content-type="http://credit.niso.org/">Funding Acquisition</role>
                    <role content-type="http://credit.niso.org/">Investigation</role>
                    <role content-type="http://credit.niso.org/">Methodology</role>
                    <role content-type="http://credit.niso.org/">Project Administration</role>
                    <role content-type="http://credit.niso.org/">Resources</role>
                    <role content-type="http://credit.niso.org/">Software</role>
                    <role content-type="http://credit.niso.org/">Supervision</role>
                    <role content-type="http://credit.niso.org/">Validation</role>
                    <role content-type="http://credit.niso.org/">Visualization</role>
                    <role content-type="http://credit.niso.org/">Writing &#x2013; Original Draft Preparation</role>
                    <role content-type="http://credit.niso.org/">Writing &#x2013; Review &amp; Editing</role>
                    <xref ref-type="aff" rid="a3">3</xref>
                    <xref ref-type="aff" rid="a4">4</xref>
                </contrib>
                <aff id="a1">
                    <label>1</label>Department of Anesthesiology, University of California San Diego School of Medicine, San Diego, CA, 92093, USA</aff>
                <aff id="a2">
                    <label>2</label>Division of Global Public Health, University of California, San Diego School of Medicine, San Diego, CA, 92093, USA</aff>
                <aff id="a3">
                    <label>3</label>Global Health Policy Institute, San Diego, CA, USA</aff>
                <aff id="a4">
                    <label>4</label>Department of Electrical and Computer Engineering, University of California, San Diego, San Diego, CA, 92093, USA</aff>
            </contrib-group>
            <author-notes>
                <corresp id="c1">
                    <label>a</label>
                    <email xlink:href="mailto:tmackey@ucsd.edu">tmackey@ucsd.edu</email>
                </corresp>
                <fn fn-type="conflict">
                    <p>
                        <bold>Competing interests: </bold>TM is a non-compensated member of the academic advisory panel of the Alliance for Safe Online Pharmacies (ASOP), a 501(c)(4) social welfare organization engaged on the issue of illicit online pharmacies. There was no involvement of anyone other than the authors in the conception, design, collection, planning, conduct, analysis, interpretation, writing, and discussion to submit this work.  Authors report no other financial relationships with any organizations that might have an interest in the submitted work.</p>
                </fn>
            </author-notes>
            <pub-date pub-type="epub">
                <day>2</day>
                <month>11</month>
                <year>2017</year>
            </pub-date>
            <pub-date pub-type="collection">
                <year>2017</year>
            </pub-date>
            <volume>6</volume>
            <elocation-id>1937</elocation-id>
            <history>
                <date date-type="accepted">
                    <day>19</day>
                    <month>6</month>
                    <year>2026</year>
                </date>
            </history>
            <permissions>
                <copyright-statement>Copyright: &#x00a9; 2017 Mackey TK and Kalyanam J</copyright-statement>
                <copyright-year>2017</copyright-year>
                <license xlink:href="https://creativecommons.org/licenses/by/4.0/">
                    <license-p>This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.</license-p>
                </license>
            </permissions>
            <self-uri content-type="pdf" xlink:href="https://f1000research.com/articles/6-1937/pdf"/>
            <abstract>
                <p>A counterfeit fentanyl crisis is currently underway in the United States.  Counterfeit versions of commonly abused prescription drugs laced with fentanyl are being manufactured, distributed, and sold globally, leading to an increase in overdose and death in countries like the United States and Canada.  Despite concerns from the U.S. Drug Enforcement Agency regarding covert and overt sale of fentanyls online, no study has examined the role of the Internet and social media on fentanyl illegal marketing and direct-to-consumer access. In response, this study collected and analyzed five months of Twitter data (from June&#x2013;November 2015) filtered for the keyword &#x201c;fentanyl&#x201d; using Amazon Web Services.  We then analyzed 28,711 fentanyl-related tweets using text filtering and a machine learning approach called a Biterm Topic Model (BTM) to detect underlying latent patterns or &#x201c;topics&#x201d; present in the corpus of tweets.  Using this approach we detected a subset of 771 tweets marketing the sale of fentanyls online and then filtered this down to nine unique tweets containing hyperlinks to external websites.  Six hyperlinks were associated with online fentanyl classified ads, 2 with illicit online pharmacies, and 1 could not be classified due to traffic redirection.  Importantly, the one illicit online pharmacy detected was still accessible and offered the sale of fentanyls and other controlled substances direct-to-consumers with no prescription required at the time of publication of this study. Overall, we detected a relatively small sample of Tweets promoting illegal online sale of fentanyls.  However, the detection of even a few online sellers represents a public health danger and a direct violation of law that demands further study.</p>
            </abstract>
            <kwd-group kwd-group-type="author">
                <kwd>fentanyl</kwd>
                <kwd>prescription drug abuse</kwd>
                <kwd>digital surveillance</kwd>
                <kwd>fentanyl</kwd>
                <kwd>counterfeit</kwd>
                <kwd>Twitter</kwd>
                <kwd>social media</kwd>
                <kwd>internet pharmacies</kwd>
            </kwd-group>
            <funding-group>
                <funding-statement>TM received funding for the data collection phase of this study from an ASOP pilot research grant exploring prescription drug abuse risks online. Though this pilot research grant supported the overall process of capturing data, the aims of the grant were not related to this study.</funding-statement>
            </funding-group>
        </article-meta>
    </front>
    <body>
        <sec sec-type="intro">
            <title>Introduction</title>
            <p>A fentanyl crisis is currently underway in the United States, characterized by an influx of counterfeit fentanyl-laced prescription drugs (e.g. Xanax
                <sup>&#x00ae;</sup>, Norco
                <sup>&#x00ae;</sup>, OxyContin
                <sup>&#x00ae;</sup>, and Oxycodone) now being advertised, sold and consumed by the public
                <sup>
                    <xref ref-type="bibr" rid="ref-1">1</xref>
                </sup>. The result of this counterfeit infiltration into the U.S. drug supply chain has been an alarming increase in fentanyl-related overdose, deaths and seizures, due to illicitly produced products containing undeclared fentanyls and related fentanyl analogues/variants (e.g. acetyl fentanyl, butyrfentanyl, and furanylfentanyl.)
                <sup>
                    <xref ref-type="bibr" rid="ref-1">1</xref>&#x2013;
                    <xref ref-type="bibr" rid="ref-3">3</xref>
                </sup>
            </p>
            <p>Fueling this public health emergency is a global network of fentanyl producers and traffickers including countries such as China (where powdered fentanyls and other synthetic opiates are sold with pill presses or sold as precursors used to manufacture fentanyl), Mexico (where fentanyls are manufactured in clandestine laboratories and smuggled into the USA) and Canada (where fentanyls are also manufactured and sold locally)
                <sup>
                    <xref ref-type="bibr" rid="ref-1">1</xref>
                </sup>.  Unfortunately, criminals see an opportunity to sell cheap and deadly versions of fentanyl-laced pills to satiate demand driven by a national prescription opioid and heroin epidemic, with the U.S. Drug Enforcement Agency (DEA) estimating that a kilogram of fentanyl could generate $5&#x2013;20 million in retail counterfeit sales
                <sup>
                    <xref ref-type="bibr" rid="ref-1">1</xref>
                </sup>.</p>
            <p>Other studies have explored the public health consequences of counterfeit fentanyl and have argued for better surveillance and harm reduction approaches to curb consumption and demand
                <sup>
                    <xref ref-type="bibr" rid="ref-2">2</xref>,
                    <xref ref-type="bibr" rid="ref-3">3</xref>
                </sup>.  However, the potential impact and easy accessibility of fentanyls sold direct-to-consumer via the Internet and social media has not been examined.  Other studies have identified illegal marketing and direct sales of other prescription controlled substances by illicit online pharmacies and the use of popular social media sites to promote these services
                <sup>
                    <xref ref-type="bibr" rid="ref-4">4</xref>&#x2013;
                    <xref ref-type="bibr" rid="ref-10">10</xref>
                </sup>.</p>
            <p>No study has assessed the role of Internet in fentanyl product promotion, despite concerns from the DEA about documented covert and overt sale of fentanyls online
                <sup>
                    <xref ref-type="bibr" rid="ref-1">1</xref>
                </sup>.  To better characterize these risks, we conducted a social media surveillance study using big data approaches to identify fentanyl promotion and sale via the popular microblogging platform Twitter. </p>
        </sec>
        <sec sec-type="methods">
            <title>Methods</title>
            <p>Our data collection was part of a larger study exploring user behavior characteristics of illicit prescription drug abuse mediated by Twitter, with tweets collected over a five-month period (June&#x2013;November 2015) filtered for the prescription drug abuse-related keywords, including the term &#x201c;fentanyl.&#x201d;
                <sup>
                    <xref ref-type="bibr" rid="ref-5">5</xref>,
                    <xref ref-type="bibr" rid="ref-6">6</xref>,
                    <xref ref-type="bibr" rid="ref-10">10</xref>
                </sup> Data collection and analysis was carried out using a methodology combining Amazon Web Services (AWS) cloud computing virtual instances for data collection and machine learning algorithms to analyze tweets using both assisted and unassisted protocols used in previously published studies
                <sup>
                    <xref ref-type="bibr" rid="ref-5">5</xref>,
                    <xref ref-type="bibr" rid="ref-6">6</xref>,
                    <xref ref-type="bibr" rid="ref-10">10</xref>
                </sup>.</p>
            <p>Specifically, we used a Biterm Topic Model (BTM) that can identify underlying latent patterns or &#x201c;topics&#x201d; present in a large corpus of tweets
                <sup>
                    <xref ref-type="bibr" rid="ref-6">6</xref>,
                    <xref ref-type="bibr" rid="ref-11">11</xref>
                </sup>. Given the input corpus of text, and a predefined number k, the model outputs a set of k latent topics present in the corpus.  Each topic represents an underlying pattern associated with fentanyl twitter conversations that were then isolated and coded using human annotation.  </p>
            <p>The number of topics to be detected was set to 40 and the alpha and beta parameters of BTM were set to 1 and 0.01 respectively.  The output of the BTM was set to display the top 10 words with the highest weight for each topic.  Each topic was manually reviewed based on these words and classified into &#x201c;news&#x201d; and &#x201c;online pharmacy&#x201d; themes (results discussed below).  We also used a keyword filtering process using common terms included in the text of tweets associated with illegal online drug promotion, also known as &#x201c;selling arguments,&#x201d; (e.g. &#x201c;buy&#x201d;, &#x201c;discount&#x201d;, &#x201c;price&#x201d;) in conjunction with the BTM outputs. Code used in this study is available from 
                <ext-link ext-link-type="uri" xlink:href="https://github.com/kjanani/health_topicmodeling">https://github.com/kjanani/health_topicmodeling</ext-link>.</p>
        </sec>
        <sec sec-type="results">
            <title>Results</title>
            <p>Our study collected 28,711 fentanyl-related tweets during 2015, a period when the fentanyl crisis was escalating
                <sup>
                    <xref ref-type="bibr" rid="ref-6">6</xref>,
                    <xref ref-type="bibr" rid="ref-11">11</xref>
                </sup>. The majority of topics detected in the whole corpus of fentanyl-related tweets were news-related (97.3%, n=27,940), detailing counterfeit fentanyl dangers reported in national and local media outlets.</p>
            <p>After isolating Tweets related to news reports, we used a keyword filtering process in conjunction with machine learning to detect the subset of tweets associated with illegal online drug promotion.  Using this approach, we detected 771 (&lt;1% of total) tweets promoting the marketing and sale of fentanyls and other controlled substances online. These tweets were then manually annotated by the authors (inter-coder reliability kappa=0.98) to assess if they included a hyperlink enabling direct-to-consumer sale and purchase of illegal fentanyls.  Nine unique tweets (not duplicates or retweets) and their associated hyperlinks were then identified for further analysis.</p>
            <p>When examining website content of hyperlinks, 6 were associated with online classified ads, 2 with illicit online pharmacies, and 1 could not be classified due to traffic redirection.  These results indicate some interesting trends.  First, it appears that individual drug dealers use online classifieds ads to digitally advertise &#x201c;street buys&#x201d; of controlled substances (Example A, 
                <xref ref-type="fig" rid="f1">Figure 1</xref>). Additionally, one illicit online pharmacy that we detected and which is currently was still accessible at the time of initial publication of this study. offers the sale of fentanyls and other controlled substances direct-to-consumers with no prescription required (Example B).  Its website purportedly offers Abstral&#x00ae; 800mg (fentanyl brand name) for $3.00 per tablet, and based on further inspection of 
                <ext-link ext-link-type="uri" xlink:href="https://www.whois.net/">WHOIS data</ext-link>, has its internet domain registration identity and location masked by an Internet privacy service company.</p>
            <fig fig-type="figure" id="f1" orientation="portrait" position="float">
                <label>Figure 1. </label>
                <caption>
                    <title>Screenshots of fentanyl promotion and sale via Twitter.</title>
                </caption>
                <graphic orientation="portrait" position="float" xlink:href="https://f1000research-files.f1000.com/manuscripts/14002/96b93fca-4a88-4c90-bf6f-40af4c3ce17e_figure1.gif"/>
            </fig>
            <p>We note that there are certain limitations to this study.  First the study only examined online promotion and availability at a single point-of-time, as websites of this nature are often removed or become inactive.  Additionally, because of the illegal nature of the websites identified, we did not purchase fentanyls and test them for authenticity and potency.  Buying controlled substances and making payments to an illicit online pharmacy raises serious legal concerns and is generally illegal.</p>
        </sec>
        <sec sec-type="discussion">
            <title>Discussion</title>
            <p>Our results indicate that in our entire corpus of fentanyl-related tweets, direct-to-consumer sale of fentanyls occurred infrequently.  This is not surprising, given the deadly nature and high potency of fentanyl, and that many victims may not actively seek its purchase
                <sup>
                    <xref ref-type="bibr" rid="ref-1">1</xref>,
                    <xref ref-type="bibr" rid="ref-2">2</xref>
                </sup>. However, the presence of even a few online sellers is concerning, as these actions represent a clear violation of Federal law (e.g. Controlled Substances Act and the Ryan Haight Online Pharmacy Consumer Protection Act) and directly endanger the public.  Importantly, these sites may only be the tip of the iceberg, as hyperlinks in one result directed us to online user forums (Google Groups) actively selling fentanyl online (Example C). </p>
            <p>Our data collection occurred in late 2015, a period arguably at an early stage of the fentanyl crisis epidemiological curve, which may limit the generalizability of the results.  Hence, this indicates that more research is needed to accurately characterize the dangers of the online environment in relation to counterfeit fentanyls.  This includes longer term follow-up studies examining online promotion and access to fentanyl and interrogating other online platforms (including other social media sites, chat rooms/user forums, the deep web, and private sales facilitated by online and mobile technology).</p>
        </sec>
        <sec>
            <title>Data availability</title>
            <p>Raw datasets have not been made available per concerns regarding user information and confidentiality of publicly available processed data. This data is stored at the Global Health Policy Institute and is available upon request in a de-identified and aggregated dataset. Please contact the institute&#x2019;s general contact email (
                <ext-link ext-link-type="uri" xlink:href="mailto:ghpolicyinstitute@gmail.com">ghpolicyinstitute@gmail.com</ext-link>) for further information.</p>
            <p>The code for BTM can be found at 
                <ext-link ext-link-type="uri" xlink:href="https://github.com/kjanani/health_topicmodeling">https://github.com/kjanani/health_topicmodeling</ext-link>.</p>
            <p>Archived code: 
                <ext-link ext-link-type="uri" xlink:href="http://dx.doi.org/10.5281/zenodo.1038177">https://doi.org/10.5281/zenodo.1038177</ext-link>
                <sup>
                    <xref ref-type="bibr" rid="ref-12">12</xref>
                </sup>
            </p>
        </sec>
    </body>
    <back>
        <ref-list>
            <ref id="ref-1">
                <label>1</label>
                <mixed-citation publication-type="journal">
                    <collab>DEA</collab>:
                    <article-title>Counterfeit Prescription Pills Containing Fentanyls: A Global Threat [Internet]</article-title>. dea.gov.<year>2016</year>; [cited 2017 Feb 17].
                    <ext-link ext-link-type="uri" xlink:href="https://www.dea.gov/docs/Counterfeit%20Prescription%20Pills.pdf">Reference Source</ext-link>
                </mixed-citation>
            </ref>
            <ref id="ref-2">
                <label>2</label>
                <mixed-citation publication-type="journal">
                    <person-group person-group-type="author">

                        <name name-style="western">
                            <surname>Green</surname>
                            <given-names>TC</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Gilbert</surname>
                            <given-names>M</given-names>
                        </name>
</person-group>:
                    <article-title>Counterfeit Medications and Fentanyl.</article-title>
                    <source>

                        <italic toggle="yes">JAMA Intern Med.</italic>
</source>American Medical Association,<year>2016</year>;<volume>176</volume>(<issue>10</issue>):<fpage>1555</fpage>&#x2013;<lpage>7</lpage>.
                    <pub-id pub-id-type="pmid">27533891</pub-id>
                    <pub-id pub-id-type="doi">10.1001/jamainternmed.2016.4310</pub-id>
                </mixed-citation>
            </ref>
            <ref id="ref-3">
                <label>3</label>
                <mixed-citation publication-type="journal">
                    <person-group person-group-type="author">

                        <name name-style="western">
                            <surname>Frank</surname>
                            <given-names>RG</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Pollack</surname>
                            <given-names>HA</given-names>
                        </name>
</person-group>:
                    <article-title>Addressing the Fentanyl Threat to Public Health.</article-title>
                    <source>

                        <italic toggle="yes">N Engl J Med.</italic>
</source>
                    <year>2017</year>;<volume>376</volume>(<issue>7</issue>):<fpage>605</fpage>&#x2013;<lpage>7</lpage>.
                    <pub-id pub-id-type="pmid">28199808</pub-id>
                    <pub-id pub-id-type="doi">10.1056/NEJMp1615145</pub-id>
                </mixed-citation>
            </ref>
            <ref id="ref-4">
                <label>4</label>
                <mixed-citation publication-type="journal">
                    <person-group person-group-type="author">

                        <name name-style="western">
                            <surname>Mackey</surname>
                            <given-names>TK</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Liang</surname>
                            <given-names>BA</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Strathdee</surname>
                            <given-names>SA</given-names>
                        </name>
</person-group>:
                    <article-title>Digital social media, youth, and nonmedical use of prescription drugs: the need for reform.</article-title>
                    <source>

                        <italic toggle="yes">J Med Internet Res.</italic>
</source>
                    <year>2013</year>;<volume>15</volume>(<issue>7</issue>):<fpage>e143</fpage>.
                    <pub-id pub-id-type="pmid">23892156</pub-id>
                    <pub-id pub-id-type="doi">10.2196/jmir.2464</pub-id>
                    <pub-id pub-id-type="pmcid">3742396</pub-id>
                </mixed-citation>
            </ref>
            <ref id="ref-5">
                <label>5</label>
                <mixed-citation publication-type="journal">
                    <person-group person-group-type="author">

                        <name name-style="western">
                            <surname>Katsuki</surname>
                            <given-names>T</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Mackey</surname>
                            <given-names>TK</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Cuomo</surname>
                            <given-names>R</given-names>
                        </name>
</person-group>:
                    <article-title>Establishing a Link Between Prescription Drug Abuse and Illicit Online Pharmacies: Analysis of Twitter Data.</article-title>
                    <source>

                        <italic toggle="yes">J Med Internet Res.</italic>
</source>
                    <year>2015</year>;<volume>17</volume>(<issue>12</issue>):<fpage>e280</fpage>.
                    <pub-id pub-id-type="pmid">26677966</pub-id>
                    <pub-id pub-id-type="doi">10.2196/jmir.5144</pub-id>
                    <pub-id pub-id-type="pmcid">4704982</pub-id>
                </mixed-citation>
            </ref>
            <ref id="ref-6">
                <label>6</label>
                <mixed-citation publication-type="journal">
                    <person-group person-group-type="author">

                        <name name-style="western">
                            <surname>Kalyanam</surname>
                            <given-names>J</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Katsuki</surname>
                            <given-names>T</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Lanckriet</surname>
                            <given-names>G</given-names>
                        </name>

                        <etal/>
</person-group>:
                    <article-title>Exploring trends of nonmedical use of prescription drugs and polydrug abuse in the Twittersphere using unsupervised machine learning.</article-title>
                    <source>

                        <italic toggle="yes">Addict Behav.</italic>
</source>
                    <year>2017</year>;<volume>65</volume>:<fpage>289</fpage>&#x2013;<lpage>95</lpage>.
                    <pub-id pub-id-type="pmid">27568339</pub-id>
                    <pub-id pub-id-type="doi">10.1016/j.addbeh.2016.08.019</pub-id>
                </mixed-citation>
            </ref>
            <ref id="ref-7">
                <label>7</label>
                <mixed-citation publication-type="journal">
                    <person-group person-group-type="author">

                        <name name-style="western">
                            <surname>Raine</surname>
                            <given-names>C</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Webb</surname>
                            <given-names>DJ</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Maxwell</surname>
                            <given-names>SR</given-names>
                        </name>
</person-group>:
                    <article-title>The availability of prescription-only analgesics purchased from the internet in the UK.</article-title>
                    <source>

                        <italic toggle="yes">Br J Clin Pharmacol.</italic>
</source>Blackwell Publishing Ltd,<year>2009</year>;<volume>67</volume>(<issue>2</issue>):<fpage>250</fpage>&#x2013;<lpage>4</lpage>.
                    <pub-id pub-id-type="pmid">19154446</pub-id>
                    <pub-id pub-id-type="doi">10.1111/j.1365-2125.2008.03343.x</pub-id>
                    <pub-id pub-id-type="pmcid">2670383</pub-id>
                </mixed-citation>
            </ref>
            <ref id="ref-8">
                <label>8</label>
                <mixed-citation publication-type="journal">
                    <person-group person-group-type="author">

                        <name name-style="western">
                            <surname>Forman</surname>
                            <given-names>RF</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Block</surname>
                            <given-names>LG</given-names>
                        </name>
</person-group>:
                    <article-title>The Marketing of Opioid Medications without Prescription over the Internet.</article-title>
                    <source>

                        <italic toggle="yes">J Public Policy Mark.</italic>
</source>
                    <year>2006</year>;<volume>25</volume>:<fpage>133</fpage>&#x2013;<lpage>46</lpage>.
                    <pub-id pub-id-type="doi">10.1509/jppm.25.2.133</pub-id>
                </mixed-citation>
            </ref>
            <ref id="ref-9">
                <label>9</label>
                <mixed-citation publication-type="journal">
                    <collab>U S Government Accountability Office HGG</collab>:
                    <article-title>INTERNET PHARMACIES: Federal Agencies and States Face Challenges Combating Rogue Sites, Particularly Those Abroad [Internet]</article-title>. gao.gov.<year>2013</year>, [cited 2016 Jan 11].
                    <ext-link ext-link-type="uri" xlink:href="http://www.gao.gov/assets/660/655751.pdf">Reference Source</ext-link>
                </mixed-citation>
            </ref>
            <ref id="ref-10">
                <label>10</label>
                <mixed-citation publication-type="journal">
                    <person-group person-group-type="author">

                        <name name-style="western">
                            <surname>Mackey</surname>
                            <given-names>TK</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Kalyanam</surname>
                            <given-names>J</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Katsuki</surname>
                            <given-names>T</given-names>
                        </name>

                        <etal/>
</person-group>:
                    <article-title>Machine Learning to Detect Prescription Opioid Abuse Promotion and Access via Twitter</article-title>.
                    <italic toggle="yes">Am J Public Health</italic>: published online before print October 19,<year>2017</year>.<fpage>e1</fpage>&#x2013;<lpage>e6</lpage>.
                    <pub-id pub-id-type="pmid">29048960</pub-id>
                    <pub-id pub-id-type="doi">10.2105/AJPH.2017.303994</pub-id>
                </mixed-citation>
            </ref>
            <ref id="ref-11">
                <label>11</label>
                <mixed-citation publication-type="journal">
                    <person-group person-group-type="author">

                        <name name-style="western">
                            <surname>Yan</surname>
                            <given-names>X</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Guo</surname>
                            <given-names>J</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Lan</surname>
                            <given-names>Y</given-names>
                        </name>

                        <etal/>
</person-group>:
                    <article-title>A biterm topic model for short texts.</article-title>
                    <source>

                        <italic toggle="yes">The 22nd international conference.</italic>
</source>New York, New York, USA: ACM,<year>2013</year>;<fpage>12</fpage>.
                    <pub-id pub-id-type="doi">10.1145/2488388.2488514</pub-id>
                </mixed-citation>
            </ref>
            <ref id="ref-12">
                <label>12</label>
                <mixed-citation publication-type="journal">
                    <person-group person-group-type="author">

                        <name name-style="western">
                            <surname>Mackey</surname>
                            <given-names>T</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Kalyanam</surname>
                            <given-names>J</given-names>
                        </name>
</person-group>:
                    <article-title>Detection of Illicit Online Sales of Fentanyls via Twitter.</article-title>
                    <source>

                        <italic toggle="yes">Zenodo.</italic>
</source>
                    <year>2017</year>.
                    <ext-link ext-link-type="uri" xlink:href="http://dx.doi.org/10.5281/zenodo.1038177">Data Source</ext-link>
                </mixed-citation>
            </ref>
        </ref-list>
    </back>
    <sub-article article-type="reviewer-report" id="report28804">
        <front-stub>
            <article-id pub-id-type="doi">10.5256/f1000research.14002.r28804</article-id>
            <title-group>
                <article-title>Reviewer response for version 1</article-title>
            </title-group>
            <contrib-group>
                <contrib contrib-type="author">
                    <name>
                        <surname>Ferrara</surname>
                        <given-names>Emilio</given-names>
                    </name>
                    <xref ref-type="aff" rid="r28804a1">1</xref>
                    <role>Referee</role>
                </contrib>
                <aff id="r28804a1">
                    <label>1</label>Information Sciences Institute, University of Southern California, Marina del Rey, CA, USA</aff>
            </contrib-group>
            <author-notes>
                <fn fn-type="conflict">
                    <p>
                        <bold>Competing interests: </bold>No competing interests were disclosed.</p>
                </fn>
            </author-notes>
            <pub-date pub-type="epub">
                <day>18</day>
                <month>12</month>
                <year>2017</year>
            </pub-date>
            <permissions>
                <copyright-statement>Copyright: &#x00a9; 2017 Ferrara E</copyright-statement>
                <copyright-year>2017</copyright-year>
                <license xlink:href="https://creativecommons.org/licenses/by/4.0/">
                    <license-p>This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.</license-p>
                </license>
            </permissions>
            <related-article ext-link-type="doi" id="relatedArticleReport28804" related-article-type="peer-reviewed-article" xlink:href="10.12688/f1000research.12914.1"/>
            <custom-meta-group>
                <custom-meta>
                    <meta-name>recommendation</meta-name>
                    <meta-value>approve</meta-value>
                </custom-meta>
            </custom-meta-group>
        </front-stub>
        <body>
            <p>This is a well written report on a timely and important research topic, directly connected to the current US opioid epidemic. Overall, the methodology is sound and the findings are interesting. One limitation of the data collection that could be addressed in future studies is the focus on one single keyword (fentanyl): a quick search using brand names (e.g., oxycodone, oxycontin) and slang&#x00a0;terms / street names&#x00a0;usually associated with this drug (e.g., ox, ocs) produce many&#x00a0;additional hits, including at times associated with illicit person-to-person sales. The authors carried out a scrupulous investigation of the clearnet domains associated with the links discovered on Twitter &#x2013; I wonder if a similar analysis could be carried out on the Dark Web (often associated with the illicit sale of drugs): this would require significant changes in the data collection infrastructure thus I can only expect it could be done in a follow up study.</p>
            <p>Is the work clearly and accurately presented and does it cite the current literature?</p>
            <p>Yes</p>
            <p>If applicable, is the statistical analysis and its interpretation appropriate?</p>
            <p>Yes</p>
            <p>Are all the source data underlying the results available to ensure full reproducibility?</p>
            <p>Partly</p>
            <p>Is the study design appropriate and is the work technically sound?</p>
            <p>Yes</p>
            <p>Are the conclusions drawn adequately supported by the results?</p>
            <p>Yes</p>
            <p>Are sufficient details of methods and analysis provided to allow replication by others?</p>
            <p>Yes</p>
            <p>Reviewer Expertise:</p>
            <p>social networks, data science, machine learning, AI</p>
            <p>I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.</p>
        </body>
    </sub-article>
    <sub-article article-type="reviewer-report" id="report28806">
        <front-stub>
            <article-id pub-id-type="doi">10.5256/f1000research.14002.r28806</article-id>
            <title-group>
                <article-title>Reviewer response for version 1</article-title>
            </title-group>
            <contrib-group>
                <contrib contrib-type="author">
                    <name>
                        <surname>Vida</surname>
                        <given-names>R&#x00f3;bert G.</given-names>
                    </name>
                    <xref ref-type="aff" rid="r28806a1">1</xref>
                    <role>Referee</role>
                </contrib>
                <aff id="r28806a1">
                    <label>1</label>Department of Pharmaceutics and Central Clinical Pharmacy, Faculty of Pharmacy, University of P&#x00e9;cs, P&#x00e9;cs, Hungary</aff>
            </contrib-group>
            <author-notes>
                <fn fn-type="conflict">
                    <p>
                        <bold>Competing interests: </bold>No competing interests were disclosed.</p>
                </fn>
            </author-notes>
            <pub-date pub-type="epub">
                <day>11</day>
                <month>12</month>
                <year>2017</year>
            </pub-date>
            <permissions>
                <copyright-statement>Copyright: &#x00a9; 2017 Vida RG</copyright-statement>
                <copyright-year>2017</copyright-year>
                <license xlink:href="https://creativecommons.org/licenses/by/4.0/">
                    <license-p>This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.</license-p>
                </license>
            </permissions>
            <related-article ext-link-type="doi" id="relatedArticleReport28806" related-article-type="peer-reviewed-article" xlink:href="10.12688/f1000research.12914.1"/>
            <custom-meta-group>
                <custom-meta>
                    <meta-name>recommendation</meta-name>
                    <meta-value>approve</meta-value>
                </custom-meta>
            </custom-meta-group>
        </front-stub>
        <body>
            <p>An up-to-date and innovative methodological approach of a possible global problem of illegal trade of pharmaceuticals via the Internet is absolutely useful. The main messages are in accordance with the scientific literature.</p>
            <p> </p>
            <p> The concern highlighted by the Dr. Marvin D. Shepherd regarding the dissonance between the introduction and the other parts of the publication is real and similarly I suggest the reconsideration and rewrite of that part. Also it would be advisable to include the comparison of the legal and illicit use and highlight the sources of fentanyl products or fentanyl containing products (outpatient and inpatient settings). Beside the counterfeit fentanyl crisis another possible public health danger is hitting the health care system of the United States is the shortage of multiple drugs including opioid drugs as well (e.g.: fentanyl) necessitate the use of alternative supply channels (e.g.: importation of drugs outside the U.S., internet pharmacies) make the health sector more susceptible to illegal and counterfeit products.</p>
            <p> </p>
            <p> Beside the network science the Big Data type of analysis is a novel and hopefully in the future a routinely used methodology to identify and monitor illegal vendors and their connections in the field of illegal internet sale of pharmaceuticals. The more detailed description of the method or even the representation with a figure would help the understanding of it.</p>
            <p> </p>
            <p> The characterisation (e.g.: ordering is possible, LegitScript or VIPPS NABP legitimacy, products being sold, prescription requirements) of the identified online pharmacies would have highlighted the illegal internet sale of fentanyl products more accurately.</p>
            <p> </p>
            <p> Moreover the inclusion of comparison of the results with the literature data regarding other social media platforms or the simple google search methodology would give a more complete picture of this public health danger.</p>
            <p> </p>
            <p>Is the work clearly and accurately presented and does it cite the current literature?</p>
            <p>Yes</p>
            <p>If applicable, is the statistical analysis and its interpretation appropriate?</p>
            <p>Yes</p>
            <p>Are all the source data underlying the results available to ensure full reproducibility?</p>
            <p>Yes</p>
            <p>Is the study design appropriate and is the work technically sound?</p>
            <p>Yes</p>
            <p>Are the conclusions drawn adequately supported by the results?</p>
            <p>Yes</p>
            <p>Are sufficient details of methods and analysis provided to allow replication by others?</p>
            <p>No</p>
            <p>Reviewer Expertise:</p>
            <p>NA</p>
            <p>I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.</p>
        </body>
    </sub-article>
    <sub-article article-type="reviewer-report" id="report27555">
        <front-stub>
            <article-id pub-id-type="doi">10.5256/f1000research.14002.r27555</article-id>
            <title-group>
                <article-title>Reviewer response for version 1</article-title>
            </title-group>
            <contrib-group>
                <contrib contrib-type="author">
                    <name>
                        <surname>Shepherd</surname>
                        <given-names>Marvin D.</given-names>
                    </name>
                    <xref ref-type="aff" rid="r27555a1">1</xref>
                    <role>Referee</role>
                </contrib>
                <aff id="r27555a1">
                    <label>1</label>Division of Health Outcomes and Pharmacy Practice, College of Pharmacy, The University of Texas at Austin, Austin, TX, USA</aff>
            </contrib-group>
            <author-notes>
                <fn fn-type="conflict">
                    <p>
                        <bold>Competing interests: </bold>No competing interests were disclosed.</p>
                </fn>
            </author-notes>
            <pub-date pub-type="epub">
                <day>30</day>
                <month>11</month>
                <year>2017</year>
            </pub-date>
            <permissions>
                <copyright-statement>Copyright: &#x00a9; 2017 Shepherd MD</copyright-statement>
                <copyright-year>2017</copyright-year>
                <license xlink:href="https://creativecommons.org/licenses/by/4.0/">
                    <license-p>This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.</license-p>
                </license>
            </permissions>
            <related-article ext-link-type="doi" id="relatedArticleReport27555" related-article-type="peer-reviewed-article" xlink:href="10.12688/f1000research.12914.1"/>
            <custom-meta-group>
                <custom-meta>
                    <meta-name>recommendation</meta-name>
                    <meta-value>approve</meta-value>
                </custom-meta>
            </custom-meta-group>
        </front-stub>
        <body>
            <p>My main concern is the discontinuity between the introduction and the research conducted. The introduction focused on problems of pharmaceutical purchased which were&#x00a0;laced with counterfeit fentanyl. This is true and is a major health hazard as pointed out by the authors. Fentanyls are a deadly pharmaceuticals and again this was emphasized by the authors. However, the paper's research focus is on tweets from&#x00a0;people who are interested in purchasing fentanyl&#x00a0;illegally. It wasn't on the counterfeit fentanyl laced products.&#x00a0;I suggest that the intro be rewritten and I mention the legitimate uses of&#x00a0;fentanyl. Please mention that one of fentanyl's major uses&#x00a0;is as&#x00a0;a pain reliever in&#x00a0;hospital surgical suites and must be carefully controlled by medical practitioners. To reiterate, the introduction&#x00a0;doesn't match well the research content.</p>
            <p> </p>
            <p> The methodology approach was was appropriate and innovative. I know they provided&#x00a0;a link for a full&#x00a0;description of the search terms, but in reading it I suggest that they provided at least a partial list of the names used&#x00a0;i.e. different spellings of fentanyl, different names for the same product (foreign names). Searching tweets was original, but&#x00a0;it didn't produce huge number of tweets.&#x00a0;
                <bold>This is worth reporting in that few people tweet&#x00a0;to purchase fentanyl</bold>. The first thing I thought of was why tweet when you can do a google search and get right to the web sites?&#x00a0;
                <bold>This needs to be put in the discussion as to why the results were so low.&#x00a0;</bold>&#x00a0;I am sure there are other methods to find the product.</p>
            <p>Is the work clearly and accurately presented and does it cite the current literature?</p>
            <p>Yes</p>
            <p>If applicable, is the statistical analysis and its interpretation appropriate?</p>
            <p>Yes</p>
            <p>Are all the source data underlying the results available to ensure full reproducibility?</p>
            <p>Yes</p>
            <p>Is the study design appropriate and is the work technically sound?</p>
            <p>Yes</p>
            <p>Are the conclusions drawn adequately supported by the results?</p>
            <p>Yes</p>
            <p>Are sufficient details of methods and analysis provided to allow replication by others?</p>
            <p>No</p>
            <p>Reviewer Expertise:</p>
            <p>NA</p>
            <p>I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.</p>
        </body>
    </sub-article>
</article>
