Ethical research standards in a world of big data [version 1; peer review: 2 approved with reservations]

In 2009 Ginsberg et al. reported using Google search query volume to estimate influenza activity in advance of traditional methodologies. It was a groundbreaking example of digital disease detection, and it still remains illustrative of the power of gathering data from the internet for important research. In recent years, the methodologies have been extended to include new topics and data sources; Twitter in particular has been used for surveillance of influenza-like-illnesses, political sentiments, and even behavioral risk factors like sentiments about childhood vaccination programs. As the research landscape continuously changes, the protection of human subjects in online research needs to keep pace. Here we propose a number of guidelines for ensuring that the work done by digital researchers is supported by ethical-use principles. Our proposed guidelines include: 1) Study designs using Twitter-derived data should be transparent and readily available to the public. 2) The context in which a tweet is sent should be respected by researchers. 3) All data that could be used to identify tweet authors, including geolocations, should be secured. 4) No information collected from Twitter should be used to procure more data about tweet authors from other sources. 5) Study designs that require data collection from a few individuals rather than aggregate analysis require Institutional Review Board (IRB) approval. 6) Researchers should adhere to a user’s attempt to control his or her data by respecting privacy settings. As researchers, we believe that a discourse within the research community is needed to ensure protection of research subjects. These guidelines are offered to help start this discourse and to lay the foundations for the ethical use of Twitter data.


Introduction
The growing popularity of social media sites presents a unique opportunity to study human interactions and experiences. Twitter, one of the most popular social media sites, allows users to 'microblog' by sharing 140 character messages with their social network. Although Twitter doesn't disclose the number of people who use its service, estimates are in the hundreds of millions and perhaps as many as half a billion 1 . Approximately 340 million tweets are sent every day around the world 2 . Researchers have begun to use this data to answer questions in a variety of fields [3][4][5][6][7][8][9] . Recent reflections on the data collection practices of the US National Security Administration have spurred similar meditations on the ethics of digital research 10 . The concern is that Twitter data could conceivably be used in a way that violates the privacy and rights of the tweet authors.
Twitter data have already been used in a number of studies to detect influenza-like illness (ILI) [3][4][5][6] , risky behaviors associated with the transmission of HIV 7 , sentiments about childhood vaccination programs 8 , and political sentiments 9 . These study designs generally feature count data, rather than user-specific data. For example, there are multiple studies that compare the proportion of tweets about flurelated symptoms to public health data on influenza-like incidence (ILI). An increase in syndromic flu tweets might indicate that an outbreak is occurring. These data are usually reported either at the national level or without any geographic parameters. Another common study design aims to determine public sentiments by counting words, phrases, and emoticons that co-occur with keywords like 'Obama'. These sentiment indicators can be used to infer public opinions about political elections, mental health, or consumer products.
Unlike Facebook, Pinterest and other competitors, Twitter provides several application programming interfaces (APIs) that allow real-time access to vast amounts of content. Data streamed through the APIs include metadata about the authors, including the text location from their profile (e.g. 'Baltimore'), their time zone, the time they sent the tweet, the number of friends and followers they have, the number of tweets they have ever sent, and more ( Figure 1). Approximately 1% of tweets have a geolocation, which uses GPS to append the author's precise geographic coordinates to the tweet. Geolocations are sufficiently detailed to determine from which wing of a building a tweet was sent. The default privacy settings do not enable geolocation, but do make a user's tweets and metadata available through the API. Users can modify their setting to make their profile private, which shields their account from public view online and from the API 11 .
There are numerous ways to access the data through these APIs: one method is through the 'garden hose', which is a random sample of approximately 1% of all live-streamed tweets. Other access methods include the search API which enables searching for particular users, hashtags, or locations, and author-specific queries which can retrospectively gather up to 3,200 tweets from a single user 11 . Furthermore, in 2010 Twitter donated its entire historical record of tweets to the US Library of Congress. Detailed plans for these data are not yet available, but the Library of Congress has indicated that it intends to collaborate with academic institutions to make the data available to researchers 12 .
The strength of tweets as a data source is in the volume; collection through the garden hose API brings in approximately 60,000-100,000 tweets per day. However because tweets are short and often lack context, it is difficult for computers to determine tweet content automatically. For this reason, researchers primarily use tweet data to conduct population-level research concerned with trends and even without informed consent 13 . According to these guidelines, tweets are text that users publish for the purpose of sharing with others. The weakness of this argument is that it fails to distinguish between population-level research and research focused on selected individuals. It would be clearly unethical for a researcher to follow one specific shopper around the mall and gather data exclusively about him without his consent. However, simply counting or observing behavior in aggregate at a mall is an acceptable research practice. The difference is that the latter example adheres to a level of privacy that the observed individual might expect from being in public, whereas the former violates those natural privacy boundaries. A similar distinction is needed in digital research.
As an example of the potential privacy pitfalls of digital research, suppose investigators were interested in the social networks of adolescents suffering from depression. A research plan might look like this: the investigators gather geocoded tweets that contain words relevant to the topic of interest, as shown in Figure 2. They filter for geocodes that correspond to school locations in order to patterns. Study designs rely on large volumes of data to accommodate false positives and negatives. A typical data set contains millions of tweets and many thousands of tweet authors. However, a user-centric use case involving Twitter is not inconceivable. Researchers interested in social network analysis, qualitative research, and rareevent topics may eventually turn to Twitter as a data source. Potential methodologies include building a social network out of @mentions (the @ is Twitter lexicon for referencing another user); mining qualitative data from specific user's accounts; or conducting prospective research by following a person or small group of people over time. These user-centric approaches are fundamentally different from population-level studies, and may require different ethical considerations than aggregated study designs. Additional methodologies might also involve interacting with Twitter users, which will not be addressed here.
Under non-digital circumstances, ethics guidelines suggest that collecting information from a public space where people could 'reasonably expect to be observed by strangers' is considered appropriate identify adolescent users. From there, a simple query to the Twitter API returns a list of followers for each of those presumably depressed adolescents. They now have a social network. For each member of the network, they mine the user's tweet histories to find identifying details such as their real names. The researchers then use the gathered information to 'snowball' data collection by curating from a variety of different sources like Facebook, tumblr and the White Pages. They can collect birth dates, cell phone numbers, home addresses, favorite hangout spots, "likes" and "dislikes", etc. The final result would be detailed demographic information for potentially thousands of people who exhibit symptoms of depression or are connected to a depressed adolescent. Current guidelines do not prohibit this kind of research activity. However, if the same information were collected through surveys or other traditional means, Institutional Review Board (IRB) approval would be needed.
According to the US Department of Health and Human Services Policy for Protection of Human Research Subjects, data that are publicly available are exempt from requiring IRB approval 14 . Because Twitter data are public, they technically fall under this exemption. Furthermore, Twitter's privacy policy makes no secret of the fact that user data are indexed by search engines, archived within the US Library of Congress, and are available through an API 15 . However, it is unlikely that many users follow the link to read the lengthy and complex document. One study found that it would take 244 hours a year for an average internet user to read every privacy policy of the unique sites they visit 16 .
The US Consumer Privacy Bill of Rights (CPBR) 17 may instead serve as a useful framework for guiding researchers conducting research with Twitter. The CPBR was issued by the Obama administration in February 2012 in order to "give consumers clear guidance on what they should expect from those who handle their personal information, and set expectations for companies that use personal data." There are seven principles enumerated by CPBR in Table 1.

Proposed guidelines for the ethical use of Twitter data
The objectives, methodologies, and data handling practices of the project are transparent and easily accessible This information should be published in manuscripts, published on the web for the public to access, and provided to IRB (when relevant). Going forward, collaboration between the research community and Twitter to provide information to users about ongoing research and relevant results may also be beneficial. Transparency regarding uses of Internet data for research purposes is needed for fostering 'privacy literacy' so that the users can make informed decisions about participating in Twitter.
Study design and analyses respect the context in which a tweet was sent A tweet author discussing his mental health, for example, does not do so with the intention of sharing that data with researchers; he does it to communicate with his digital community. Qualitatively analyzing these communications as if they are offered for research consumption does not align with the context in which the tweets were created. Twitter participants can reasonably expect to rely on some anonymity of the crowd to manage privacy.
The anonymity of tweet authors is protected, ensuring that subjects should not be identifiable in any way To preserve source anonymity, direct quotes or screen names are not publishable, nor are any details that could be used to identify a subject. Any and all information that could be entered into a search engine to trace back to a human source should be protected. A composite of multiple example tweets may instead be used for illustrative purpose. Geolocations in particular should be scaled to a larger geographic area in order to avoid violating the privacy of those tweet authors. The Title 13 of the Data Protection and Privacy Policy, the federal law under which the Census Bureau is regulated, expressly forbids publishing GPS coordinates 18 ; researchers should adhere to this guideline as well.
Tweet data are not used to harvest additional information from other sources Focused collection is also important for preserving anonymity. It is possible to use data collected from Twitter to discern the identities of tweet authors, which can then be used to find and collect additional information from additional sources. For example an author's username, identifying details provided in tweet texts, or geolocations could all be used to collect data about that individual from other sources like Facebook, LinkedIn, Flickr, or public records.
Twitter users' efforts to control their personal data are honored Researchers may not follow a user on Twitter in order to gain access to a protected account. Doing so would violate that user's efforts to control his or her personal data.

Transparency
Consumers have a right to easily understandable and accessible information about privacy and security practices.
Respect for context Consumers have a right to expect that companies will collect, use, and disclose personal data in ways that are consistent with the context in which consumers provide the data.

Security
Consumers have a right to secure and responsible handling of personal data.

Focused collection
Consumers have a right to reasonable limits on the personal data that companies collect and retain.

Accountability
Consumers have a right to have personal data handled by companies with appropriate measures in place to assure they adhere to the Consumer Privacy Bill of Rights.

Individual control
Consumers have a right to exercise control over what personal data companies collect from them.
Access and accuracy Consumers have a right to access and correct personal data in usable formats.
Author contributions CR conceived of the guidelines and drafted the manuscript. BL contributed to both the development of the guidelines and preparation of the manuscript.

Competing interests
No competing interests were disclosed. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Grant information
Researchers work collaboratively with IRB just as they would for any other human subject data collection There is not currently an expectation that researchers engaging in research using Twitter will interface with their IRB. As discussed above, studies that could be conceived as individual-based should require IRB approval, whereas research designs that use data in aggregate (e.g. counts of keywords) may proceed without explicit consent. In turn, review boards should keep abreast of social network mining methodologies and corresponding ethical considerations in order provide informed guidance to researchers.

Conclusions
Research involving Twitter is growing in popularity, but the issues surrounding the ethics of using it as a data source have not yet been closely examined. There are hypothetical study designs that could use Twitter data in a way that violates the privacy and ethical treatment of participants. In order to avoid those misuses, six guidelines derived from the US Consumer Privacy Bill of Rights are proposed. We welcome discourse in the research community on this topic, and encourage further discussion.
Please use the #EthicalTwitter hashtag on Twitter to participate in this discussion online.

Sherry Emery
Institute for Health Research and Policy, University of Illinois at Chicago, Chicago, IL, USA In general, I think this is a very helpful piece, which lays out some practical and legitimate privacy risks that arise when working with social media data. And the authors' suggestions for the code of ethics are thoughtful, yet warrant some debate.
My main concern is that corporate interests, most prominently Facebook -but also app developers and marketers by the legion -are using these data to both manipulate and profit from users digital networks and behaviors. More discussion of the appropriateness of applying traditional IRB standards to social media research is needed. In particular, the relative uselessness of the 'terms of use' in the context of IRB standards is notable. But in practice, this is the standard that commercial entities are applying. For researchers to be held to a different standard risks abdicating social media research to corporate interests--and the public good of generating knowledge from these data will be lost unless that knowledge directly serves somebody's bottom line.
I'd like to see a more thorough consideration of the practical implications of potential privacy breaches from social media research. Would these meet the minimal risk criterion used by most IRBs? If so, then some of the suggestions in the proposed guidelines may be unnecessarily 'protective.' Competing Interests: No competing interests were disclosed.
I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Reviewer Report 06 March 2014
Moreover the aim and applicability of the guidelines is unclear. How would these guidelines be enforced? Would they be applied to all studies? Is there any mechanism for exceptions to this? For instance if I wanted to do a study comparing health behaviour on Twitter and Facebook, I would need to collect data from both networks. But this violates the guidelines. Or are these guidelines only for studies where IRB approval would not be obtained? There is much discussion over whether social media studies require any IRB approval at all 1,2,3 . But your last guideline seems to disagree with this. The title of the paper refers to "big data" but the paper itself really concerns itself with Twitter (with some discussion of linkability). Are the guidelines to apply to all big data studies?
Given that this paper really only targets US ethics committees and IRBs, I was surprised that there was no discussion of the Common Rule or the recent Menlo Report 4,5 . An opinion article in Science last year also discussed US regulation 6 .
One of the guidelines explicitly describes the context in which a tweet was sent. But there is no mention of contextual integrity 7 which seems to be the most common framework for studying this, and has indeed been applied to social media research 8,9,10 .
Danah Boyd has talked widely about the ethics of big data research 11,12 and her work should be discussed. Concerns about Twitter specifically have also been raised 13,14 and Neuhaus and Webmoor specifically propose "agile ethics" for the study of such networks 15 . How would agile ethics fit in with your guidelines?
The implicit assumption in your guidelines is that studies should occur without consent and involve trawling the firehose. But these kinds of studies in themselves are controversial 16 . Moreover is there any reason why informed consent could not be obtained for some of these studies, especially since you touch on consent when discussing your hypothetical shopping centre study? We have looked at this empirically 17 .
One striking difference between research and the collection of data for business purposes is that researchers are typically interested in reproducibility. Indeed many research funding bodies worldwide are now insisting on data archiving and sharing as a requirement, and this would apply to social media studies as well (something that has been criticised in the past 18 ). Should your guidelines not address data sharing? Technical challenges exist, including the linkability problems that you discuss 19 and the difficulty of anonymising data 20 , while solutions are still nascent 21,22 .
The BPS reference (13) should be updated 23 . Moreover you ought to compare and contrast other ethical guidelines for Internet research 24,25,26 , and other proposals for social media and health research 27,28 .
At one point you mention "privacy literacy" but there is no reference and it was unclear what you mean here. Privacy "salience" has been discussed in the literature 29 but might be slightly different.
Finally, I qualify my review by stating that I am not a life scientist and that this is the first review that I have written for F1000Research. Indeed, I had not heard of it prior to my review request. I do question the appropriateness of a scientific journal for this work. If the intention of the authors is, as stated, to encourage more discussion then it would be good to focus this in one place. Since F1000Research allows comments from readers, then perhaps this does provide a better avenue for discussion than over Twitter, which can be difficult over long time-scales and is not particularly persistent. But in my opinion an even better mechanism for discussion and debate would be to hold a workshop and then publish a workshop report. Certainly further discussion with other communities, and examination of the discussion that has already taken place, would be useful to inform and improve this document. In the first instance, I think it would be good to see a revised document that shows more awareness of the current work, and where controversy exists, discusses why the authors have taken their chosen stance when proposing their guidelines.