A Taxonomy to Support the Statistical Study of Funding-induced Biases in Science

The biomedical community is a leader in research on bias in science, including funding-induced bias. To facilitate this research, we have developed a taxonomy of fifteen different types of potential funding-induced bias. We describe each type of bias, as well as giving a snapshot of existing research and briefly discussing the potential for various forms of statistical analysis. We also introduce the concept of an amplifying bias cascade, wherein bias builds through successive iterations. David Wojick ( ) Corresponding author: dwojick@craigellachie.us Wojick D and Michaels P. How to cite this article: A Taxonomy to Support the Statistical Study of Funding-induced Biases in Science 2015, :886 (doi: ) [version 1; referees: 2 not approved] F1000Research 4 10.12688/f1000research.7094.1 © 2015 Wojick D and Michaels P. This is an open access article distributed under the terms of the Copyright: Creative Commons Attribution , which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Licence This study was funded by the Cato Center for the Study of Science. The authors declared that no grants were involved in Grant information: supporting this work. Competing interests: No competing interests were disclosed. 22 Sep 2015, :886 (doi: ) First published: 4 10.12688/f1000research.7094.1 1


Introduction
The purpose of this report is to provide a framework for doing statistical research on the problem of funding-induced bias in science. In recent years the issue of bias in science has come under increasing scrutiny within the scientific community. The research question is does biased funding skew research in a preferred direction, one that supports an existing mission, practice, product, policy or paradigm?
Our working definition of "funding-induced bias" is any scientific activity where the prospect of funding influences the result in a way that benefits the funder.
While the basic concept of funding-induced bias is simple, the potential forms that this bias might take are far from simple. Science is a complex social system and funding is a major driver. In order to facilitate research into funding-induced bias it is necessary to isolate specific kinds of bias. Thus the framework presented here is basically a taxonomy of types of funding-induced bias.
For the purposes of future research the concept of funding-induced bias is analyzed in the context of the various practices in science where bias can occur. Fifteen different scientific practices are analyzed, ranging from the budgeting and funding for research to the publishing and communication of results. We make no distinctions regarding the source of the funding. The funding organization may be a commercial firm, a non-profit entity or a government agency, or even an individual.
What we have found is that some types of bias are being studied extensively, including quantitatively. Various aspects of peer review and publication bias, especially in biomedicine, appear to be the most heavily researched types of bias. The role of funding in inducing bias is frequently alluded to as a potential financial conflict of interest. But the role of funding is not the focus of most research into bias, which tends to look more at the practice of bias than at its cause. Thus a new research thrust is likely needed.
The concept of funding-induced bias is one that specifically occurs in the discussion of and research into some of the fifteen bias research types that we have identified, but not in all of them. It tends to occur where specific funding is the issue. We are here using the concept more broadly, to include cases where the funding may be somewhat distant from the activity in question, such as in the communication of research results.

Cascading amplification of funding-induced bias
In this report we are mostly concerned with individual types of funding-induced bias. But there is an intrinsic sequence to the various biases we have identified and this raises the possibility of cascading amplification. By amplification we mean one biased activity is followed by another, such that the first bias is increased.
A simple, and perhaps common, example of amplification might be when the hype in a press release is exaggerated in a news story. Let's say the press release overstates the importance of the research result, but with some qualification. The news story then reports the result as a great breakthrough, far more strongly than the press release, ignoring the latter's qualifications. In this way the original bias has been amplified.
There is also the possibility of cascading amplification. This is the case where one biased activity is followed by multiple instances of amplification. Using our example, suppose a single biased press release generates many different news stories, which vie with one another for exaggeration. This one-to-many amplification is properly termed a cascade.
Moreover, there is the possibility of cascading amplification on a very large scale and over multiple biased stages. Here is an example of how it might work. 1. A funding organization receives biased funding for research.

They issue multiple biased Requests for Proposals (RFPs).
3. Multiple biased projects are selected for each RFP. 4. Many projects produce multiple biased articles, press releases, etc.
5. Many of these articles and releases generate multiple biased news stories. 6. The resulting amplified bias is communicated to the public on a large scale.
One can see how in this instance a single funding activity, the funder's budget, might eventually lead to hundreds of hyperbolic news stories. This would be a very large scale cascading amplification of funding-induced bias.

Protecting paradigms as a source of bias
Thomas Kuhn pointed out on his groundbreaking work "The Structure of Scientific Revolutions" that fundamental beliefs can take over a scientific field 1 . He called these entrenched beliefs "paradigms" and noted that they tend to direct scientific thinking in specific directions. Once these beliefs become entrenched they are difficult to dislodge, despite growing evidence that they may be incorrect. Moreover science, like any human endeavor, is subject to fads and fashions.
Clearly processes like paradigms and fashions can influence the funding of research. Kuhn notes that the paradigms tend to specify not only what the important questions are, but also what the answers are expected to look like, and these questions and answers are the focus of research funding. At some point this influence can become bias, especially when the paradigm becomes questionable. This may be especially true when the outdated paradigm is built into the mission, products or policies of a research funding organization.
Biased funding in turn biases the funded research in complex and subtle ways. The purpose of this project is to systematically analyze many of these forms of funding-induced bias in science, in order to further future research. It should be noted however that this sort of bias need not be dishonest, and often is not, even though funding is involved. As Kuhn points out, defending a paradigm is the norm in scientific activity. Thus many of the biases are basically ideological in nature. Funding is simply part of the ideology, often a central part.

Indicators of possible bias: controversy and allegations
1) Bias does not go unnoticed, so controversy over specific aspects of the science related to a funding organization's mission may be evidence of funding-induced bias. Controversy may include one or more of the following general aspects of the science. 2) Allegations of specific practices of bias. The strongest evidence of bias may be specific allegations, along the lines of the fifteen practices of bias described at length below.

Approaches to the research: test the practices in question for bias, preferably using quantitative statistical methods
The combination of specific sources and levels of bias, with aspects of controversy and allegations of biased practices yields a large number of specific possible cases that can be investigated individually. Thus the first step in research will often be to determine the specific case or cases in question. The goal is to be precise as to the possible source, scope, science and type of bias involved.
For example, the specific case may indicate which documents need to be analyzed for evidence of funding-induced bias. In particular, the mission aspect is a good starting point for bias research, because bias that is not related to organization's mission is unlikely to be funding-induced.
However, some of our fifteen bias practices are not directly funded by funding organizations. Rather the funding inducement is indirect, rendered by the community as it were. For example, publication in leading subscription journals is often not funded but it is an important indirect component of competing for future funding.

Assessing the potential for quantification of each type of bias
The analysis below of each of the fifteen practices of bias includes a brief assessment of the potential for quantification. This assessment includes suggesting feasible approaches to successful quantification for each practice. This quantification assessment typically includes the following issues.

A. Data issues
Is there enough data to support quantification? For example, peer review and selection of proposals and journal articles tend to be black boxes, with little available data. Citations and co-authorship might be viable proxies for peer review. Persistence might be a proxy for selection.
How best to sample the available data? Computer based sampling methods, if feasible, are preferable to manual methods, because the latter are relatively expensive. Randomized sampling methods are preferable to convenience samples but are not always possible. Likewise, is the available data complete or partial? Partial data is often self sampled, which limits the scope of the resulting conclusions. Is a lot of processing of the data involved, before it can be analyzed? If so then how should it be done?

B. Quantification analysis method issues
What is the suggested best method of quantification? In particular is it subjective or objective, that is, is human judgment and classification of the data involved, or just simple counting of clearly defined instances. In the latter case the analysis might be done by computer, which is typically cheaper than manual analysis, depending on the amount of programming involved.
Are indirect methods required? Where direct data is not available, using proxy data may be feasible, but this involves linking the proxy to the bias in question. Semantic approaches may also be feasible. For example, in the case of the hyping of research results in press releases, an approach to quantification might be by counting the occurrence of potentially hyperbolic words in a sample of press releases but not in the related abstracts or articles.

Potential practices of funding-induced bias
In this section we briefly describe fifteen specific practices of bias in the context of funded scientific research. For convenience, as well as introductory discussion, these fifteen are first listed below as a kind of table of contents for this section: 1. Funding agency programs that have a biased focus.
2. Agency Strategic Plans, RFPs, etc., with an agenda, not asking the right questions.
3. Biased peer review of research proposals. 4. Biased selection of research proposals by the agency program. 5. Preference for modeling using biased assumptions. 6. Biased peer review of journal articles and conference presentations. 7. Biased meta-analysis of the scientific literature.
8. Failure to report negative results. 9. Manipulation of data to bias results.
10. Refusing to share data with potential critics. 11. Asserting conjectures as facts.
12. False confidence in tentative findings.
13. Exaggeration of the importance of findings by researchers and agencies.
14. Amplification of exaggeration by the press. 15. More funding with an agenda, building on the above, so the cycle repeats and builds.
While each of the biased practices listed above may occur in isolation, there is also a potential sequence to them, a cascade as it were. Science is a complex social system and funding is a major driver. Some of the practices listed above do not involve direct funding, but each can clearly be influenced by the existence or mere prospect of such funding.
Each of these fifteen specific biased practices is discussed briefly below. Taken together they provide a kind of "field guide" to funding-induced bias.
1. Funding agency programs that have a biased focus.
In some cases sponsors fund entire research programs that may be biased in their very structure. For example by ignoring certain scientific questions that are claimed to be important or by supporting specific hypotheses, especially those favorable to the organization's goals.
Organizational funding requests and final research budgets are sometimes public documents, which are available for analysis. In many cases both the request and the funding occurs at the program level. Therefore, one basic approach to looking for bias is to examine how the funds are allocated to various research programs and questions. For example, suppose there are two competing hypotheses, one of which favors an organization's mission, product or policy, while the other does not. Heavy funding of the former, compared to the latter, might be evidence of bias.
Given that this bias might be measured in dollars there is an excellent prospect for quantification. However, the budget documents might not break the dollars out into the funding categories needed to do the bias analysis, which presents a data problem. The use of proxies or estimation techniques may be necessary in such cases. Relatively subjective interpretation may also be required.
2. Strategic Plans, RFPs, etc., with an agenda, not asking the right questions.
Research proposals may be shaped by Strategic Plans and Requests for Proposals (RFP's). These documents often specify those scientific questions that the funding organization deems important, hence worthy of funding. Thus the resulting research proposals may be biased, speaking to what the funder claims is important rather than what the researcher thinks.
There is a small, but interesting, research topic called the "funding effect". Full text Google Scholar search for 2010-2014 gives just 211 hits, of which just 5 find this term in the title. Expanding the period to 2002-2014 gives 329 full text hits and 8 in the title. It appears that the term "funding effect" was coined by Sheldon Krimsky around 2005 and most of the title occurrences are in papers by him. Thus there may be increasing attention to this concept but the literature is still very small. Moreover, most of the focus is on the biasing effect of commercial funding, such as by drug companies. For a Krimsky review article see "Do Financial Conflicts of Interest Bias Research? An Inquiry into the "Funding Effect" Hypothesis" 2 .
A much more common concept related to funding-induced bias is financial conflict of interest (FCOI). Google Scholar search for "FCOI" in titles for the period 2010-2014 gives zero hits. However it does find 186 occurrences in searching the full text, which suggests some research interest. Searching for the full phrase "financial conflict of interest" gives just 9 hits in titles, but over 5,000 in full text. These appear to be mostly research on either biomedical or professional activities.
Searching on the broader concept phrase "conflict of interest" gives over 600 occurrences in titles. However, most the top hits appear to be either guidance or disclosures, not research on conflict of interest. Full text search gives over 240,000 hits. This very large number appears to be the effect of widespread conflict of interest policies, such that many articles include conflict disclosure clauses.
There are numerous ways in which the research funders can say what they are looking for. These are probably some of the best sources of evidence of bias in research funding.
Examples include strategic plans, requests for proposals and clarifications or amendments thereto, scientific conference presentations by funding organization officials, pre-proposal conferences, as well as funding organization reports on the science, especially in relation to their mission.
Analysis of these sources is likely to be interpretative and opportunities for quantitative analysis may be limited. However, an example of a quantitative analysis might be the amount of text devoted to a hypothesis that supports an organization's mission, product or policy, compared to the amount given to a competing hypothesis. Another might be patterns of occurrence of apparently biased statements in multiple sources. Where the documents include funding levels there is also the possibility of finding monetary measures of bias.

Biased peer review of research proposals.
This bias may involve rejecting ideas that appear to conflict with the established paradigm, funding agency mission, or other funding interest. See also Bias #6: Biased peer review of journal articles and conference presentations.
Peer review bias is the subject of considerable public discussion in the scientific community, as well as extensive scientific research. However, peer review is also used in the selection of papers to publish in scholarly journals and much of the discussion does not distinguish between peer review of proposals and articles. Thus there is some overlap between the literature snapshot provided here and that given under Bias #6 (Biased peer review of journal articles and conference presentations).
A Google Scholar search on articles published 2010-2014 with "peer review" in the title gives about 3000 hits, which suggests a great deal of research. To be sure, some of these hits are false in the sense of not being analyses of peer review, but bias is mentioned frequently in the snippets so a lot of the research is focused on that topic. It appears that most of this research is focused on publications, not proposals. Full text search gives over 200,000 hits. This large number suggests that the term "peer review" probably occurs frequently in passing. A major review of peer review bias that covers both proposals and publications was published by Lutz Bornmann, entitled simply "Scientific peer review" 3 . Google Scholar lists 120 citations for this article so it is widely recognized.
Unfortunately, the peer review process is typically not publicly available. This is especially true for those proposals that are rejected. Nether the proposals or the reviews, or even the names of the reviewers, are typically available for bias analysis.
Thus the prospects for bias research might be limited in this case, because of the secrecy, or they might involve indirect methods. For example, one might survey the researchers in the pool of candidates from which the reviews for a given program or project are likely to have been drawn, looking for evidence of bias.
In any case the prospects for simple quantification would seem to be limited, with a lot of interpretation required. Simply getting good data is the first research challenge.

Biased selection of research proposals by the funding organization.
The selection of proposals is ultimately up to the funding program officers. As with the selection of peer reviewers, there is some concern that some funding organizations may be selecting research proposals specifically to further the organization's agenda.
A Google Scholar search on "biased funding of proposals" reveals some research on bias in proposal selection. However, it appears to be mostly focused on issues other than missions, products and policies. Topics include racial bias, gender bias and avoiding risky projects.
Google Scholar gives about 25,000 hits for documents containing all three of the terms "proposal", "funding" and "bias" in full text, for the five year period 2010-2014. Some of these relate to bias in selecting proposals for funding.
When a proposal is selected for funding there may be some form of public notice, such as a press release, which can be used for bias research. However, the amount of information given may vary from case to case, ranging from a mere abstract to a detailed discussion of the technical proposal. The amount of funding may or may not be disclosed. Some funding organizations provide a compilation of funded proposals, which may facilitate comparisons and the search for funding patterns that might suggest bias in selection.
Unfortunately the many proposals that are not funded are seldom made available. This secrecy makes it much more difficult to look for bias in proposal selection. After all, bias can be as much a matter of which proposals are not selected as it is about which are selected.
Given that dollar amounts are involved there is the potential for quantification of bias in funding. There is also the matter of the number of proposals funded and other measurable features of selection. This might include who receives how much funding, what the funding is for, etc. All things considered the potential for quantification is relatively high for some aspects of bias in proposal selection. The fact that there is little information available about the many proposals that are not selected is certainly a hindrance. 5. Preference for modeling using biased assumptions.
The use of computer modeling is now widespread in all of the sciences. There is a concern that some funding agencies may be funding the development of models that are biased in favor of outcomes that further the agency's policy agenda.
Unfortunately, "bias" is a technical term in the modeling literature, making it difficult to find studies that are looking specifically at funding related bias. Google Scholar estimates about 230,000 hits in the five year period 2012-2014 for studies using both "bias" and "modeling" in their text.
Adding the term "politically" reduces the hits to about 16,000 but these appear to be mostly modeling political processes, not looking at political bias in modeling itself. Many are focused on media bias. The same appears to be true for Google searches.
By the same token, Google Scholar search on "flaws" and "modeling" finds about 22,000 studies but most appear to be about modeling flaws, not flaws in modeling.
Google Scholar full text search on "incorrect model" gives about 2,900 hits but these appear to be mostly technical discussions of modeling or specific models, unrelated to possible funding bias.
There appears to be very little scientific research on potential funding-induced bias in the construction or use of scientific models. This is surprising, given the extent to which models are used in developing and defending paradigms, products, missions and policies. This appears to be a major gap in policy related research. It is possible that a lot of research on biased models is being done in connection with providing comments on proposed regulations and similar policy efforts, where these are based on modeling. Apparently Google and Google Scholar do not cover these document domains.
Assessing funding bias in computer models may be difficult, for several reasons. These models can be very complex and technical. They also may be proprietary, or only run on very large computers. These difficulties may explain the apparent lack of research on funding-induced bias.
One approach might be to mine the technical discussion of the model or models in question, as well as the model documentation. The modelers may even be openly biased. Here the primary challenge may be to relate the bias to the funding organization's policy, product, mission or paradigm in question.
Where the model code is available it may be possible to run it using assumptions that differ from those supporting the agency mission or actions. Or it may be possible to develop an alternative model. Even a relatively simple model can indicate that a more complex model is biased. In some cases there may even be multiple models giving different results. It may also be possible to find modelers who are familiar with the code and who know where biases may be hidden. But due to the highly technical nature of models this may be a difficult line of research.
Note too that modeling bias may be due to the selection or manipulation of input data, rather than to the construction of the model itself. Looking at the input data is a different research approach.
As for quantification, while computer models are mathematical, the assessment of model bias may not be statistical in nature. The goal may be to quantify the magnitude of the bias, rather than the frequency of its occurrence. 6. Biased peer review of journal articles and conference presentations.
This issue is analogous to the potential bias in peer review of proposals, as discussed above. As in that case, this bias may involve rejecting ideas that conflict with the established paradigm, agency mission, or other funding interests.
Peer review bias is the subject of considerable public discussion in the scientific community, as well as extensive scientific research. However, peer review is also used in the selection of proposals to fund and much of the discussion and research does not distinguish between peer review of proposals and articles. Thus there is some overlap between the snapshot provided here and that given under Bias #3 (Biased peer review of research proposals).
A Google Scholar search on articles published 2010-2014 with "peer review" in the title gives about 3000 hits, which suggests a great deal of research. To be sure, some of these hits are false in the sense of not being analyses of peer review, but bias is mentioned frequently in the snippets so a lot of the research is focused on that topic. It appears that most of this research is focused on publications, not proposals. Full text search gives over 200,000 hits. This large number suggests that the term "peer review" probably occurs frequently in passing.
Much of the research into biased peer review occurs within the biomedical community. In part this is probably because issues affecting health and medicine can be quite serious. In addition, biomedicine is a very large research area, compared to the other specialties within science. For example, the US Federal basic research budget for the NIH is larger than the combined budgets for all other forms of basic research.
The biomedical community even has a regular gathering on the issue of peer review and publication. This is the "International Congress on Peer Review and Biomedical Publication" which is held every five years. The Seventh Congress was held in 2013, with 47 presentations and 63 posters 4 .
Biased peer review of articles submitted to journals is already an active research area, so the primary challenge is to focus on the policy, product, mission or paradigm-supporting aspect. Unfortunately, just as with proposals, the journal peer review process is typically not publicly available. This is especially true for those articles that are rejected.
Neither the submissions or the reviews, or even the names of the reviewers, are typically available for bias analysis. There are beginning to be exceptions to this secrecy. Some journals are even making the reviews public, especially for the accepted articles.
One might be able to arrange with the publisher to gain access to this secret data, especially if the scientific issue in question is quite narrow. Journals have probably become sensitive to the issue of bias. In many cases this issue might fall to the editorial board, not the publisher. They might even welcome some analysis.
Thus the prospects for bias research might be limited in this case, just as in the case of proposals, because of secrecy. Or the bias research might involve indirect methods.
For example, one might survey the researchers in the pool of candidates from which the reviews for a given journal are likely to have been drawn, looking for evidence of bias. A journal might even make its reviewer list available for analysis, when it will not do so for individual articles.
Suppose one has the journal's list of reviewers and there is a related controversy regarding an agency's policy or paradigm. If the relevant reviewers can be classified according to their position on the controversy, then the list can be tested for its balance of representation. Of course this assumes that all reviewers carry equal weight so it is a relatively rough test. For example, in the case of the climate change debate one could look for skeptics versus warmers on the reviewer list.
In any case the prospects for simple quantification would seem to be limited, with a lot of interpretation required. Getting good data is the first research challenge. 7. Biased meta-analysis of the scientific literature.
Meta-analysis refers to studies that purport to summarize a number of research studies that are all related to the same research question. For example, meta-analysis is quite common in medical research, such as where the results of a number of clinical trials for the same drug are examined.
There is a sizeable literature in at least two fields on bias in metaanalysis. These fields are clinical medical trials and psychology. Some sample articles include work by Mueller et al. 5 and by Ferguson and Brannick 6 .
Given that meta-analysis bias is already a significant research area, the challenge is primarily to adapt it to the realm of fundinginduced bias. This would seem to be primarily a matter of doing three things. First, choose the meta-analytical document or documents to be analyzed. Second, identify the specific bias to be analyzed for, then compare the available literature with that chosen for the meta-analysis.
The first choice for meta-analyses to be analyzed might well be documents produced by, or funded by, the funding agency. This is especially true for documents specifically designed to support agency policies. Scientific review articles related to hypotheses which support agency policies are another likely candidate. In some cases the potential bias itself will dictate which documents should be analyzed for bias.
It is not clear that quantification can play a major role in this sort of bias research. For example, if a meta-analysis is found to be ignoring scientific papers reporting negative results, how many such papers there are may not be the issue. This may be more a matter of the strength of evidence, not a matter of counting up the sides.
8. Failure to report negative results.
This topic has become the subject of considerable public debate, especially within the scientific community. Failure to report negative results can bias science by supporting researchers who promote questionable hypotheses.
There is a considerable literature on this topic, often under the heading of publication bias. Google Scholar full text search on "publication bias" for 2010-2014 gives over 22,000 hits, while title only search gives 236 hits. This bias is also termed "reporting bias" with 93 Google Scholar title hits and over 18,000 full text hits. These are relatively large numbers, indicating significant research activity.
There is a plain language listing of related bias types, with good references, from the blog Editage Insights: "Publication and reporting biases and how they impact publication of research" by Velany Rodriguez 7 .
Given that publication bias is already an active research area; the primary challenge is to look for bias that is related to funding or which supports funding organization needs. Thus the starting point is probably the agency policy or paradigm.
A lot of this research is quantitative because it looks at bodies of research results, rather than at individual results. Publication bias is typically a pattern, not a single action. The scope may vary from a single journal up to an entire field. 9. Manipulation of data to bias results.
Raw data often undergoes considerable adjustment before it is presented as the result of research. There is a concern that these adjustments may bias the results in ways that favor the researcher or the agency funding the research.
A full text Google Scholar search on "data manipulation" for the five year period 2010-2014 yields about 19,000 results. However, it appears that most of these are about tools and methods for benign data processing. A few address manipulation as a form of bias.
Thus there is an ambiguity in the use of the term data manipulation. Sometimes it refers to benign data processing but at other times it refers to questionable manipulation. However, it is clear that there is a significant body of research into the latter, which means the biased form of data manipulation.
Another approach to the literature is from the direction of scientific fraud, even though bias need not be fraudulent. A full text search on "fraud" and "data manipulation" for the period gives about 1,200 hits. Searching on "fraudulent" and "data manipulation" gives over 6,000 hits. Clearly the scientific community is concerned about fraudulent data manipulation and this is a significant research area.
The kind of funding-induced bias we are concerned with here falls somewhere in between benign data processing and outright fraud. While that middle ground exists in the literature it is not easy to find. Clearly this is a complex issue.
Given that there is already active research into possible bias in data manipulation, the principal challenge seems to be to focus some research on possible cases of funding-induced manipulation. It is likely that this research will involve specific cases, rather than statistical patterns. However, the manipulation itself will often be quantitative.
10. Refusing to share data with potential critics.
A researcher or their funding organization may balk at sharing data with known critics or skeptics, because of the negative effect it may lead to.
Data sharing is a major topic of research and discussion within the scientific community. Google Scholar returns about 29,000 full text hits for "data sharing" for the five year period 2101-2014. Searching on titles gives about 1,600 hits. These are relatively large numbers.
Many of these articles are related to policy issues promoting data sharing, while many others are about specific cases, especially data repositories. (There is also a different use of the term, related to the design of computer network systems.) There appears to be little work directly focused on not sharing data, especially for funding related reasons, although the general topic of not sharing data may be discussed in passing in articles promoting data sharing. In fact a full text search on "not sharing data" returns about 160 hits. Many of these articles are reporting surveys exploring researcher's reasons for not sharing their data.
There are, however, some well known cases of scientists refusing to share policy relevant data. In the US one of the most prominent is the so-called Six Cities study regarding the long term health effects of airborne fine particulates. See for example the work of Kabat 8 . A Google search on "Six Cities study controversy" (without the quotation marks) provides many additional sources.
It appears that despite these prominent cases there is relatively little research into the practice of refusing to share data that is used to support funding organization policies, products, missions or paradigms.
Research in the area of refusing to share data because of its policy implications might be largely anecdotal. That is, one might look for allegations. Another possible approach might be to analyze agency Freedom of Information requests, to see how many pertained to attempts to get policy relevant data. Here the results might well be quantitative.

Asserting conjectures as facts.
It can be in a researcher's, as well as their funding organization's, interest to exaggerate their results, especially when these results support an agency policy or paradigm. One way of doing this is to assert as an established fact what is actually merely a conjecture.
Speculation is a widely used term. Google Scholar lists over 1300 occurrences of "speculation" in titles for the period 2010-2014. These appear to be mostly studies related to forms of financial speculation. Search for the term occurring anywhere in the text during this period gives over 70,000 hits, many of which are probably incidental.
Narrowing the term to "scientific speculation" gives about 800 full text hits, just 5 in titles. Here there is interesting work in the computational biology community using semantic analysis to try to identify speculative statements. These approaches may well be applicable to the problem of speculation presented as fact.
Much of this semantic research also uses the term "speculative statements" and Google Scholar search on that term gives about 240 occurrences in full text and 2 in titles, for the period. Many of these occurrences appear to be from this relatively small research community. A sample article is by Malhotra 9 .
The bias of asserting conjectures as facts is largely semantic in nature. It can occur anywhere in the life cycle of research, from agency documents to journal articles and media reports. It is basically a form of exaggeration, but with an epistemic dimension, claiming to know what is in fact not known.
There are various forms of semantic research that might be used to look for this bias in relation to agency policy. In a simple case one might first isolate key claims that are controversial, look for patterns of assertion that express them as settled. One might also look for the inclusion of policy prescriptions in the statement of the science.
A broader analysis might look at the language being used in presenting the science. Good scientific writing is carefully crafted to be cautious. Terms like suggests, possibly, likely, may, might, etc. occur frequently when conclusions are stated. The lack of this sort of qualifying language might be diagnostic for the bias of asserting conjectures as facts. Semantic techniques like term vector similarity might be useful here.
A lot of semantic analysis is quantitative in nature, especially when terms or occurrences are being counted. This is likely to be the case when one is gauging the level of confidence.

False confidence in tentative findings.
Another way for a researcher, as well as their funding agency, to exaggerate their results is by claiming that they have answered an important question when the results actually merely suggest a possible answer. This often means giving false confidence to tentative findings.
Google Scholar reports about 2500 articles using the exact term "false confidence" in the 2010-2014 time period. However, this term occurs just 5 times in article titles, suggesting that the concept per se is not a focal point for research.
Some are using the term in passing, but in many cases this concept is the point of the analysis. However, these analyses appear to be mostly narrative, with little quantification. In many cases the article is of an editorial nature, see for example Michaels 10 .
All in all it seems that there is relatively little scientific research on the problem of false confidence, even though it is widely discussed.
As with the bias of asserting conjectures as facts, the bias of false confidence is semantic in nature. It can occur anywhere in the life cycle of research, from agency documents to journal and media reports. It is basically a form of exaggeration, but with an epistemic dimension, namely claiming an unjustified weight of evidence for a given finding.
Moreover, as with exaggeration in general, one can look at how results are reported in the media or in press releases, compared to how they are stated in the journal.
A lot of semantic analysis is quantitative in nature, especially when terms or occurrences are being counted. While this seems not to have been done for the problem of false confidence bias in the reporting of research, there is no obvious reason why it cannot be done.

Exaggeration of the importance of findings by researchers and agencies.
Researcher and agency press releases sometimes claim that results are very important when they merely suggest an important possibility, which may actually turn out to be a dead end. Such claims may tend to bias the science in question, including future funding decisions.
For "science" plus "hype" Google Scholar gives over 16,000 hits in a full text search for the period 2010-2014. Many are looking at specific cases where exaggeration may be an issue, often with a theme of "hope or hype". However, the title search returns just 9 hits, a further indication that this language is primarily found in articles about specific cases of possible hype, not in studies of the occurrence of hype in science. A useful introductory article is by Rinaldi 11 .
Then too, a Google Scholar full text search on "exaggeration" and "press releases" gives over 17,000 hits for the period 2010-2014.
Oddly there are just two hits for the combination of these terms in titles, but many of the text hits are in fact on studies of press releases and exaggeration, including in science. Thus this is an active research area, including studies of press releases about scientific findings.
Note that our different types of exaggeration-related bias are not always distinguished. Thus the number of articles related to each type may be greater than is indicated by the literature snapshots.
Exaggeration of importance is a third type of exaggeration, along with presenting speculation as fact and presenting tentative findings with false confidence. Unlike the other two, exaggeration of importance is about the future more than the findings. It is basically a claim about the future direction that science will take because of the findings being reported.
As with the other types of exaggeration, this type is also basically semantic in nature (but without so much of the epistemic dimension). Because it is forward looking it is likely to be characterized by future tense statements, which may even be a semantic basis for finding candidate statements. However, the prospects for quantification are unclear, because this seems to be more a case of specific instances, rather than a pattern of bias.
14. Amplification of exaggeration by the press. The bias due to exaggeration in press releases and related documents described above is sometimes, perhaps often, amplified by overly enthusiastic press reports and headlines.
Google Scholar gives over 5000 hits for "media bias" 2010-2014 with 163 in the title. This literature appears to be found mostly political science, economics and communications journals, with a focus on political cases. However, a full text Google Scholar search on the co-occurrence of the three terms "exaggeration", "science" and "news" for the same period gives over 18,000 hits (with just one occurrence in a title). A significant fraction of these many articles are exploring media exaggeration of scientific reports. Note too that some of the articles returned on searches related to our other types of exaggerationrelated bias may address media bias as well.
The existing media bias research is a good model for research into funding related bias. What needs to be done in some cases is to change the focus from political bias to policy bias. This is not a stretch as the two are relatively closely related. Policy is often the outcome of the political process.
Looking for paradigm supporting bias in scientific reporting may be more difficult. Here it will be necessary to carefully consider the scientific controversies that relate to a given agency's policies. This sort of bias may be more subtle than overt political bias. Nevertheless, the existing research into media bias looks to be a good model. Some of the existing research is quantitative in nature, but much is not. A lot of it seems to be interpretative. An interesting issue here is whether bias and exaggeration come mostly from the media or from the original press releases. A recent quantitative study by Sumner illustrates a useful approach 12 .
15. More funding with an agenda, building on the above, so the cycle repeats and builds.
The biased practices listed above all tend to promote more incorrect science, with the result that research continues in the same faulty direction. These errors may become systemic, by virtue of a biased positive feedback process. The bias is systematically driven by what sells, and critical portions of the scientific method may be lost in the gold rush.
There appears to be very little research looking at systematic linkages between combinations of the types of bias identified above, and subsequent funding. Some of these types of bias are attracting considerable research on an individual basis, but not in relation to subsequent agency funding.
However, the concept that perverse incentives are damaging science is getting some discussion in the scientific community. See for example the work of Schekman 13 and of Michaels 14 .
In this case one is probably looking for funding that occurs after the other types of bias, where the prior bias supported the funding agency's mission, policy or paradigm. Quantification is certainly plausible, especially given that dollars is one of the measures.

Some conclusions and observations
Some types of bias are being studied extensively and quantitatively. Various aspects of peer review and publication bias, especially in biomedicine, appear to be the most heavily researched types of bias.
The role of funding in inducing bias is frequently alluded to as a potential financial conflict of interest. But it is not the focus of most research, which tends to look more at the practice of bias than at its cause. Thus a new research thrust is likely needed.
The role of government funding in inducing policy-driven bias seems to have received very little attention, even though it may be widespread. There are certain exceptions, most noticeably in the climate change debate and environmental policy in general. But here the attention is more a matter of public concern than one of quantitative scientific research.
The notion of cascading systemic bias, induced by funding, does not appear to have been much studied. This may be a big gap in the research on science policy. Moreover, if this sort of bias is indeed widespread then there is a serious need for new policies to prevent it, both at the funder level and within the scientific community itself.

4.
I confirm that I have read this submission and believe that I have an appropriate level of expertise to state that I do not consider it to be of an acceptable scientific standard, for reasons outlined above. Thank you for the opportunity to review this paper. While it tackles an important subject, we have serious reservations about its approach and conclusions, and do not approve it.
The fact that this is to be published in an "Opinion" section does not absolve the writers from clearly expressing what text is actually opinion, what text is fact, and what facts are used to support their opinion. We would urge more scholarship in how the information is presented. The discussion within each category, what information is available, and how to get it appears very superficial.
There is no information provided as to when the checks in Google Scholar were made, or how often they were made, or if the authors checked the "hits" to verify applicability to the topic at hand. The numbers are not static and can change in time. In Category 9, we get 1540 hits for "fraudulent" and "data manipulation" -they get "over 6000." We get 3590 hits for "fraud" and "data manipulation" and they get "about 1200".
They refer to "Scientific Peer Review" by Lutz Bornmann and say that it is "widely recognized" because Google Scholar lists "120 citations for this article". The authors not indicate whether the citations are used in a positive or negative light, or even in the same concept as these authors think. As of 10/17/15, the first 6 citations were by Bornmann himself.
The authors make the statement: " For the purposes of future research the concept of funding-induced bias is analyzed in the context of various practices in science where bias can occur." There is little to no "analysis" provided, other than Google Scholar hits, which is meaningless for analysis as it has to many hidden variables to allow adequate evaluation, followed by suggestions as to how information may be obtained. The suggestions do not provide specific means to obtain information.
The "taxonomy" has no information to justify the 15 categories chosen. They are not clearly defined as to meaning, context, and criteria. They supply no studies to support that any of these numbered items actually existed in form to affect any funding process. They may, but some type of evidence should be presented to provide justification for the category to exist and to be considered a discrete concept.
List number 11 is : "Asserting conjecture as fact." "It can be in a researcher's, as well as their 4.

8.
List number 11 is : "Asserting conjecture as fact." "It can be in a researcher's, as well as their funding organizations, interest to exaggerate their results, especially when these results support an agency policy or paradigm, One way to do this is to assert as an established fact what is merely a conjecture." The authors then list Google Scholar hits for the word "speculation." There is no evidence offered for any associations with titles. There is no evidence offered that the hits for the word "speculation" were even associated with science. Google Scholar draws from a wide variety of indexing databases, and blogs, and various other items with internet access.
The authors offer no evidence for statistically significant associations between the words they use for "hits" and any science-based studies.
Some examples of statements meriting citation support (there are others): "The concept of finding-induced bias is one that specifically occurs in the discussion of and research into some of the fifteen bias research types that we have identified, but not in all of them. It tends to occur where specific funding is the issue". What studies have been done that verify that this even occurs, that the fifteen bias types are indeed areas where bias is possible, and that the tendency to occur with specific funding issues is factual and not conjecture?
"For example, the US Federal basic research budget for the NIH is larger than the combined budget for all other forms of basic research." From where and when was this information obtained?
Under the section "Quantification analysis method issues", the authors write "What is the suggested best method of quantification? In particular is it subjective or objective, that is is human judgment and classification of the data involved, or just simple counting of clearly defined instances." Qualitative studies (based in subjectively-derived data sets) would be the description of the former, and quantitative studies (based on analytical analysis of discrete data) would be the latter. A typical definition for general science can be found via internet sources: " research dealing with phenomena that are difficult or impossible to quantify qualitative research mathematically, such as beliefs, meanings, attributes, and symbols; it may involve content . analysis research involving formal, objective information about the world, with quantitative research mathematical quantification; it can be used to describe test relationships and to examine cause and effect relationships." (Retrieved on 10/19/15 from ) http://medical-dictionary.thefreedictionary.com/Quantitative+research The use of the term in subsequent discussion seems to indicate "comprehensive", not "quantitative".
List number 12 is "False confidence in tentative findings." "Google Scholar reports about 2500 articles using the exact term "false confidence" in the 2010-2014 time period. However, this term occurs just 5 times in article titles, suggesting that the concept is not a focal point for per se research." This statement has several weaknesses in reasoning. 1. "false confidence" is a term that is also used to convey "bravado", and thus the scholar reports may not be conceptually representative of the term as used by the authors. 2. Requiring the exact word usage to convey interest is misleading, as it may be called something different in researchers' vernacular. 3. Noting how many times the words are used in a title that is indexed with Google Scholar is misleading.

11.
how many times the words are used in a title that is indexed with Google Scholar is misleading. The concept may be included under a greater concept, about which there may be numerous research studies published under a different database (Medline, Scopus, etc) 4. Using the term "confidence" in any search function related to research can be misleading as quantitative studies typically use "confidence intervals" in their statistical analysis. These are specifically defined mathematical concepts.
The authors write that "A meta-analysis refers to studies that purport to summarize a number of research studies that are all related to the same research question." That is the wrong description of "meta-analysis", which is not a "summary". Meta-analyses take all the data from the studies they include, and perform analyses as if they are one huge data pool. Statistical results obtained are therefore not "cumulative", as the creation of a large data pool may allow differences in means, trends and associations. Quite commonly, differences in effects are found because the size of the dataset will reduce bias found in smaller samples.
"Definition of META-ANALYSIS: a quantitative statistical analysis of several separate but similar experiments or studies in order to test the pooled data for statistical significance." Retrieved on 10/19/15 from . http://www.merriam-webster.com/dictionary/meta-analysis The authors write that "It is not clear that quantification can play a major role in this sort of bias research. For example, if a meta-analysis is found to be ignoring scientific papers reporting negative results, how many such papers there are may not be the issue. This may be more a matter of the strength of evidence, not a matter of counting up the sides". As discussed previously, it is in these meta-analyses that quantification can and will play a major role. (See definition as given above). If papers are "ignored' without sound rationale, that is indeed an issue, as the sample set is incomplete. However, if papers are "ignored" and not included for issues of poor methodology, missing values, differences in measuring instruments, or inclusion of confounders not defined, then the exclusion decision can be considered reasonable. A good meta-analysis should consider all the available research, and provide sound reasoning for what studies are or are not included.
Under item number 10, the authors write, "A researcher or their funding organization may balk at sharing data with known critics or skeptics, because of the negative effect it may lead to." "There are, however, some well known (sic) cases of scientists refusing to share policy relevant data. In the US one of the most prominent is the so-called Six Cities study regarding the long term health effects of airborne fine particulates." The term "refusing" is connotatively misleading, as indicated in the first three articles (the third was actually from a book chapter) found in the Google search the authors said they run. The issue as described was in participant privacy, which might be considered a HIPPA issue in some instances. The participants had been promised privacy (confidentiality) and the researchers were (by what was in the articles) merely holding to their promise: "The year was 1997, and Dockery had arrived in Washington to tell Congress that because it had promised study participants confidentiality, Harvard couldn't share the raw data from its federally funded Six Cities study" Retrieved on 10/19/15 from . http://www.hsph.harvard.edu/news/magazine/f12-six-cities-environmental-health-air-pollution/ "The authors of both studies have resisted demands to open up their data to public scrutiny. In the case of the Harvard study, for instance, they cite the need to keep the identities and health status 12.
case of the Harvard study, for instance, they cite the need to keep the identities and health status of some 8,000 study subjects in six communities, including Watertown, Mass., confidential. They contend that, even if names and addresses are removed, it would be possible for someone to determine the identities of many subjects based on their age, hometown, and date of death. The controversy poses a test for government officials and scientific researchers, who increasingly are being asked to balance the health care privacy rights of individuals against demands for data from outside researchers, the public, and, politically motivated critics." Retrieved on 10/19/15 from https://www.bostonglobe.com/news/nation/2013/09/06/landmark-harvard-study-health-effects-air-pollution-targe .
"When Harvard researchers published the Six Cites Study suggesting that fine particulate pollution led to an unexpectedly high mortality rate, particulate-emitting industries were understandably concerned. A number of affected industries requested the original data supporting the study, but the Harvard researchers refused, because they were concerned that even the redacted data could be used to identify original study participants who had been assured of confidentiality. " (The citation numbers go to references within the book.) Retrieved on 10/19/15 from https://books.google.com/books?id=Ah6-__otORAC&pg=PA263&lpg=PA263&dq=six+cities+study+controversy .
The authors write that "There appears to be very little scientific research on potential funding-induced bias in the construction or use of scientific models." and "This appears to be a major gap in policy related research" Web of Science gave back 320 hits for "research funding"and "bias" and "modeling." AA is an employee of The Center For Scientific Integrity, which operates Competing Interests: Retraction Watch. IO is executive director of The Center For Scientific Integrity.
We confirm that we have read this submission and believe that we have an appropriate level of expertise to state that we do not consider it to be of an acceptable scientific standard, for reasons outlined above.

Reply to Reviewers by David Wojick and Patrick Michaels
The Reviewers seem to have missed the point of our article, which is simply to provide a taxonomy of bias types, in order to facilitate future research. Instead they have focused on largely incidental statements made in our discussions of various types of bias. These discussions have no bearing on the validity or usefulness of our taxonomy. As a result, their criticism is not constructive.
There are several ways in which a taxonomy can be criticized. It may contain categories which do not belong, or make false distinctions, or be incomplete. The Reviewers have not addressed these issues. In fact the word "taxonomy" occurs just once in the almost 2000 word review, where they claim we have not justified our categories.
These categories are obviously basically the workflow steps of science, from initial budgeting for research funding through to the communication of reported results. In addition we have identified three distinct forms of exaggeration in communication. Allegations of bias have been made for all 7 8 three distinct forms of exaggeration in communication. Allegations of bias have been made for all of these categories and our literature snapshots make clear that the subject of bias is an active research area for many of them. We cannot imagine a better justification than this. This is a very simple taxonomy, with well understood categories. One of us (DW) has been involved in the development of a number of very complex taxonomies. See for example and http://scholarlykitchen.sspnet.org/2013/02/05/a-taxonomy-of-confusions/ . Our bias taxonomy is http://www.cendi.gov/presentations/KOS_OSTI_Energy_Taxonomy.pdf rudimentary and transparent compared to these complex structures.
In addition, we have two new results which we consider important. The first is that there seem to be several major gaps in the research on bias. The second is the potential for bias cascades, which is arguably our most important result. The Reviewers address neither of these findings. The word "gap" only occurs once, in a quote from our report, which is simply dismissed; ironically via an inconclusive three terms Google Scholar search. The number of hits is low and the bulk of hits from three term searches are unlikely to be related to the combined topic. The word "cascade" does not occur at all.
The Reviewers first concern seems to be our use of Google Scholar (GS), which we consider to be a powerful scientometric tool. In their first comment, they claim that GS search results are volatile, offering several search results that are quite different from ours. However, we have rerun our searches and get the same results as before, so the Reviewers must simply be running different searches. The GS Advanced Search feature does allow for a certain amount of flexibility. We find no evidence of serious volatility in GS search results.
Moreover, it is important to understand the purpose of our "snapshot" searches. This is merely to gauge the relative size of the research community for each of the fifteen bias types. We deem rough order of magnitude to be sufficient. That is, are there tens of hits, or hundreds, thousands, tens of thousands, or none? In this context the difference between, say, 3,000 and 6,000 is irrelevant.
The GS searches also may facilitate future research, by pointing people to the relevant communities. In this sense a GS search is a reference to a community, just as a citation is a reference to a paper.
Some of the Reviewer comments seem to suggest that our writing should be more technical, for example, in our brief explanation of meta-analysis. However, our results have implications for science policy, as well as for research, so we have elected to be as non-technical as possible. Policy makers are often not scientists.
The Reviewers also raise a number of broad issues, based on incidental statements made in our discussions. These range from whether 120 citations indicate wide awareness of a paper, to how a taxonomy is constructed? These are indeed interesting questions in the study of science, but they have no bearing on our results. Moreover, the use of (1) citation based metrics and (2) taxonomies are standard practices, not things that we have to explain or justify.
Given that the Reviewers are science writers and journalists, it is perhaps natural that they should raise these broad issues. There also seems to be a difference of opinion regarding the need for better bias research. We are surprised at this, given that the Reviewers are from Retraction Watch.
However, it is well beyond the scope of our article to consider these issues. We therefore find no However, it is well beyond the scope of our article to consider these issues. We therefore find no reason to revise our article.
We find no conflicts of interest. We are the authors of the article. Competing Interests:

Version 1
Reader Comment 11 Oct 2015 , National Center for Nuclear Research, Poland Pawel Sobkowicz The paper touches a very important subject, extending into most domains of research. As a call for actionto understand the obvious and less obvious forms of bias due to funding processes -it is especially important as the "winning ratio" of obtaining funding for research drops to single digits.
My comments below are focused on a few of the discussed mechanisms of biasing the researchespecially those that involve the funding organizations.

The Authors state that
We make no distinctions regarding the source of the funding. The funding organization may be a commercial firm, a non-profit entity or a government agency, or even an individual.
The funding-induced bias, especially in the form of a biased focus (mechanism 1) and funding agency agenda (mechanism 2) have received in recent times, the status of official policies. A perfect example is provided by at the principles of selection of research topics and related competitions in the European Union Horizon 2020 programme. Based on the agenda set out by EU bureaucrats (for the H2020 as a whole) and by local politicians (in the form of the so called "smart specializations", RIS3 documents in each EU country) the Horizon 2020 specifically limits the funding to domains, questions and issues NOT represented in the "master documents". Research topics that are not on the list of the approved ones have negligible chances to obtain the necessary funding. Of course, because the total amount of funding is limited, there would always be proposals/topics that would not get funded. But writing topical preferences (and therefore, limitations) into long term policy documents is something much more profound.
For this reason, there are calls to restore the academic freedom of choosing the topics and methods of research by "funding people not proposals" -but so far, without much success... ------------------------There is an additional mechanism due to the competitive funding culture, not listed among the 15 proposed by the Authors: get funded or starve pressure.
The competitive process through which research teams obtain the funding necessary not only to conduct the research but, sometimes, simply to survive disturbs the scientific practice. As the proposal teams are evaluated often on the basis of past performance, in many cases "failure is not an option". This leads to the practices 7-12 (largely on the side of the researchers themselves). Reporting lack of results, or negative results may kill the chances of obtaining funding for the next proposal. Especially when the original results may kill the chances of obtaining funding for the next proposal. Especially when the original competition was based on a biased agenda and required specific results, impacts, advances etc.
---------------With respect to points 1-4: The funds allocation bias is, in a sense, trivial to analyze at the level of general public funding, such as the EU Horizon 2020 programme. The lack of competitions in certain areas is easy to document.
More difficult is the analysis of the actual funding distributed to the research teams -but one could attempt an analysis of the differences in the evaluation criteria such as the "impact" (defined as one of the three main criteria in the official H2020 documents http://ec.europa.eu/research/participants/data/ref/h2020/legal_basis/rules_participation/h2020-rules-participation_en.pdf ) Specifically, the "impact criterion" is described as ( ) http://ec.europa.eu/research/participants/data/ref/h2020/grants_manual/pse/h2020-evaluation-faq_en.pdf "Evaluators will check that the proposed activities are in line with the type of action implementing the call or topic. They will pay particular attention to key aspects of the award criteria and key elements to be provided as part of a proposal, notably: […] Under the 'Impact' criterion whereby all aspects will receive particular attention, i.e. the extent to which project outputs should contribute to the expected impacts described for the topic, to enhancing innovation capacity and integration of new knowledge, to strengthening the competitiveness and growth of companies by developing and delivering innovations meeting market needs, and to other environmental or social impacts, as well as the effectiveness of the exploitation measures. " A suggestion for research in this field might be as follows: one could then attempt to analyze the distribution of the evaluators' scores for the impact criterion for funded/not funded proposals; a more deep analysis of the ways in which the expected impacts are described by the research teams, for example the degree of copycat measures used to "prove" that the proposal does meet the challenges defined in the competition.
---------------Regarding the exaggeration of the importance of findings (mechanism 13) As the research teams are forced to focus on a constant stream of successes, to ensure continued funding, so must the funding agencies (especially the public ones). The need to justify the spending, especially in increasingly volatile social environments, and to ensure that the funding agencies themselves get the funds from the governments makes this form of bias especially important -because it affects the whole system, not just individual proposals or research subjects.
I declare no competing interests in the above comments. Competing Interests: