Personal attributes of authors and reviewers, social bias and the outcomes of peer review: a case study

Frontiers

Frontiers is a large open access scientific publisher, which published its first journal in 2007. In January 2015, Frontiers had a portfolio of 49 open-access journals with over 50,000 researchers serving on its editorial boards and more than 380 academic specialty sections. Papers are published within these sections. Each paper is assigned to a scientist acting as the editor who coordinates the review process, and is responsible for publication/rejection decisions. Reviewers are selected automatically, based on the match between their individual specialties, and key words in articles submitted for review (or can also be assigned manually by the editor). During the review process, reviewers remain anonymous. However, accepted papers carry the names of their reviewers. This gives reviewers a strong incentive not to accept papers before they have reached a good level of quality.

The Frontier’s review system is designed not so much to select papers as to improve the quality of papers in a collaborative, interactive dialog between authors (whose identities and affiliations are available to reviewers) and reviewers (who at this stage of the process are anonymous). In other words, Frontiers adopts a single blind review process. At the beginning of the review process, in which the reviewers review submitted papers independently of each other, they are asked to answer a series of open questions concerning different aspects of the paper (see Table 1). The precise set of questions depends on the nature of the paper (original research, review paper, etc.). Authors answer reviewer questions through an interactive forum. This possibility reduces misunderstandings and can significantly accelerate the submission process.

In the initial, non-interactive phase of the review process, reviewers can express their overall evaluation of the paper on a range of numerical scales (see Figure 1). However, the use of these scales is not mandatory. Nowhere in the process do papers receive an aggregate numerical score. The final decision to accept or reject a paper is taken by the journal editor, based on the overall results of the interactive review process. Acceptance rates are high. Of the papers in the Frontiers database that had reached a final publication/rejection decision on the date when we extracted the data, 91.5% were published and only 8.5% were rejected.

Figure 1. Summary scales used in the Frontiers review system.

IEEE (Spain) and IEEE (international)

The conferences in these two datasets all used WebConf (http://WebConf.iaia.lcc.uma.es), a computerized system for managing the submission and review of conference papers. WebConf was developed by a team from Malaga University led by one of the authors (RC). WebConf implements a classical review process similar to the processes used by Springer, Elsevier and other large commercial publishers. Conference contributions are usually reviewed by three independent reviewers, occasionally by two or four. Reviewers are chosen by the conference program chair, who draws on a database of potential experts called the program committee. In general, the program committee is made up of authors who have previously submitted papers in a particular area of research and have expressed their willingness to act as reviewers. The WebConf system suggests a list of potential reviewers based on the degree of matching between paper topics and reviewers field of expertise. The final selection is based on the judgment of the program chair.

Reviewers express their opinion of a paper in a conference-specific review form in which they assign scores to the paper on a number of separate scales, covering key areas of evaluation (typically including soundness, originality, clarity etc.) and textual comments. Scores on individual scales are usually expressed in terms of categories (typically: poor, fair, good, excellent). The final publication decision depends on the program chair. If one of the reviewers expresses a strongly negative view of a paper, it will typically be rejected. In cases where there is a very significant difference in reviewers’ opinions, the program chair can ask for an additional review. Acceptance rates vary between a minimum of 29.3% and a maximum of 87%. The combined acceptance rate for the seven conferences in the WebConf database was 57.9%.

Data

Frontiers. The Frontiers database includes details of all authors and reviewers for all scientific papers submitted to Frontiers (N=8,565) between June 25, 2007 and March 19, 2012, the name of the journal to which the paper was submitted, the article type (review, original research etc.), the name and institutional affiliations of the authors and reviewers of specific papers, individual reviewers scores for the summary scales shown in Figure 1, and the overall review result (accepted/rejected). At the time of the analysis, 2,926 papers had not completed the review process and were excluded. In another 1,089 cases, reviewers had not assigned numerical scores to the paper, which could not therefore be considered. Our final analysis used 9,618 reviews, for 4,549 papers. Most of the papers in the database come from the life sciences. The majority of authors and reviewers come from Western Europe and Northern America. However, the database contains a substantial number of authors and reviewers from other parts of the world.

Spanish computer science conferences (IEEE-Spain). This dataset includes details of 1,131 reviews referring to 411 papers submitted to three IEEE conferences (CAEPIA2003, SINTICE2007 and JTEL2007). The majority of authors and reviewers for these papers come from institutions in Spain and Portugal. The data provided include the name of the conference to which the contribution was submitted, the type of contribution (poster, short paper, full paper etc.) the name, gender and institutional affiliations of the authors and reviewers of specific contributions, individual reviewers scores and the final decision (accepted/rejected). All the papers in the database are in the area of computer science.

International computer science conferences (IEEE-International). This dataset provides data for 2,194 reviews, referring to 793 computer science papers submitted to four IEEE conferences (AH2002, AIED2003, ICALP2002 and UMAP2011), managed using WebConf and involving authors and reviewers from all over the world. This dataset provided the same data collected for IEEE-Spain.

Normalization of author and reviewer names

Names of authors and reviewers were canonized: accent and symbols were removed; double spaces replaced by single spaces, and upper-case characters replaced with lower-case characters. Names were rewritten in the normalized form <first name, last name>. Intermediate names were omitted. The only role of names in our analysis was to allow identification of author and reviewer gender in the case of the Frontiers dataset. Any errors would thus have only a limited effect on the analysis of gender bias for this dataset. Quality controls on automated gender assignment are described below.

Normalization of institution names

Names of institutions (universities, research institutions, companies) were canonized as above. After normalization, the name of the institution was recoded using the first three words in the full name. The only use of names of institutions in the analysis was to infer their Shanghai ranking. Quality controls on the attribution of Shanghai ranking are described below.

Gender assignment

Neither the Frontiers nor the WebConf databases included data for author and reviewer gender. In the Frontiers case, gender was inferred semi-automatically in a multistep process. First we matched the first names contained in our database to an open source dictionary providing genders for more than 40,000 first names used in different countries (gender-1.0.0.tgz, downloadable from http://pecl.php.net/package/gender). We then used volunteers of different nationalities (Chinese, Egyptian, Indian, Japanese, Korean, Turkish) to assign genders to first names not contained in the dictionary. Additional names were assigned by manually searching for specific authors and reviewers on Google and Facebook. At the end of this process, we were able to assign genders to 96.4% of authors and 87.1% of reviewers. The majority of unassigned names were Asian (mainly Chinese). Genders for WebConf authors and reviewers were inferred manually, with no missing values in the dataset.

To test the reliability of the two procedures and the impact of possible errors, we randomly selected 125 authors and 125 reviewers for each data set, searched the web sites of their respective institutions to find their genders, and compared them against those generated through our automatic process. For the Frontiers dataset, the analysis showed an error rate of 7.5% mostly due to assignment of female gender to authors and reviewers who were actually male. The error rate for the IEEE Spanish and the IEEE international datasets were much lower (0.0% and 5.2% respectively). The difference was probably due to the broader range of countries (and “unusual” first names) in the Frontiers dataset. For reasons we will consider in the discussion, errors in gender assignment are unlikely to have influenced the conclusions of our study.

Assignment of countries/geographical area/English vs. non-English speaking

To analyze potential country, regional and language biases, we assigned each author and reviewer to the country of the institution to which they were affiliated, as listed in the original datasets. Authors and reviewers with multiple affiliations were assigned to the country of the first affiliation listed. In the Frontiers dataset, we deduced country information from the affiliation given by authors and reviewers. Since 9.0% of reviews lacked information on first author affiliation and 5.3% lacked the information for reviewers, it was not possible to deduce the geographical location or the language of their affiliated institutions. In the IEEE (Spain) and the IEEE (International) datasets, authors and reviewers provided country data information directly and the number of missing values was <1.0% in both cases. The geographical region and language of authors’ and reviewers affiliated institutions were determined using the country information. USA, UK, Scotland, N. Ireland, Ireland, Australia, Canada, and New Zealand were classified as English-speaking countries. All other countries were classed as non-English speaking.

University rankings

Authors’ and reviewers’ affiliated institutions were classified in terms of their position in the 2012 Shanghai academic ranking of world universities for the life sciences (http://www.shanghairanking.com/FieldLIFE2012.html) (Frontiers) and for computer sciences (http://www.shanghairanking.com/SubjectCS2012.html) (WEBCONF). The institution names in the Shanghai ranking were normalized using the same procedure used to normalize institution names in the three datasets (see above). To check the quality of our automatic assignment of university ranking, we extracted a random sample of 125 universities from each dataset and checked the university ranking manually, finding error rates of 11.2% for Frontiers, and 9.6% for IEEE (International). The IEEE (Spain) dataset was excluded from the analysis for other reasons (see below).

Calculation of review scores

Frontiers. The Frontiers review process produces a very low rejection rate. This means that the database used for our study contained relatively few rejected papers (N=478). To create a more informative indicator of reviewers’ evaluations, we computed for each paper the average of the scores expressed by each individual reviewer for each of the summary scales shown in Figure 1 (one mean score for each reviewer). A comparison between the distributions of scores for rejected and published papers (see Figure 2) clearly demonstrates the validity of the indicator. However, it should be noted that the indicator is a construction of the authors and played no role in the review process.

Figure 2. Distribution of reviewer scores for published and rejected papers in the Frontiers dataset.

WebConf. The WebConf system asks each reviewer to assign an overall score to the paper he/she has just reviewed. Scores are expressed on a scale of 0 to 10.

Statistical analysis

For the purposes of the study, we define bias as the interaction terms δ_ij in the random intercept model:

y = b + µ_ij + β_ijA_i + γ_ijR_j + δ_ijA_iR_j + ε          (1)

where y denotes the score given in the review, b denotes the random intercept, i indexes properties of authors, j indexes properties of reviewers, and ε is the error term. This method is similar but not identical to the method proposed in 46.

Given a factor F, such as region, the variables A_i and R_j are indicator (dummy) variables indicating that the first author and reviewer belong to categories i and j of factor F, respectively, that is:

IF one author belongs to category i of F, A_i=1, ELSE A_i=0        (2)

AND

IF one reviewer belongs to category j of F, R_j=1, ELSE R_j=0        (3)

Thus β_ij is the fixed effect of author category, γ_ij is the fixed effect of reviewer category and δ_ij is the fixed effect of the interaction between author and reviewer category. Since the expected value of the random intercept b is 0, the fixed effects allow us to estimate the following mean scores:

G_ij = µ_ij + β_ij + γ_ij + δ_ij    author in category i, reviewer in category j

G_i$\check{j}$ = µ_ij + β_ij             author in category i, reviewer not in category j

G_$\check{i}$j = µ_$\check{i}$j + γ_ij              author not in category i, reviewer in category j

G_{$\check{i}$$\check{j}$} = µ_ij             author not in category i, reviewer not in category j.

We define bias B_ij of reviewers from category j of factor F towards authors from category i by the expression:

B_ij = (G_ij – G_i$\check{j}$) – (G_$\check{i}$j – G_{$\check{i}$$\check{j}$}) = δ_ij       (4)

Since the intercept and main effects cancel out, bias is the interaction term δ_ij and does not depend on the main effects β_ij and γ_ij. In other words, it is independent of any general tendency of authors in category i to write better papers than other authors, or of any tendency of reviewers in category to give generally higher scores. In this setting, reviewers from category j are biased in favor of authors from category i, if B_ij > 0 and are biased against authors from category i if B_ij < 0. Bias is significant at a level α, if we can reject the null hypothesis:

H₀ : B_ij = 0

Otherwise, we assume absence of bias.

The majority of papers in our databases had multiple authors. In preliminary studies, we explored statistical models that used this data in different ways: (i) the model used only the properties of the first author, (ii) the model used only the properties of the last author, (iii) the model considered the properties of all the authors. The three approaches yielded similar results (data not shown). In what follows, we apply the first method, unless otherwise stated.

Example

To illustrate the concept of bias defined in (4), consider a factor with two levels such as gender. Let i and j denote female (F), and $\check{i}$ and $\check{j}$ male (M). Then the terms G_ij = G_FF, G_i$\check{j}$ = G_FM, G_$\check{i}$j = G_MF, and G_{$\check{i}$$\check{j}$} = G_MM, have the following meanings:

G_FF   : mean score when first author and reviewer are female

G_FM  : mean score when first author is female and reviewer is male

G_MF  : mean score when first author is male and reviewer is female

G_MM : mean score when first author and reviewer are male

If we assumed that papers by female authors have the same quality as papers by male authors,

G_FF – G_FM > 0       (5)

would imply that female reviewers are biased in favor of female authors and

G_MF – G_MM > 0      (6)

would imply that female reviewers are biased in favor of male authors.

However, we cannot make this assumption. We therefore conclude that female reviewers are biased for or against female authors (B_FF ≠ 0), only if (5) and (6), yield different results. If both are equal and positive, we conclude that females give higher scores than men regardless of the gender of the author. If female reviewers have a positive bias, this always implies a negative bias on the part of male reviewers and vice versa. By construction, this method cannot detect biases shared by all reviewers (e.g. a bias against female authors, or authors from a particular geographical region, shared by all reviewers, regardless of gender or geographical origin).

It should also be noted that, in our example, if B = (G_FF – G_FM) – (G_MF – G_MM) > 0, the following statements are equivalent:

Similarly, if

Given the low number of rejections in our datasets, it was not possible to measure author, reviewer and interaction effects on the acceptance and rejection of papers but only on review scores. Effect sizes were measured using Cohen’s d, which provides a standardized measure of the differences between two means. It is generally believed that effect sizes of 0.2 or smaller are small, that effect sizes of around 0.5 are medium and that effects >=0.8 are large. However, the practical impact of an effect of a given size depends on the size of the reference population. Issues related to the practical impact of our findings will be addressed in the discussion.

Gender

The automatic gender assignment program described earlier assigned genders to first authors and reviewers for 8,024 reviews from Frontiers, 1,131 from IEEE (Spain) and 2,194 from IEEE (International). The relative proportions of male first authors and reviewers (male authors: 71.9% – 75.0%; male reviewers: 75.1% – 79.3%) were similar in all three datasets. In the Frontiers and IEEE (International) datasets, means scores for male first authors were significantly higher than those for female first authors (Frontiers: difference=0.07, p=0.034, d=0.1; IEEE International: difference=0.28, p=0.001, d=0.2). The IEEE (Spain) dataset showed the same pattern but the difference was not significant (difference=0.40, p=0.29, d=0.3). Reviewer gender had no significant effect on review scores in any of the datasets, nor did the interaction between author and reviewer gender have a significant effect in any of the datasets. In brief, none of the datasets showed evidence of gender bias. The significance of these results will be examined in the discussion. Complete data for the analysis can be found in Data Files 1–4.

Region

The analysis examined the role of the region of first author and reviewer institutions in determining review scores, and tested for possible regional bias. Authors and reviewers were grouped into 11 geographical regions (Africa, Australia/New Zealand, Central America/Caribbean, Central Asia, Eastern Asia, Eastern Europe, Middle East/North Africa, South America, Southern Asia, Southern Europe, and Western Europe) according to the location of their respective institutions. To avoid problems with the convergence of the mixed model algorithm and to guarantee the statistical power of the analysis, pairs of first author/reviewer regions with less than 30 reviews were discarded. Distributions of author and reviewer regions differed significantly among the three datasets. In the Frontiers and International IEEE datasets, the majority of authors and reviewers came from institutions in North America and Western Europe, while the majority of authors and reviewers in the Spanish IEEE dataset came from institutions in Southern Europe.

In all three datasets, differences between the scores of first authors from different regions were statistically significant, even after application of the Bonferroni correction for multiple hypothesis testing (see Table 2). In the Frontiers dataset, authors from North America scored significantly higher than authors from all other regions and authors from Eastern Europe, Southern Asia, and Southern Europe scored significantly lower. In the IEEE (Spain) dataset, authors from Southern Europe scored significantly higher than authors from other regions. In the IEEE (international dataset), authors from North America again scored significantly higher than authors from other regions, and authors from Southern Europe scored lower. Effect sizes were small to moderate (Cohen’s d between 0.1 and 0.7).

Differences in scores linked to the geographical location of reviewer institutions were rare (see Table 3). In the Frontiers dataset, South American reviewers gave scores that were significantly higher than those given by reviewers from other regions, even after application of the Bonferroni correction. In the IEEE (Spain) dataset, there were no significant differences in the scores given by reviewers from institutions in different regions (not shown). In the IEEE (International) dataset reviewers from Australia/NZ gave scores that were significantly higher than the average for reviewers from other regions. In all cases, effect sizes were small to moderate (Cohen’s d in the range 0.1–0.5). The number of reviews whose scores may have been affected was small (1.7% of reviews for Frontiers, none for IEEE-Spain, 6.1% for IEEE-International).

To test for bias, we applied the random intercept model to all author/review region pairs with more than 30 reviews (see Table 4). After application of the Bonferroni correction, the Frontiers dataset showed no evidence of interaction between author and reviewer region and the other two datasets showed only very limited evidence (IEEE – Spain: strong bias of reviewers from S. Europe in favor of authors from E. Asia; IEEE – International: strong bias of North American reviewers in favor of authors from Eastern Asia). However, the proportion of reviews potentially affected by these biases was small (None for Frontiers, 4.1% of papers for IEEE-Spain, 1.4% for IEEE-International). None of the datasets showed evidence for regional biases previously reported in the literature (e.g. bias of North American reviewers in favor of North American authors; bias of Australian reviewers in favour of Australian authors). We conclude that regional biases, while sometimes large, are idiosyncratic to particular review systems, and probably have only a limited effect on review results. Full data for the analysis can be found in Data Files 5–8.

Language

We hypothesized that reviewers could be biased against papers written by authors who were not native English speakers. We, therefore, analyzed potential reviewer bias for and against papers whose first authors came or did not come from institutions in English-speaking countries. As a further test, we analyzed potential bias for and against papers, which had, or did not have, at least one author belonging to an institution in an English-speaking country.

The Frontiers and the IEEE (International) datasets both included large numbers of authors and reviewers from institutions in native English-speaking, and from non-English-speaking countries. In contrast, approximately 97% of the authors and reviewers in the IEEE (Spain) dataset came from Spain. Since none of the papers with an English-language first author, and only one paper with at least one English author, were reviewed by an English-language reviewer, it was not possible to measure bias using the random intercept model. This dataset was therefore discarded from the subsequent analysis.

In the remaining datasets, papers with a first author from an institution in a non-English-speaking country scored significantly lower that papers with first authors from institutions in an English speaking country, regardless of whether they were reviewed by native English-speaking or non-native English speaking reviewers (Frontiers: difference=-0.25, p<0.001, d=0.2; IEEE International: difference=-0.54, p<0.001, d=0.5). Reviewer language had no significant effect on score (Frontiers: difference=0.03, p=0.35; d=0.0); IEEE International: difference=0.01; p=0.80, d=0.0). In neither case did we find a significant interaction between author and reviewer language (not shown). Results for papers with at least one author from an institution in an English-speaking country were similar. Details of the analysis are shown in Data Files 9–11.

Ranking of author and reviewer institutions

Reviewers from institutions with high academic prestige could be biased in favor of authors from other high prestige institutions and against authors from lower ranking institutions. To test this possibility, we classified all authors and reviewers in the three datasets by the position of their institutions in the Shanghai classifications, as described earlier. The Frontiers and the IEEE International datasets both contained significant numbers of authors and reviewers from universities in all three categories. However, a large majority of authors and reviewers in the IEEE Spain dataset came from universities in category 3. Given the lack of data for authors and reviewers from higher-ranking institutions, this dataset was excluded from the subsequent analysis.

In the Frontiers dataset, authors from universities in category 1 scored significantly higher than authors from category 2 (difference=0.17, p=0.010, d=0.1) and from category 3 (difference=0.22, p<0.001, d=0.2), regardless of the origin of the reviewer. No significant difference was observed between the scores of authors in category 2 and 3. In the IEEE International dataset, authors from universities in categories 1 and 2 received similar scores and both scored significantly higher than authors in category 3 (category 1: difference in scores=0.74, p=<0.001, d=0.6; category 2: difference in scores=0.66, p=<0.001, d=0.6).

In the Frontiers dataset, there was no significant difference between the scores given by reviewers in different categories. The IEEE (International) data showed no significant differences between scores from reviewers affiliated to institutions in category 1 and category 2 or 3 but a significant difference between scores from reviewers in category 2 and 3 (difference in scores=0.20, p=0.008, d=0.2). Neither dataset showed a significant interaction between the prestige of author and reviewer institutions. Full details of the analysis are given in Data Files 12–14.