The transparency of quantitative empirical legal research published in highly ranked law journals (2018–2020): an observational study

The credibility revolution and the role of transparency

The “credibility revolution”⁷ responded, in part, to a “crisis”⁸ reported in many fields, in which researchers were unable to replicate the findings of published studies (i.e., they collected new data using the study’s reported methods, but found results inconsistent or not as strong as the original).⁹ Failures to replicate and other controversies were well-publicized and documented in psychology.¹⁰ However, other fields that run adjacent to legal research have not been immune, such as economics¹¹ and criminology.¹² Recently, for instance, economists have described and documented reproducibility failures in studies employing secondary data.¹³

The credibility revolution involves a host of changes to the research process, such as improved transparency, higher standards of evidence, and more replication research.¹⁴

Transparency-focused reforms can make research more efficient because other researchers can leverage open data and materials to test new questions, and to synthesize existing data in meta-analyses.¹⁵ Conversely, research efforts can be wasted in the absence of open data in the sense that those data cannot be obtained by subsequent researchers seeking to reuse them. This is because researchers change email addresses and institutions or leave academic research behind altogether, making them unavailable to share data upon request.¹⁶ Moreover, many researchers who are reachable, decline to share data and materials when they are contacted, or promise to deliver the data but never follow through.¹⁷

Transparency and fuller reporting in the form of data sharing, as well as providing more details of methods and statistical analyses performed, allows other researchers to better scrutinize findings and detect errors in research.¹⁸ For instance, researchers recently discovered a case of data fraud in a study purporting to find that signing one’s name before versus after providing information in a document reduces dishonesty.¹⁹ This study has been cited often for its legal and policy consequences,²⁰ including by the UK Behavioural Insights Team (i.e., Nudge Unit).²¹ Beyond availability of the raw data, which helped other researchers to uncover the fraud, replication also played a role. Failures to replicate other studies in the paper led to increased scrutiny of the entire set of results, which eventually led researchers to take a closer look at the data. One of the authors of the problematic paper, who had worked on the non-fraudulent studies reported within the same article, wrote in response to the discovery of the fraud:²²

Note that this is a quote from Francesca Gino. When we wrote the first version of this article, Gino had not yet been accused of fraud in relation to other studies.²³ That second potential fraud attributed to Gino was also discovered by way of the underlying data being available.

In addition to data and analysis scripts (i.e., code that researchers feed into statistical software packages such as R and STATA to produce reported results), transparency is advanced through preregistration (or prospective trial registration and a pre-analysis plan, as it is called in medical research and economics respectively), which is a time-stamped statement of the research protocols and hypotheses that is posted prior to data collection.²⁴ Preregistration is designed to address publication bias (i.e., the tendency for journal editors to prefer studies that produce statistically significant results) and questionable research practices (i.e., practices that increase the likelihood of publication but decrease the likelihood of successful replication—e.g., producing results using many different empirical models and reporting only statistically significant results).

Similarly, registered reports aim to promote transparency and decrease incentives to engage in questionable research practices.²⁵ Registered reports are studies that begin with peer review of the research plan prior to data collection and are accepted or rejected based solely on the plan and whether the researcher, after collecting data, follows the plan, Early research suggests results from studies published using a registered report protocol contain a more realistic proportion of null results.²⁶

Measuring transparency and credibility-related features of published research

Several metascientific studies, across a variety of fields, have conducted “state-of-the-science” audits, in which recent published studies are randomly sampled and coded for various transparency and credibility-related features.²⁷ These metascientific studies have generally found very low levels of transparency. One study examined psychology articles published from 2014-2017.²⁸ Only about 2% of the studies sampled had available data, approximately 17% had available materials, and 3% were preregistered.²⁹ Note, however, that studies published during this timeframe were conducted in the early days of the reported crisis in psychology.³⁰

While these findings are worrisome, recent reforms in other fields may have led to an increase in transparency related practices in recent years. For instance, journals that implemented open data policies (e.g., requiring open data under some circumstances) show substantial increases in the proportion of studies with open data, albeit with imperfect compliance.³¹

Moreover, a survey across many fields directly asking researchers about when they first engaged in a transparency-related practice (open data, open materials, open code, and preregistration) found that uptake has increased in recent years, suggesting that recent reforms and initiatives are moving the needle.³²

Empirical legal research

Numerous researchers have questioned the credibility of empirical legal research. In a relatively early critique, Epstein and King reviewed all law journal articles published over a ten-year period that contain the word “empirical” in the title.³³ They found numerous errors, generally centering around poor transparency and reproducibility. For instance, many authors had not fully described how they gathered data and then reasoned from that data to their conclusion. Similar critiques have been levied since then, such as reports that empirical legal studies misinterpret statistical results (e.g., p-values), misapply statistical methods, and fail to verify that the assumptions underlying their methods were met.³⁴ Furthermore, author eminence likely plays a biasing role in empirical legal research because student editors may be especially vulnerable to accepting articles based on the status of the author. Even outside of the student context, author status has been shown to affect peer review decisions.³⁵ Most recently, Huber and colleagues found that an article submitted with a Nobel Laureate as corresponding author received over 40% fewer reject recommendations as compared to the same manuscript with a PhD student as corresponding author.³⁶

Matthews and Rantanen conducted the most recent metaresearch on empirical legal research, measuring data availability.³⁷ They sampled from the top 20 journals in the Washington & Lee rankings from 2010-2019, as well the Northwestern Law Review and the Journal of Empirical Legal Studies. They added the latter two because they provided a contrast with the other journals in the sample in terms of peer review. The Northwestern Law Review is one of the rare student-edited journals to routinely seek peer reviews for empirical work, and the Journal of Empirical Legal Studies is fully faculty-edited and peer reviewed. Matthews and Rantanen found low levels of data availability across the 614 articles in their sample, with only 12% making data available without contacting the author. Moreover – and despite its specialization on empirical works and policy encouraging authors to make their data available – the Journal of Empirical Legal Studies underperformed the other journals with only 6% data availability. These results converge with a 2021 study finding that highly ranked law journals implemented almost no transparency guidelines or requirements.

Limited data availability is especially troubling given several other aspects of empirical legal research that sets it apart from cognate fields. For instance, as individuals formally trained in the law rather than in empirical science, many authors of empirical legal work have less methodological expertise than researchers in other sciences. This lack of training may contribute to errors and unfamiliarity with methodological safeguards. The field’s lack of expertise also limits the usefulness of peer review (for journals that do use it).

These factors suggest that transparency is especially important for empirical legal research. For instance, accessible data and analytic scripts and preregistration can assist with error and bias detection. And, other aspects of transparency, such as articles that are openly available and declare funding sources and conflict of interests, help others assign credibility to reported results. Still, outside of the low data availability at elite journals, there is little current knowledge about transparency of empirical legal research. The last large study that assessed a broad array of transparency indicia was conducted 20 years ago. It included only articles with “empirical” in the title³⁸ and the results were not quantified in a way that makes them easy to update and revisit. This study seeks to fill these gaps.

Overview and design

To estimate the transparency of credibility-related features of recent empirical legal research, we examined a sample of 300 law journal articles published between 2018 and 2020. We chose this sample size because it is consistent with many previous transparency studies.³⁹ Based on those authors’ reports⁴⁰ of how long it took them to extract the relevant features of each article, we judged that coding 300 articles was a practical target given our available resources. To provide a comparison between the student-edited journals (that tend to not use peer review, but rather the judgment of student editors to make acceptance decisions) and faculty-edited journals (that tend to rely on peer review), we chose 150 articles from each. We classified articles as empirical if they included original analyses using descriptive or inferential statistics of original or pre-existing quantitative data (e.g., survey studies, content analyses of judicial decisions, meta-analyses).

As described below, we coded features generally related to transparency, such as accessibility, statements about the availability of data, analytic scripts, and other research materials, whether the study was preregistered, and declarations of conflicts of interest and funding sources. We also coded general methodological aspects of those studies, such as whether they were experiments and the types of statistics performed. These provide some background understanding of our sample and may bear on the importance of transparency (e.g., providing analytic code is most relevant to studies using inferential statistics). This is the first study of its kind in empirical legal research, and we are not testing hypotheses; thus, the results should be considered descriptive and exploratory. This study is preregistered and provides open data, code, and materials.

We deviated from previous studies measuring transparency in two main ways. First, previous studies using this type of protocol focused on fields whose journals contain a high proportion of empirical research (e.g., psychology, organizational behavior research, otolaryngology, addiction medicine),⁴¹ so they randomly sampled studies without screening out studies that did not use empirical methods. This approach would have been inappropriate for the current study because it would have led us to include a large number of non-empirical studies (~90% of published work, according to a prior estimate).⁴² As a result, we developed an approach for early screening of non-empirical research (see literature search string below). We also deviated from some previous studies by sampling only from highly ranked journals. This may have biased our results towards finding higher research transparency than the field generally has, because higher rank typically translates to greater selectivity, and thus should in principle enable higher standards. Note also that given the perceived importance of the journals in our sample, low levels of transparency would be especially concerning.

Identifying empirical articles: Search string used to generate sample

To develop a search string to more efficiently identify and sample articles that met our specifications, we conducted a preliminary examination of the literature. We coded 2019-2020 articles from 10 law journals that Washington and Lee ranks in the top 25 (1,024 total articles).⁴³ Through reading those articles, we identified 92 (or 9% of the sample) meeting our definition of empirical within this dataset.⁴⁴

Using the knowledge from that preliminary examination, we first considered two different ways of more quickly identifying empirical articles without reviewing the full text. First, we considered selecting only articles with the word “empirical” in the title as Epstein and King had done in their landmark study. However, only 10% of the empirical articles in the preliminary examination sample had the word “empirical” in their title. This strategy, therefore, would miss a great deal of empirical work, raising concerns about the representativeness of the sample and making it more difficult to find our target of 300 recent empirical studies. We also considered selecting only articles with “empirical” in their abstract; however, that strategy would have missed approximately 50% of the articles identified by the more intensive method used in our preliminary examination.

Ultimately, we decided to use the words in the abstracts of the 92 empirical articles we identified in our preliminary examination, and to write a search string based on those words. That search string is:

One limitation of this strategy is that, in our preliminary examination, about 8% of the empirical articles we identified did not have an abstract. As a result, any search strategy that uses abstract searches is bound to miss a small proportion of empirical articles, such as commentaries with a trivial empirical component. This may bias our findings towards including more instances of systematic data analysis that would be adverted to in an abstract. Despite this limitation, the search method is efficient (i.e., full text searches would have yielded too many false positives for our team to review) and reproducible (i.e., the full search string and results are provided, as are all exclusions and reasons for exclusion).

Sample

Figure 1 details our sampling process and exclusions. We used the search string described above to search Scopus for articles published between 1^st January 2018, and the date of our search, 29^th January 2021. We populated our overall sample of 300 articles with 150 articles from the top 25 student-edited journals from the Washington and Lee rankings (W&L) (based on its “combined score” in 2019) and 150 articles from the top 25 faculty-edited journals (by 2019 impact factor) in the Web of Science’s “law” database.⁴⁵ That is, we applied our search string to both of those journal lists. The Washington and Lee search returned 596 articles and the Web of Science search returned 859 articles (see Extended data). We decided to sample from high impact journals because we judged that these articles would be most influential among both researchers and policymakers, and thus transparency is especially important.

Figure 1. The screening procedure for building the student-edited (W&L) and faculty-edited samples (WoS).

Articles were first identified through the Scopus search string described in the methods. They were then screened for eligibility in random order until the samples were complete. The excluded articles and the reasons for their exclusion are available in the Extended data, “W&L screened out” and "Web of Science screened out”.

Because searches returned several of what we classified as non-empirical articles (e.g., the abstract contained the word “data” to describe data regulation laws), one author (JC) randomly sorted both lists and then screened out articles that did not meet our inclusion criterion (i.e., the study includes an analysis of quantitative data) until we reached the pre-specified sample of 150 articles for each group (Figure 1). Of the 596 articles in the W&L sample, we needed to review 510 to obtain our sample of 150 (i.e., 31% of those reviewed were selected, the rest were excluded). For the Web of Science sample, we needed to review 383 to find 150 empirical articles (i.e., 40.1% of those reviewed were selected, the rest were excluded).

The relatively high rate of exclusions suggests that our search string was overly inclusive, adding more work for us but reducing the chance that we missed a large proportion of empirical articles. The articles screened out and the reasons for their exclusion are described in our Extended data (“W&L screened out” and “Web of Science screened out”). After we initiated coding of these articles with the protocol below, we found that 8 were incorrectly categorized as empirical, so we selected the next 8 from the list as replacements. These are the numbers that are reflected in Figure 1 and above.

Coding procedure

Articles were coded using the structured form developed by Hardwicke and colleagues.⁴⁶ Following the Hardwicke et al., protocol (as well as other transparency coding projects for systematic reviews, see O’Dea et al.),⁴⁷ each article was coded by two of the authors, with disagreements resolved through discussion between those coders and a third author if the coders could not agree (see Extended data). The coders were all trained on five articles and did not begin coding the target sample of articles until they reached consensus on the five training articles. As we discuss below, two items were difficult to code, and so we discontinued coding them and do not present the result for them. For multiple-study articles (we defined studies as distinct data collection activities), we coded only the first-reported study. Coding one article in the student-edited sample took about 30-45 minutes. Coding an article in the faculty-edited sample took about 10-20 minutes. This reflects the longer length of the articles in the student-edited sample and that their methods and data were frequently difficult to locate due to the lack of a standard article format. We coded articles from February to September 2021.

The features of the articles that we coded are detailed in the coding sheet and in Table 1 (and further detailed in our preregistration). Some of these features are relevant background information on the studies, such as the statistics used by the researchers, the nature of the data, and data sources. Others are relevant to the transparency and credibility of the research, such as whether authors stated that data and analysis scripts were available, whether the study was preregistered, and whether it was a replication (replications have helped uncover spurious results in prior studies).

With respect to data availability, Hardwicke et al. attempted to code whether authors provided a clear reference to where the data could be found (“source of data provided but no explicit availability statement”).⁴⁸ Due to difficulty coding this item, they did not report this and instead collapsed these types of data references into “no – there was no data availability statement”. Because we expected the current study to include several cases of authors analyzing pre-existing data and datasets, we initially attempted to preserve this as a distinct item in our coding form. However, our coders also encountered difficulty with it (e.g., sometimes articles would provide a vague reference to another article, and, when we accessed that article, it referenced yet other articles). So, our results also collapse these types of data references into the “no data availability statements” category (as we note below, our data availability results are closely in line with Matthews and Rantanen, lending confidence in our data availability conclusions). We did, however, include a separate item for secondary data studies (Table 1) in which we coded whether authors provided an index of the secondary data items (e.g., references to the judicial decisions included).⁴⁹

We report 95% confidence intervals calculated using the Sison-Glaz method for multinomial proportions.⁵⁰

Deviations from preregistration

Our study deviated from our preregistration in two ways. First, we originally planned to code sample size but did not complete this coding because studies did not provide a single sample size. Second, as noted above, we originally planned to code whether the authors provided the source of the data, but we did not complete this because it was impractical for reasons noted in the previous paragraph.

Sample characteristics

General characteristics of our sample are reported in Table 2, specifically the proportion of articles that: analyzed original or secondary data; used human participants; reported an experiment; were a synthesis (which we operationalized as studies that self-identified as a systematic review or meta-analysis); and reported descriptive or descriptive and inferential statistics. Secondary data analysis was more common (65% of studies, 95% CI = [59%, 70%]) than analysis of original data. Secondary data were also more frequently employed in the student-edited journals (79%, 95% CI = [73%, 85%]) than in the faculty-edited journals (51%, 95% CI = [43%, 59%]). Furthermore, 40% (95% CI = [35%, 46%]) of studies relied on human participants. This figure was 21% (95% CI = [15%, 27%]) among the student-edited journals and 60% (95% CI = [53%, 69%]) among the faculty-edited journals.

Turning to methodology, our sample contained fewer experiments (which require random assignment according to our definition) relative to secondary data analyses (18% of studies, 95% CI = [14%, 22%]). Syntheses were very uncommon, with only six in the sample (all six in the faculty-edited sample). Most articles (68% (95% CI = [62%, 73%])) contained descriptive and inferential statistics (the remaining 32% reported only descriptive statistics). 78% (95% CI = [72%, 85%]) of the faculty-edited articles used inferential statistics versus 57% (95% CI = [49%, 65%]) in the student-edited sample.

Among the 194 articles that used secondary data, 53 or 27% (95% CI = [21%, 35%]) of articles analyzed judicial decisions, 11 (6% (95% CI = [0%, 13%])) analyzed company documents, and a further 11 analyzed statutes or legislation (see “table 2secondary” in Extended data). Human participants were recruited from a variety of groups, with 12 of the 121 articles (10% (95% CI = [2%, 19%])) sampling from university students, 35 (29% (95% CI = [21%, 38%])) sampling from the general population, and 74 (61% (95% CI = [53%, 70%])) sampling from special populations. Those special populations⁵² included difficult-to-reach groups such as judges, young offenders, and government employees (see “table 2 special” in Extended data).

Article accessibility

The articles in our sample were generally easy to access as compared to estimates from previous metascientific studies in criminology and psychology (Table 3, Figure 2).⁵³ 86% (95% CI = [82%, 90%]) of articles had publicly available versions – 100% of the student-edited journal articles and 71% (95% CI = [65%, 79%]) of the faculty-edited group. 70% of articles (95% CI = [65%, 76%]) were gold open access, meaning they were accessible on journals’ websites. This was the case for 100% of the articles in student-edited journals, whereas 41% (95% CI = [33%, 49%]) of the faculty-edited articles were gold open access. Empirical legal researchers also regularly use pre- and post-print services to provide open access versions of their work. 42% (95% CI = [36%, 48%]) of articles in the overall sample were downloadable on SSRN and 22% (95% CI = [18%, 27%]) were downloadable on ResearchGate.

Figure 2. Article availability, funding statements, and conflict of interest statements in empirical legal research.

The left column includes articles from the student-edited sample and the right column is from the faculty-edited sample. Numbers within bars refer to the number of articles that meet the given standard.

Conflicts of interest and funding statements

Turning to conflicts of interest and funding statements, we found that most articles did not provide any such declaration. In fact, only 11% (95% CI = [8%, 15%]) of articles include a conflicts of interest statement. Conflicts of interest statements were more common in the faculty-edited journals with only one article in the student-edited sample containing such a statement. As to statements of funding sources, 40% (95% CI = [35%, 46%]) of articles contained a statement. Again, such statements appear to be rarer in the student-edited sample (see Table 3).

Data availability

The availability of the data, analysis scripts, and materials in our sample was generally low (Table 3, Figure 3). Just 19% (95% CI = [15%, 23%]) of articles provided a statement that data are available. Of articles with data availability statements, the most common means for sharing data were via a third-party repository (39%, 95% CI = [26%, 53%]), by contacting the author (28%, 95% CI = [16%, 42%], and via a personal or institutional website (21%, 95% CI = [9%, 35%]) (see “tabledatahow” in Extended data). We checked whether the data referenced in the statements were readily available (i.e., whether we could access them without further steps, such as contacting the author). Only about half (53%, 95% CI = [40%, 66%]) were readily available, making the effective data availability rate about 10%. This figure closely corresponds to Matthews and Rantanen’s 12% estimate of data availability (also without contacting authors) at predominantly student-edited journals published from 2010 to 2019.⁵⁴

Figure 3. Assessment of transparency and credibility-related characteristics of empirical legal research.

The student-edited sample is reported in the left column and the right column is the faculty-edited sample. Numbers within bars refer to the number of articles that meet the given standard. Data availability, analysis script availability, and preregistration bars include the full sample (150 per group), whereas the bars for materials availability include only the articles that collected original data. Note that this figure reflects availability statements, whereas, discussed in text, actual accessibility was considerably lower.

In the social sciences, much of the move towards providing data availability statements has occurred in the context of psychological research, where original data are often collected. As a result, it may be useful to drill down on articles reporting on original data. Limiting our analysis to these articles (N = 106), we found 29% (95% CI = [22%, 39%]) included a data availability statement, whereas only 13% (95% CI = [9%, 18%]) of articles reporting on secondary data did so (N = 194).

For secondary data, as noted above, we coded the steps authors took to provide information about the dataset. In most cases, authors did not provide any details about the dataset (see “table_secondarySteps” in Extended data). In 26 of the 194 (13%, 95% CI = [8%, 20%]) articles reporting on secondary data, the authors provided an index of the secondary data (e.g., a list of judicial decisions relied on). Several others linked to sources, such as external websites, that were no longer accessible.

Analysis-script availability

Very few studies included a statement about the availability of their analysis scripts (6%, 95% CI = [4%, 9%]). Providing analysis code is especially important when reporting inferential statistics (e.g., to determine the exact statistical test and assumptions the authors used), but of the 203 studies that relied on inferential statistics, only 8% (95% CI = [5%, 12%]) made their code available. Even these figures are somewhat inflated, however, because only for approximately half of the articles with script availability statements could we access the scripts without taking further steps (again, due to dead links and statements indicating that the scripts were available on request).

Materials availability

The materials availability results presented in Table 3 and Figure 3 are limited to studies with original data. We presented them this way because sharing of study materials (e.g., survey instruments, vignettes) is arguably less applicable to analysis of existing data. However, some studies analyzing secondary data do involve useful materials that could be shared, such as the coding sheets used by researchers who tally different sorts of judicial decisions. Of studies that reported on original data, about 44% (95% CI = [35%, 54%]) stated that materials were available. Recall that this figure does not mean that all materials were made available, but rather that authors stated that at least some materials were available. Moreover, we were able to access materials for only 39 of the 47 (83%, 95% CI = [74%, 94%]) studies that stated that materials were available, making the effective material available rate about 37% among studies that report on original data.

Preregistration

Almost no studies reported being preregistered (3%, 95% CI = [1%, 5%]). Of the 8 preregistered studies, we could not access the preregistrations of 2. The purported locations of the 8 preregistrations were: the Open Science Framework (5 studies), the AsPredicted.org registry (1 study), the PROSPERO registry (for syntheses; 1 study), and the Evidence in Governance and Politics (EGAP) registry (which is hosted by the Open Science Framework; 1 study).

Replication

Very few studies stated that they were reporting the results of a replication (4%, 95% CI = [2%, 6%]).