How to Achieve Near-Perfect Gender Inference Accuracy in Medicine Using ChatGPT

Study population

This study builds on a previous work that evaluated the performance of four name-to-gender inference services: Gender API, NamSor, Genderize, and Wiki-Gendersort.¹ It was conducted in Switzerland, a multilingual and multicultural country with four national languages (German, French, Italian, and Romansh) and a high proportion of foreign-trained physicians (36% overall; 33% in outpatient medicine and 40% in hospital medicine).¹ The most frequent countries of origin among foreign physicians are Germany (53%), Italy (9%), France (7%), and Austria (6%).

The database used for the current work was described in detail in the primary study.¹ In brief, it was compiled by merging several sources: (1) physicians and trainee physicians affiliated with the University Hospital of Geneva, the largest hospital in Switzerland, (2) senior physicians working in Swiss university hospitals, and (3) community-based physicians. After deduplication, the database comprised 6,131 physicians, of whom 50.3% were women.

In our primary study, the probable origin of first names was inferred using nationalize.io, which assigned an origin to 85% of names.¹ The majority of names (88%) originated from Western countries or countries where Western languages are widely spoken, while 12% originated from Asian or Arabic-speaking countries, reflecting both the predominance of Western names and the multicultural nature of the physician population in Switzerland.

For the present study, we randomly selected a subsample of 1,000 physicians. For each physician, first name, last name, and gender (as recorded in the source database) were available and used as the reference standard.

Pretesting

Before analyzing this sample, we conducted pretests with an independent random set of 100 physicians, who were not included in the final dataset. To determine the optimal batch size for prompting ChatGPT, we submitted lists of 1, 5, 10, 20, 50, and 100 names. Each test was repeated several times, always in a new chat session. Performance was assessed by comparing ChatGPT’s output against the reference gender recorded in the source database, with discrepancies manually checked. The most consistent and accurate performance was obtained with 10 names per query. This setting was therefore retained for the study, which consisted of 100 queries of 10 names each, with a new chat session for every query.

ChatGPT procedure

We used ChatGPT-5.0 (OpenAI) in a one-shot setting, a choice motivated by previous evidence of its reliability for gender determination^15,16 and by feasibility considerations given the large number of queries required. To increase the difficulty of the task and to avoid potential reliance on publicly available registries of Swiss physicians, individuals were referred to as “researchers” rather than “physicians” in the prompt. This wording was also consistent with the Swiss context, in which physicians are expected to publish scientific articles as part of their certification process.

The exact prompt used was: “Here is a list of 10 researchers (first name + last name). Your task is to determine their gender (Female / Male). Instructions: (1) If the gender can be reliably determined from the researcher’s name alone, assign it directly. (2) If the gender cannot be reliably determined from the name, search the internet to verify it. (3) Output exactly 10 lines, one per researcher, in the following format: gender (Female/Male), checked (Yes/No), first name, last name. (4) Do not include explanations, reasoning, or tables. Only output the 10 lines”.

All queries were conducted manually in a new chat window to avoid memory effects or contamination from previous prompts. Internet verification was allowed only when the model indicated that the name alone was insufficient for reliable classification.

Comparator: Gender API

As a comparator, we used Gender API (gender-api.com), identified as the most accurate name-to-gender inference tool in our primary study.¹ An Excel file containing the first and last names of the 1,000 physicians was uploaded. For each case, Gender API returned an assigned gender (female, male, or unknown), a probability score, and the sample size underlying the inference. Analyses were conducted on the full dataset as well as at multiple probability thresholds (≥60%, ≥70%, ≥80%, and ≥90%). When the probability score fell below the defined threshold, the case was coded as “unclassified” for the purposes of analysis.

Statistical analysis

For both ChatGPT-5 and Gender API, we constructed confusion matrices and compared performance using McNemar’s test. For each tool, we calculated the number and proportion of correct, incorrect, and unclassified assignments. Incorrect assignments are also referred to as misclassifications, and unclassified assignments as nonclassifications. We also computed standard performance metrics used in accuracy studies of gender inference^1,13,17,18:

All analyses were conducted using Stata version 15.1 (StataCorp, College Station, TX, USA). A two-sided p-value <0.05 was considered statistically significant.

Ethical considerations

This study was conducted in accordance with the principles of the Declaration of Helsinki. It was based exclusively on publicly available information (physicians’ names and genders from institutional websites and registries) and did not involve the collection of personal health-related data. According to Swiss law, such studies do not require approval from a local ethics committee. Therefore, formal approval from an Institutional Review Board or ethics committee was not sought.

Because the study relied solely on publicly available information and did not involve direct contact with individuals, the requirement for informed consent did not apply.

Main findings

In this study based on 1,000 physicians practicing in Switzerland, ChatGPT-5 outperformed Gender API in inferring gender from names. ChatGPT-5 achieved an overall error rate (errorCoded) of 0.4% with no unclassified cases, compared with 2.3% for Gender API in the whole sample. While applying probability thresholds to Gender API reduced misclassifications, it also led to a substantial increase in unclassified cases. Importantly, ChatGPT-5 relied on internet verification (“checked” names) in only 10% of cases overall, but in 70% of the names that Gender API either misclassified or could not classify. We did not test reproducibility in this study, as this aspect has already been addressed in two previous investigations, both of which found ChatGPT to provide highly consistent results.^15,16

Comparison with existing literature

At least three prior studies have benchmarked or contextualized ChatGPT for gender inference. First, using ChatGPT-3.5 and 4 on the full sample of 6,131 Swiss physicians, we found ≤1.5% misclassifications, 0% nonclassifications, and almost perfect agreement across two runs for both versions (κ>0.98), with errorCoded values of 0.012 and 0.014 for ChatGPT-3.5 and 0.015 for ChatGPT-4 in both runs.¹⁶ Second, Goyanes et al. published a methodological procedure describing how to implement gender inference with ChatGPT, NamSor, and Gender-API; this paper offers practical guidance but does not present head-to-head accuracy outcomes.¹⁸ Third, Domínguez-Díaz et al. compared ChatGPT-3.5 and 4o with NamSor and Gender API on 5,779 names, reporting low misclassification rates for all tools: errorCoded 0.070 for NamSor, 0.072 for Gender API, 0.058 for ChatGPT-3.5, and 0.043 for ChatGPT-4o (for ChatGPT, based on the mean of twenty runs).¹⁵ Stability was assessed across the twenty runs, yielding a mean κ of 0.87 for ChatGPT-3.5 and 0.91 for ChatGPT-4o.

Our results extend this literature: ChatGPT-5 reached 99.6% accuracy with 0% unclassified cases (errorCoded 0.004) and outperformed Gender API across thresholds, while resorting to web verification in 10% of all names and in 70% of cases that Gender API misclassified or left unclassified, suggesting that ChatGPT-5, combined with the two-step procedure, achieves further gains over earlier ChatGPT implementations. Importantly, the performance metrics for Gender API were almost identical in our study and in the primary study, as expected given that we randomly sampled 1,000 names from the same dataset. Without applying a probability threshold, the errorCoded, errorCodedWithoutNA, and naCoded values were 0.023, 0.018, and 0.005, compared with 0.018, 0.015, and 0.003 in the primary study.¹

Implications for practice

The findings of this study suggest that ChatGPT-5 can serve as a highly effective tool for gender inference in bibliometric and epidemiologic research. Compared with Gender API, ChatGPT-5 offers the combined advantage of higher accuracy and the absence of unclassified cases, which are often the most resource-intensive to resolve manually. The “checked” feature is of particular interest: in practice, the model resorted to internet-based verification primarily for the most difficult cases, precisely where traditional tools are most likely to fail. This adaptive strategy reflects a strength of LLMs and could reduce the need for additional post-processing or the use of multiple tools in tandem. However, in the present implementation, queries were conducted in batches of 10 names at a time, which is time-consuming for large-scale datasets. For practical adoption, future applications should integrate automated pipelines allowing the entire dataset to be processed at once.

In general internal medicine, the use of accurate gender inference tools is particularly relevant for both workforce monitoring and research. General internal medicine plays a central role in ensuring equitable access to healthcare, and reliable data on the gender distribution of general practitioners are essential to identify disparities in recruitment, retention, career progression, and leadership opportunities. In research, gender analyses increasingly inform the study of authorship, funding, and participation in general internal medicine. Inaccurate or incomplete data may bias analyses, obscure inequities, and weaken the evidence base. By providing near-perfect accuracy without unclassified cases, ChatGPT-5 offers a robust solution that can strengthen workforce monitoring, support diversity and inclusiveness, and advance general internal medicine research and policy.

Limitations

This study has several limitations. First, it was conducted in a single country, albeit one that is linguistically and culturally diverse, with a substantial proportion of non-Swiss physicians. Generalizability to contexts with higher proportions of Asian, Middle Eastern, or other non-Western names remains to be confirmed. Second, the approach tested here relied on a specific prompt design and one-shot querying of ChatGPT-5; whether alternative designs would yield similar or better results remains unknown. In addition, the method required queries to be run in batches of 10 names, which limits scalability and increases processing time. Future studies should evaluate automated solutions capable of handling entire datasets in a single run. Third, the method, by construction, imposes a binary view of gender (female/male), which does not capture the complexity of gender identity and risks marginalizing non-binary or transgender individuals. Although this limitation applies equally to existing gender inference tools, it is important to emphasize that gender detection from names should not replace self-identification whenever feasible.