<p>Automated evaluation of large language model response concordance with human specialist responses on physician-to-physician eConsult cases</p>

David Wu; Fateme Haredasht; David Wu; Vishnu Ravi; Liam McCoy; Yingjie Weng; kanav chopra; Selin Everett; George Nageeb; Wenyuan Chen; Stephen Ma; Saloni Maharaj; Jessica Tran; Leah Rosengaus; Lena Giang; Olivia Jee; Ethan Goh; Jonathan Chen

doi:10.7490/f1000research.1120313.1

Home Browse Automated evaluation of large language model response concordance...

Poster

NOT PEER REVIEWED

Download

metrics

VIEWS

129

downloads

SEE MORE DETAILS

CITE

How to cite this poster:

Wu D, Haredasht F, Wu D et al. Automated evaluation of large language model response concordance with human specialist responses on physician-to-physician eConsult cases [version 1; not peer reviewed]. F1000Research 2025, 14:1017 (poster) (https://doi.org/10.7490/f1000research.1120313.1)

NOTE: it is important to ensure the information in square brackets after the title is included in this citation.

Automated evaluation of large language model response concordance with human specialist responses on physician-to-physician eConsult cases

David Wu, Fateme Haredasht, David Wu, Vishnu Ravi, Liam McCoy, Yingjie Weng, kanav chopra, Selin Everett, George Nageeb, Wenyuan Chen, Stephen Ma, Saloni Maharaj, Jessica Tran, Leah Rosengaus, Lena Giang, Olivia Jee, Ethan Goh¹, Jonathan Chen

Published 01 Oct 2025 (https://doi.org/10.7490/f1000research.1120313.1)

Author Affiliations

¹ Stanford Center for Biomedical Informatics Research, Stanford, California, USA

Copyright © 2025 Wu D et al. This is an open access work distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Metrics
129 Views
4 Downloads

DOWNLOAD 881.10 KB

CITE

Published 01 Oct 2025

Automated evaluation of large language model response concordance with human specialist responses on physician-to-physician eConsult cases
[version 1; not peer reviewed]

Author Affiliations

¹ Stanford Center for Biomedical Informatics Research, Stanford, California, USA

Competing Interests

No competing interests were disclosed

Keywords

LLM-as-judge, decompose-then-verify, benchmark, ai, clinical concordance, clinical benchmark

Comments

0 Comments

All commenters must hold a formal affiliation as per our Policies. The information that you give us will be displayed next to your comment.

User comments must be in English, comprehensible and relevant to the article under discussion. We reserve the right to remove any comments that we consider to be inappropriate, offensive or otherwise in breach of the User Comment Terms and Conditions. Commenters must not use a comment for personal attacks. When criticisms of the article are based on unpublished data, the data should be made available.

I accept the User Comment Terms and Conditions

Affiliation

Please enter your institution.

Note: To add your institution or organisation, start typing the name and then select the correct name from the list. Where applicable, the name will appear in both the original language and in English. Do not paste in the name. If the name does not appear in the drop-down list, we will display the information you have entered.

Please select your country/region.

Competing Interests

Please disclose any competing interests that might be construed to influence your judgment of the content's validity or importance.

Use of this website is subject to the F1000 Research Limited (F1000) General Terms and Conditions.

Submission of user comments to this website is subject to additional Terms and Conditions. By clicking "I accept the User Comment Terms and Conditions" before you submit your first comment, you agree to be bound by these conditions every time you submit a comment.

Terms relating to user comments

This Agreement shall begin on the date hereof.
Certain parts of this website offer the opportunity for users to post and exchange opinions, information, material and data ('Comments') in areas of the website. F1000 does not screen, edit, publish or review Comments prior to their appearance on the website and Comments do not reflect the views or opinions of F1000, its agents or affiliates. Comments reflect the view and opinion of the person who posts such view or opinion. To the extent permitted by applicable laws F1000 shall not be responsible or liable for the Comments or for any loss cost, liability, damages or expenses caused and or suffered as a result of any use of and/or posting of and/or appearance of the Comments on this website.
F1000 reserves the right to monitor all Comments and to remove any Comments which it considers in its absolute discretion to be inappropriate, offensive or otherwise in breach of these Terms and Conditions.
You warrant and represent that:
1. You are entitled to post the Comments on the website and have all necessary licenses and consents to do so;
2. the Comments do not infringe any intellectual property right, including without limitation copyright, patent or trademark, or other proprietary right of any third party;
3. the Comments do not contain any defamatory, libelous, offensive, indecent or otherwise unlawful material or material which is an invasion of privacy
4. the Comments do not contain any contaminating or destructive features or devices such as viruses, time bombs or coding designed to interrupt, destroy or limit the functionality of this website or any of this website's user's computer equipment or software; and
5. the Comments will not be used to solicit or promote business or custom or present commercial activities or unlawful activity.
You hereby grant to F1000 a non-exclusive royalty-free license to use, reproduce, edit and authorize others to use, reproduce and edit any of your Comments in any and all forms, formats or media.
You hereby agree to indemnify and keep indemnified F1000, its affiliates, contractors and agents from and against any and all losses (including without limitation direct, indirect and consequential loss), costs, claims, damages or expenses of whatever nature and howsoever caused arising directly or indirectly from any breach of these Terms and Conditions or arising from the Comments posted on this website or content contained in any email sent using the facilities provided by the website by you including without limitation as a result of any infringement of any intellectual property or other proprietary rights, libel, defamation, obscenity or the Comments being otherwise unlawful.

Competing Interests

Please disclose any competing interests that might be construed to influence your judgment of the content's validity or importance.