ChatGPT Does Not Pass American College of Gastroenterology Tests

ReachMD Healthcare Image

05/23/2023

by Elana Gotkine

The natural language processing model Chat Generative Pretrained Transformer (ChatGPT) does not pass the American College of Gastroenterology (ACG) Self-Assessment Tests, according to a study published online May 22 in the American Journal of Gastroenterology.

Kelly Suchman, M.D., from the Zucker School of Medicine at Hofstra/Northwell and the Long Island Jewish Medical Center in New Hyde Park, New York, and colleagues assessed ChatGPT-3 and ChatGPT-4 performance on the 2022 and 2021 ACG Self-Assessment Tests. A 70% score or higher was required to pass the assessment.

The researchers found that ChatGPT-3 and ChatGPT-4 scored 65.1% and 62.4%, respectively, on 455 included questions across the two exams.

"Recently, there has been a lot of attention on ChatGPT and the use of artificial intelligence across various industries. When it comes to medical education, there is a lack of research around this potential ground-breaking tool," a co-author said in a statement. "Based on our research, ChatGPT should not be used for medical education in gastroenterology at this time and has a way to go before it should be implemented into the health care field."

More information: Kelly Suchman et al, ChatGPT Fails the Multiple-Choice American College of Gastroenterology Self-Assessment Test, American Journal of Gastroenterology (2023). DOI: 10.14309/ajg.0000000000002320

Copyright © 2023 HealthDay. All rights reserved.

Citation: ChatGPT does not pass American College of Gastroenterology tests (2023, May 23) retrieved 23 May 2023 from https://medicalxpress.com/news/2023-05-chatgpt-american-college-gastroenterology.html

This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.

Facebook Comments

Register

We’re glad to see you’re enjoying Global Neurology Academy…
but how about a more personalized experience?

Register for free