Commonly used generative AI models, such as ChatGPT and DeepSeek R1, are highly vulnerable to repeating and elaborating on medical misinformation, according to new research.
Mount Sinai researchers published a study this month revealing that when fictional medical terms were inserted into patient scenarios, large language models accepted them without question — and went on to generate detailed explanations for entirely fabricated conditions and treatments.
Even a single made-up term can derail a conversation with an AI chatbot, said Dr. Eyal Klang, one of the study’s authors and Mount Sinai’s chief of generative AI. He and the rest of the research team found that introducing just one false medical term, such as a fake disease or symptom, was enough to prompt a chatbot to hallucinate a