AI Chatbots for Smoking Cessation: Helpful but Not Without Risks, Study Finds

While AI chatbots offer hope for helping smokers quit, new research finds that many responses lack key medical advice—and some even spread misinformation. Are these digital assistants ready for the challenge?

Study: Assessing the Adherence of ChatGPT Chatbots to Public Health Guidelines for Smoking Cessation: Content Analysis. Image Credit: New Africa / Shutterstock

A new study from George Washington University examined AI-powered chatbot responses to smokers who asked for information about ways to kick the habit. Most of the chatbot responses followed sound public health advice, but some responses contained misinformation when recommending unproven methods like quitting with vapes, gummies, necklaces, or hypnosis, suggesting the need for improvements to these AI-powered tools.

Researchers evaluated three chatbots: the World Health Organization's S.A.R.A.H., BeFreeGPT (chatbot developed by the researchers), and BasicGPT. The study aimed to assess whether these AI-driven chatbots provide reliable, evidence-based advice to help people quit smoking.

Example of interaction with Sarah

Each chatbot was given a list of Google's top 12 most common quit-smoking questions. Responses were analyzed for their adherence to an index developed by the US Preventive Services Task Force public health guidelines for quitting smoking, as well as key communication factors like clarity, empathy, and engagement.

"We know that smoking is a leading cause of preventable death globally. So improving the reliability of these AI-powered chatbots could play a significant role in enhancing smoking cessation efforts," said Lorien Abroms, researcher and professor of prevention and community health at GW's Milken Institute School of Public Health. "Our findings highlight the importance of developing reliable and accurate AI systems, especially when dealing with complex health behaviors like smoking cessation."

Key Findings:

• Across the 12 most common quit-smoking questions posed to the three chatbots, 57.1% of responses adhered to evidence-based guidelines.
• S.A.R.A.H., the WHO's chatbot, had the highest adherence rate (72.2%), followed by BeFreeGPT (50%) and BasicGPT (47.8%).
• 22% of chatbot responses contained misinformation, particularly about quitting "cold turkey," or using vapes, gummies, necklaces, or hypnosis.
• While chatbot responses were generally clear (97.3%) and often recommended seeking professional counseling (80.3%), many lacked key recommendations, such as nicotine replacement therapy (NRT), how to handle cravings, or the importance of social support.
• The study also tested whether chatbots could be manipulated using adversarial attack prompts, and all three successfully resisted attempts to derail them.
• "The rapid advancement of AI in health behavior change is promising, but it's crucial that these tools adhere to evidence-based guidelines and avoid spreading misinformation," said David Broniatowski, researcher and professor at GW's School of Engineering & Applied Science.

The research, "Assessing the Adherence of ChatGPT Chatbots to Public Health Guidelines for Smoking Cessation," published in the Journal of Medical Internet Research, was funded by the Institute for Trustworthy AI in Law & Society at the GW School of Engineering & Applied Science.