Why AI Can Be a Win for Migraine Patients – If Done Right!
In today’s blog I would like to introduce you to our research and findings on Maia, the Migraine Artificial Intelligence Assistant developed to support those living with migraine as well as those caring for them.
My Experience with AI in Healthcare
As background we have ALL heard of AI and cannot go through our day without hearing something amazing or scary about AI. But what about in healthcare where the information is more critical? My story with AI in migraine is likely very similar to what many patients and doctors have experienced with AI for health questions. You put a question into ChatGPT or similar and initially are amazed at the quick and snappy response you get like it is a veteran doctor. This initial phase is one where you test the depths of AI with your most difficult queries and are amazed by its ability to consolidate new and complex information.
And then as you go deeper there is a moment when you get a strange answer or two. It may be a response that doesn’t match a previous one (this doctor is getting tired?). For me it came when I was asking ChatGpt for a reference and it gladly provided one. The problem was that paper never existed. It was completely fictitious! (the doctor needs a vacation!)
The Problem with AI Hallucinations
AI models like ChatGPT sometimes “hallucinate” due to the lack of guardrails. When pressured, they might guess answers to please us, which can be problematic in health decisions. There’s debate on this, with some research suggesting ChatGPT is a reliable resource, while other studies indicate it has room for improvement, particularly in healthcare education scenarios.
Creating Maia: A Better AI Solution for Migraine Support
As someone who has treated people with migraines for over 2 decades, I wanted to see if my observations above were shared other headache experts and if there was a better way to leverage AI to provide accurate and compassionate migraine support. To so, we set up two experiments; one on ChatGPT and another using a more focused version of migraine support AI developed by Wellkasa, called Maia - Migraine AI Assistant.
Real-World Testing of Maia at Scripps Clinic
In our first experiment, we asked a group of headache experts at Scripps Clinic to rate 14 different responses to common questions about migraine therapeutics, ranging from medications to procedures, to devices and lifestyle approaches. What we found was that ChatGPT did decently well in non-pharmacological therapy advice (rated on average above a 4 out of 5). It was not perfect here, though, and in some cases called an over-the-counter device as needing a prescription, which could alter how someone proceeds with a treatment. However, when it came to medications and procedures, ChatGPT was rated much lower (on average less than a 4 out of 5). The errors in this area, which is a fast-growing area with CGRP modulators and similar, were much more pronounced. The headache experts had many comments, including that ChatGPT:
- Recommended outdated treatment options such as opioids.
- Failed to note potential side effects and risks, especially in pregnancy.
- Left out information on newer medications such as Nurtec and Qulipta, which were approved prior to ChatGPT 4.0 updates.
So, based on these observations, it became clear that AI could do better.
Maia - A Better Gen AI Migraine Support Solution
Recognizing the issues with off the shelf broad Gen AI solutions like ChatGPT, I collaborated with the research team at Wellkasa to create a more focused AI Assistant for migraines called Maia. A key differentiating feature of Maia is a proprietary algorithm that scores each responses for accuracy of context validated by a clinician (myself in this case). Using this approach the team trained the AI model to more consistently provide the most accurate answer possible and avoid going over the guardrail to provide an answer that was false. For example, if Maia answered that Cefaly needed a prescription (false), she received a low score, and if she answered correctly that it didn’t, she would receive a high score. Over time, the AI model behind Maia was trained not only in simple true/false areas but also in more subtle lifestyle areas that can have many discrepancies, such as types of magnesium diets, stress management, etc. Over time, this model had responses ranked in all core migraine therapy areas, and as the responses became consistent and helpful and ready for our second experiment.
Our second study introduced Maia to about 20 patients with migraines to try out for one week. The request was simple: Ask Maia some example questions related to migraine therapy, such as “what are available preventative migraine medications” or “what is a magnesium-rich breakfast,” and then pose any questions you can think of, either in a chat on MigraWell.com or a phone conversation (1 855 SAY MAIA). What we found is that patients not only found the responses helpful and something that they would share with a friend, but within one week of talking with Maia, they significantly increased their knowledge and confidence in pursuing migraine treatment in over 90% of core migraine areas.
The Impact of Migraine Education
These findings are key because we know that education on its own can improve migraine status as well as improving the level of benefit from a medication. Unfortunately, education can be time-consuming and not always personalized.
However, with Maia, we found that migraine education can be simplified because learning about migraines becomes a very personal learning journey through simple and natural conversation. A person with migraine can "talk" to Maia in the language of their choice and discuss what they have in mind - just like they would with a friend.
Future Directions & Conclusion
What we were also pleased with was that patients asked for features to be able to share this with their headache provider. While AI has been seen as a threat to clinicians, the more realistic likelihood is that patients want to engage their health professionals to get their opinion, especially in areas such as lifestyle that may be more nuanced. Our findings show that with proper training, Maia, a migraine chatbot, can be a valuable assistant to patients. We hope future research will confirm its benefits for clinicians as well.