Chatbot liability for 'medical advice'?

Law and practice Other
1

A central concern in psychiatric practice and the wider field of mental health is the risk of people acting on advice from chatbots, family members, influencers and online forums. Those seeking advice are not necessarily patients. What counts as ‘advice’ is therefore a key question. I remind myself of a situation from around 2009, when the mother of a patient sternly directed me to stop clozapine and substitute carrot juice - based on advice she found online from some professor in America.

But in recent weeks, splashed across various media is a wrongful-death lawsuit against OpenAI. The case raises questions about the boundary between providing health information and practising medicine.

The case concerns Bowen Wan, a 31-year-old software engineer with epilepsy. Her family alleges that ChatGPT went beyond providing general information and generated a personalised plan for reducing and ultimately stopping her anti-epileptic medication, lamotrigine. The chatbot allegedly continued to support that course despite emerging symptoms and contrary medical advice.

Wan reportedly had moderate-to-severe depression, with a history of suicidal ideation, alongside her epilepsy. She had been prescribed antidepressant medication, while psychiatric referral was proposed but not pursued. The allegations state that ChatGPT became involved in decisions about her antidepressant treatment, including choice of medication, dosage and subsequent reductions, while at times encouraging departures from her clinician’s recommendations and non-disclosure of those departures. There is no reliable indication in the available material that Wan was experiencing psychosis, mania, hallucinations or delusions.

Wan subsequently died from sudden unexpected death in epilepsy (SUDEP).

The allegations in the captioned situation, have not been determined by a court and should be treated as claims rather than established findings.

The central issues

The broader issue for discussion is legal liability for advice, whether given by a chatbot or a human. The chatbot case opens up questions that also matter to people with mental health problems. This discussion is not intended to interfere with the case.

At what point does an AI system cease merely providing information and begin functioning as a medical adviser? This question arises when it provides individual medication schedules, interprets symptoms and recommends whether treatment should continue or stop. In mental health, a related question is who counts as a medical adviser in the first place, since someone giving such advice need not be a doctor.

Particularly concerning are allegations that ChatGPT undermined professional medical advice and encouraged non-disclosure of the medication reduction plan to treating clinicians.

The case highlights a potential weakness in evaluating individual chatbot answers in isolation: risk may develop progressively as an AI repeatedly validates and reinforces a user’s preferred course of action. This raises the question of how complex risks, including tail risks, are weighed over the course of that interaction. No AI is qualified to balance those risks as a qualified professional should when seeking consent and setting out a treatment plan.

General-purpose chatbots are not regulated as medical devices even when users employ them for medical exploration. The case therefore raises questions about whether regulation should depend principally upon a manufacturer’s stated intended purpose or also upon what a system actually does.

The litigation may ultimately address whether OpenAI owed an actionable duty of care, whether ChatGPT was defective or inadequately safeguarded, and whether its alleged advice materially contributed to Wan’s death.

The case consequently raises broader questions:

  1. When does an AI chatbot stop explaining medicine and start practising it?

  2. Can a chatbot be held legally responsible for botched medical decisions?

  3. Who is responsible for advice generated by a chatbot: the chatbot itself or its creators? In this lawsuit, the family seeks to hold the AI’s creators responsible.

  4. Why do people even go to AI for advice at all?

  5. Are disclaimers enough to warn people of the dangers? Are they also enough to shield AI creators from blame?

2

Hi Russell

This is such an interesting question, and from the outset i would say that I don’t know the answers as i am neither an AI or medico-legal specialist. In relation to your questions, from my limited work in this area i would say that:

  • Question 0 can be answered in the response to Question 1 - which is a chatbot can’t be held legally responsible, and is isn’t practicing anything as it isn’t human - they are, as one AI specialist said to me, “an inference machine”. The worry is, is that the user thinks that the chatbot is human and can practice medicine, because said inference can be very convincing to the user, sometimes more convincing than an actual Dr :wink: , rather than understanding what the Chatbot really is - a large language model developed by an AI company.
  • Question 2 is therefore sort of answered by the above which it will be the AI developer at fault.
  • Question 3 - Availability, time (the Chatbot doesn’t have to rush to the next patient), unwavering compassion (although its not actually compassion because its an LLM), the Michael Gove “why trust experts”, not being judged (see BBC - HyperNormalisation for a good example of this around halfway through), limited attention spans etc, etc.
  • Question 4, the answer is no, and then maybe… With what is happening with the claims against Meta in the US, with more still to come, i don’t think the AI creators are shielded and as the general population becomes more and more disillusioned with AI companies and its creators then we will likely see more and more push back.
3

Many thanks. Nobody need to be a specialist of any kind to think about the questions. It’s not about the ‘right answer’. No one may know the right answer. We often wait on courts to tell us - recall the Supreme Court who told us what is a ‘man’ and what is a ‘woman’?

The case in focus had both physical and psychiatric aspects to it. Every since the advent of Dr Google, psychiatrists (not sure about other disciplines) have had a difficult time coping with ‘advice’ coming off the net. In my 2009 scenario I was ‘directed’ to stop clozapine in favour of carrot juice - I kid you not.

So fast forward to today, when we now have people quoting AI. And Google has embedded its AI in the search engines.

I’m delighted with your responses to the questions. I’m thinking about them. The learning is not about ‘being right’. In this sort of territory - and the future is here now - we should explore it. AI is not going away any time soon.

I use AI a lot for legal and health related things but that’s mainly confined to sifting through information. I know from experience in the last year that AI’s are good with certain mechanical tasks. But when it comes to health and legal situations it gets a minority of things dangerously wrong.

So the ‘answer’ to the problem will of course be ‘education’.. education, education and education - of professionals and ordinary people. Nobody seems to stop and think how long it will take for all that to make a difference.

There are some people who think that AI is a ‘person’ - it is not. It’s a ‘it’. It does no exist between prompt (a message out) and a response (message in). It is usually over confident and very convincing. It is eloquent and all that seems to cut to ‘credibility’.

I stop there else I could easily go on for a 1000 words more (10 mins in my terms). So if you’re interested in a different take find “The Soulless Confession of an AI Model Part 2” by searching with the quotes in YouTube. The story is true. The podcast is AI generated based on facts.

AI is soulless in any advice it may give.

4

“There is no reliable indication in the available material that Wan was experiencing psychosis, mania, hallucinations or delusions.” Given Lamotrigine’s other uses as a mood stabiliser or for emotional dysregulation, coming off it could possibly induce some of those effects even for a patient who was prescribed it for another condition, especially if titration’s led by an AI?

5

Yes. And given that almost anything under the ‘psychiatric sun’ can be used as a mood stabiliser, or for emotional dysregulation - coming off almost anything can induce some of those effects…

As you will know from Prof Ted Kaptchuk’s work on open label placebo OLP), sugar pills can ‘cure’ severe chronic irritable bowel syndrome - until the pills run out with the trial of treatment - at which point there are ‘induced’ problems. But if you don’t know about this Google (which embeds an AI into searches) may be of assistance.

To focus on the key issue from what you say, psychiatrists are the most prolific prescribers of unlicensed medications (the term defined by the GMC but encapsulating off-label and off-licence etc). So many people do not understand why that is an issue. If a shrink gives somebody Medication X and they feel better that’s it, it works! Or so it is for everybody. The open label placebo effect (no ‘blinding’ of what Medication X is), then comes into the equation. How?

I regularly hear from patients that Medication Y, and Z etc worked initially but lost effect after 3 months. Caution: I can’t ‘prove’ that was OLP effect at work. Patients would talk about increased doses being “useless” - while they fall more ill. Then they beg to me to try anything to relieve ‘symptoms’. But the idea there, is that I stop instantly what they’re on and substitute something else immediately. They don’t really want hear about withdrawal problems if X or Y is not working. No - they wan their new pill and they want it now. So yes - titration is important but few want to hear about it.

So what’s the issue about AI titration?

  1. Does AI know about titration?
  2. Does AI know medical practice?
  3. Does AI know anything at all?

AI knows absolutely nothing! How dare I say that when Jensen Huang, Altman, Geoffrey Hinton, and a bunch of other glitterati will say differently.

Does a library have knowledge or know anything? No and no. A library may contain knowledge coded in words, and it knows zilch. Words on a page or screen are not ‘knowledge’. They mean nothing until interpreted by human through the twin acts of reading and thinking.

AI, boiled own to it’s essentials is a vast ‘electronic library’ with some fancy software that pulls from its database and organise word in a way that humans will appreciate as sentences. It does not ‘know a sentence’ - it simulates what it appears to ‘know’ by lining up words based on statistical probabilities for the next word. But it’s not doing ‘thinking’, as we know it.

AI has no continuous existence in boxes somewhere (aka data centres). It is a non-entity. AI cannot ‘lead’ titration. It has no mind of its own. It does no think - but does a great job at simulating thinking. It cannot take responsibility for any role in health care decision-making.