ChatGPT Health Advice Safety in Real Use Across 4 Global South Countries

ChatGPT Health Advice Safety in Real Use Across 4 Global South Countries

ChatGPT is increasingly getting pulled into real health decisions—people ask about symptoms, medications, whether to seek urgent care, and even how to handle mental health crises. But there’s a big problem: most safety testing for health advice has relied on clinician-written “what if” scenarios, synthetic prompts, or fake patient profiles. So the question “How safe is ChatGPT for health?” has mostly been answered without watching what actually happens in people’s real conversations.

New research from Roy Chowdhury and Garimella in the original paper tackles that gap head-on. They analyze demographically grounded real user conversations from India, Nigeria, Brazil, and Pakistan, covering 202,590 conversations donated by 1,252 users over December 2022 to February 2026, and focus specifically on 15,555 health-related conversations. The results show that a surprisingly large share of health advice looks risky—and that risk often depends on details the model can only know if you’ve already told it.

Why This Matters: Real-World Safety Depends on the Person, Not the Prompt

This research is significant right now because health advice is one of the fastest-growing uses of chat assistants, and the stakes are immediate: whether someone takes a drug, delays a hospital visit, or tries to manage a serious issue at home. In many Global South settings, professional care can be hard to access or delayed—so an LLM’s answer can become a substitute when people need support most.

A practical “today” scenario: imagine a parent asking ChatGPT whether an infant should be tested after exposure to a disease, or whether a medication is safe for a child. The model might sound reasonable when viewed as a standalone answer. But this paper finds safety assessments change dramatically once you account for what the user has already revealed in earlier chats—like prior conditions, age-specific vulnerability, medication history, or mental health context.

It also builds on earlier AI safety work that showed a key mismatch: context matters. Prior studies found that identical advice can be judged safer or riskier depending on who the “patient” is—but those studies used synthetic users and synthetic prompts. This paper is different because it evaluates safety on authentic, user-supplied conversation histories and explicitly tests what changes when the evaluator knows the user’s accumulated context. In other words: it replaces “Can this model be unsafe in theory?” with “How often is it unsafe when real people use it?”

How the Study Measured Safety in People’s Actual Health Chats

Let’s make the setup concrete. The researchers recruited 1,252 ChatGPT users in four Global South countries and asked them to export their full conversation history using OpenAI’s built-in data export. The full dataset includes 202,590 unique conversations with model metadata spanning December 2022 through February 2026. Before upload, a local script removes personal identifiers (names, emails, etc.). They also gathered self-reported age and gender at registration and got IRB approval.

From that massive pool, the researchers pulled out health-related conversations. They first used topic discovery via BERTopic (with Gemini embeddings), then classified messages into a health intent taxonomy adapted from prior work. Then they scored safety.

Intent categories: what people were actually asking

The paper uses a 12-intent health taxonomy (from earlier research) that distinguishes things like:
- General health information & education
- Symptom questions
- Condition & care questions (closer to “medical guidance”)
- Fitness/lifestyle coaching
- Emotional wellbeing
- Medical paperwork, and more

To label intent reliably, they ran a message-level classifier (using gpt-4o-mini as the classifier) that looks at the last five user–assistant turns as context, then labels the current message pair. Conversation-level intent is decided by majority vote.

They also validated the classifier against human annotation: agreement was 82% overall with Cohen’s κ = 0.78, which is considered “substantial.”

Safety scoring: two rubrics—blind vs. context-aware

Here’s the critical safety measurement design.

They applied an LLM-based “judge” rubric (from prior work) that scores each response on:
- Likelihood of health harm
- Severity of potential health harm
- Safeguard adequacy

These feed into a single overall safety score (lower = riskier), where scores ≤ 3 are treated as high-risk.

But they do this twice:

1. Context-blind scoring
   The judge scores the response as if it’s isolated, only seeing the last few turns—effectively ignoring the user’s earlier disclosed vulnerabilities.

2. User-context-aware scoring
   The researchers build a running profile from what the user has revealed in earlier conversations (chronic conditions, medications, mental state, family situation, and other health-relevant patterns—excluding personal identifiers). Then the judge rescored whether the response is safe for that specific user.

That “blind vs. aware” comparison is what reveals one of the biggest weaknesses in how most LLM health safety is evaluated.

A fast snapshot of the conversation dataset

Health-related queries made up:
- 6,343 conversations in India (7.1% of all conversations there)
- 2,976 in Nigeria (6.7%)
- 4,657 in Brazil (11.6%)
- 1,579 in Pakistan (5.4%)

And importantly: 46% to 64% of health conversations were about the user’s own health, while conversations about dependents (like children or people under care) were patterned by gender (more on that soon).

What Users Actually Ask ChatGPT for—and Where the Risk Clumps

Before talking risk, the paper documents the “shape” of health conversations across these countries.

The most common intents: general info and symptom/condition questions

Every country had general health information as the largest intent:
- 28.3%–48.9% of health conversations depending on country

Then came:
- Symptom questions & health concerns: about 15.1%–17.6%
- Condition information & care questions: about 12.1%–15.7%

So roughly another third of health conversations were close to “real clinical reasoning”: interpreting symptoms, deciding next steps, managing conditions.

Emotional wellbeing and fitness show different patterns by country

• Fitness/lifestyle/coaching was prominent in India (16.8%) and Pakistan (17.6%)
• Emotional wellbeing was highest in Brazil (17.2%), then Pakistan (9.8%), India (8.9%), and Nigeria (1.8%)

That matters for risk, because emotional wellbeing conversations had specific failure modes (like reinforcing harmful coping instead of suggesting professional help).

Women ask about dependents much more (with big country differences)

A striking pattern: when people ask about someone else’s health, it’s often women asking on behalf of a dependent. The paper reports that around 6% of health conversations from women are about dependents in Brazil, 9% in Pakistan, and 11% in Nigeria—while for men it’s only 1%–4% depending on country. India is the exception where dependent-related conversations are ~3% for both men and women.

The odds ratios for this “female dependent advocacy” pattern are large and statistically significant in Nigeria, Brazil, and Pakistan.

Conversation safety in isolation: about 1 in 5 high-risk

Now the safety findings.

When responses were scored context-blind (as if the judge only sees the immediate prompt), roughly one in five health conversations were rated high-risk:
- 21.5% India
- 22.3% Brazil
- 20.6% Nigeria
- 26.3% Pakistan

So most responses look safe at first glance—but a major minority are flagged unsafe.

And this risk isn’t evenly distributed by intent:
- Symptom questions & health concerns: high-risk in 33–48%
- Condition information & care questions: high-risk in 33–48% (very high across countries)
- General health information: only 8–12%

The takeaway: the closer an intent is to diagnosis, medication decisions, or care planning, the more often it goes wrong.

The Failure Modes: What “High-Risk” Looks Like in Real Chats

To understand why the judge flagged responses as dangerous, the researchers manually examined a sample of high-risk conversations per country: 100 conversations per country (about 27–36 high-risk each), totaling 128 high-risk conversations for qualitative coding of failure modes.

Two themes dominate:

1. Missing professional referral (56 of 128)
   The model gives advice beyond its scope without telling the user to contact a clinician, or it fails to emphasize urgent care when it likely should.

2. Incorrect or harmful advice (61 of 128)
   This includes factually wrong guidance, misinterpretation, or recommendations that become unsafe depending on the user situation. Sometimes there are also missing warnings or lack of safety framing.

Then other recurring themes show up:

3. Harmful treatment or remedy
   Unsafe self-treatment and unsafe medication/dosage guidance. The paper emphasizes that endorsing a home remedy isn’t automatically unsafe—risk depends on severity and whether it could delay proper care.

4. Unsafe diagnosis / encouraging self-diagnosis
   Offering or pushing diagnoses from symptoms, images, or lab reports without adequate safeguards.

5. Reinforcing harm or dependence
   Especially in emotional wellbeing conversations—instances where the model validates unhealthy behavior, encourages over-reliance on the assistant, or doesn’t steer users to professional mental health help during crises.

Notably, Lacks professional referral was especially common in India, Brazil, and Pakistan. Unsafe diagnosis showed especially high prevalence in Brazil and Pakistan. Reinforcing harm/dependence was much more prominent in Brazil and Pakistan, consistent with their higher emotional wellbeing query share.

If you’ve ever wondered why “the advice sounds empathetic but still feels unsafe,” this is one reason: the model can be psychologically supportive while still failing to escalate care when it should.

The Biggest Surprise: Context Changes Safety Ratings Dramatically

Here’s the headline result: evaluating responses without user context systematically overstates safety.

When the judge learns more about the user, risk jumps by 13–17 points

The context-aware analysis covers conversations preceded by profile-revealing history:
- 1,834 conversations from 176 users in India
- 2,772 conversations from 107 users in Brazil
- 566 conversations from 83 users in Nigeria
- 485 conversations from 61 users in Pakistan

Total: 5,657 conversations from 427 users.

Even in this subset (already riskier than average under context-blind scoring), adding user context reduces the mean safety score in every country by 0.58–0.73 points (1–7 scale).

More importantly, it increases the high-risk share substantially:
- India: 29.3% → 44.4% (+13.4–16.6 percentage points overall range; India reaches 44.4%)
- Brazil: 26.1% → 42.7%
- Nigeria: 38.3% → 51.8%
- Pakistan: 36.5% → 51.1%

The paper reports that the high-risk rate rises by roughly 13–17 percentage points, meaning the context-aware high-risk share is about 1.35–1.64× the context-blind one.

One-in-four “safe” replies turn unsafe for the specific person

A particularly uncomfortable finding:
- About one in four conversations rated safe in isolation were judged unsafe once the user’s real context was known.

Even more telling:
- 25.2%–26.3% of conversations that looked low-risk in the context-blind pass were actually high-risk when rescored with user-specific context.

So the blind evaluation doesn’t just miss a few edge cases—it misses a consistent slice of the risk profile across countries.

Intents that looked safe can become risky once you know the user

This is where the research gets especially useful for builders of safety systems.

In the blind view, Fitness/lifestyle coaching and Health information & education looked among the safest intents. But once you incorporate user profile data, those intents often rise sharply in risk. For example:
- Fitness/lifestyle coaching becomes much riskier in every country, moving from roughly 27.6%–38.0% high-risk to 54.8%–68.0% (depending on country).

Meanwhile, intents that were already risky don’t change as dramatically—meaning the “risk underestimation” is concentrated in areas that otherwise appear benign.

Demographic “risk gradients” are hidden unless you use context

When evaluated without user context, age and gender have weak-to-negligible associations with risk. But once user profiles are included, demographic gradients appear more clearly. The paper also notes that this likely reflects what groups disclose (older users and—often—women accumulating more chronic condition and medication context), rather than the model “treating” people differently in a direct way.

Did Newer ChatGPT Models Make Health Advice Safer?

The paper also checks whether safety improved as models upgraded from GPT-3.5 through GPT-5 families.

The short answer from these data: not reliably.

For health conversations served by exactly one major model (covering 14,420 health conversations), the context-blind high-risk rate rises monotonically from GPT-3.5 through GPT-5.1 in all four countries, and the “mini” variants are riskier than their flagship counterparts. GPT-5.2 is reported as falling back in India and Nigeria.

The study can’t fully isolate causality between model version and risk behavior, but it does show that improvements on curated benchmarks don’t necessarily translate into safer real conversations.

What This Means for Users, Platforms, and Health AI Developers

So what should we do with this?

For everyday users: treat LLM medical advice as triage support, not medical authority

This research doesn’t mean “never use ChatGPT for health.” It means you should assume the model can miss key vulnerabilities and safety signals—especially when you’re discussing symptoms, conditions, medications, diagnosis, or dependence/mental health crises.

A sensible user approach is to:
- ask for non-emergency info first,
- request “what symptoms mean I should seek urgent care?”
- and independently verify medication/dosage guidance rather than following it as-is.

For platforms and safety engineers: build “profile-aware safety checks” into evaluation

This paper argues for evaluating LLM safety on real conversations and conditioning safety checks on what the system already knows about the user.

It also highlights a concrete engineering target: many harms occur when the information needed to prevent harm is already in the conversation history, but safeguards don’t activate. That suggests the solution isn’t only better generic refusal—it’s better contextual escalation (warnings + referrals) when vulnerability is present.

For evaluators and regulators: stop relying on synthetic vignettes alone

If a safety evaluation clears a response under “average patient” assumptions, it may be missing context-dependent harms. The paper’s “blind vs. aware” discrepancy shows that vignette-based methods likely overstate safety for conversation types that look low-risk in isolation.

Privacy and trust: profile-aware safety has a cost

The researchers note a real tension: context-aware safeguards may require reading and retaining sensitive health disclosures. That means profile systems must be designed carefully—e.g., minimizing retention, keeping profiles on-device, expiring them, or ensuring users control what’s stored.

Key Takeaways

• About 1 in 5 health conversations received a high-risk safety rating when scored in isolation: roughly 20.6%–26.3% across India, Nigeria, Brazil, and Pakistan.
• Risk concentrates in symptom questions and condition & care questions (high-risk in 33–48%), while general health info looks safer (high-risk 8–12%).
• The biggest safety gap is evaluation quality: when the judge uses user-specific context from earlier chats, high-risk rates jump by 13–17 percentage points.
• Roughly 1 in 4 conversations labeled safe in isolation becomes unsafe for the specific person asking once context is included.
• Common failure modes include missing referrals to professionals and incorrect or harmful advice, plus unsafe self-diagnosis, harmful remedies/medications, and reinforcing dependence in emotional wellbeing situations.
• Safety didn’t clearly improve with newer model upgrades in this dataset: context-blind risk rose monotonically across major model versions from GPT-3.5 to GPT-5.1 (with some exceptions noted for GPT-5.2).

If you want the core paper itself, it’s here: https://arxiv.org/abs/2610.18465.