Accessible AI depends on what you disclose—and on where it “travels” afterward

Accessible AI depends on what you disclose—and on where it “travels” afterward

Introduction: when disability info gets shared, remembered, and misused

If you use an LLM assistant—ChatGPT, Claude, Gemini—you already know it can be ridiculously helpful. But new research shows there’s a quieter, more sensitive side to that help: when disabled people tell these assistants about disability, that information doesn’t just sit there. It can travel across contexts—especially when the assistant has cross-session memory.

This blog post is based on new research from the original paper by Atieh Taheri, Mahya Tazike, Patrick Carrington, and Jeffrey P. Bigham. The authors interviewed 12 disabled adults in the United States who use LLM-based assistants (at least twice a week, or used them regularly before cutting back). They asked people when, how, and why they disclose disability to AI systems—and how that compares to disclosing disability to human beings.

The headline finding is simple but kind of uncomfortable: people aren’t just deciding what to share. They’re deciding whether it’s appropriate for that disclosure to persist and reappear later in the wrong place.

Why this matters: AI “help” becomes accessibility infrastructure—whether companies designed it that way or not

This is significant right now because conversational AI is rapidly becoming a daily interface for lots of tasks: writing, planning, troubleshooting, health questions, and interpreting documents. For disabled users, those tasks often require access needs (like requesting formats, accessible routes, or feedback style). The research suggests that LLM assistants are no longer just tools—they’re turning into something closer to infrastructure for access.

Here’s a scenario where this applies today: imagine a wheelchair user who asks an assistant for wheelchair-accessible date ideas. Later, without meaning to, they ask the assistant for something unrelated—maybe a generic recommendation list. If the assistant “remembers” the earlier context too aggressively (or applies it through a drift pattern), you can end up with recommendations that don’t match the current need—or you may get “help” that misunderstands your limitations. The problem isn’t the first disclosure; it’s the afterlife of that disclosure.

This builds on earlier AI research about privacy and personalization, but with a disability-specific lens. Prior work often treats privacy as whether data is stored or shared. This paper shows something more nuanced: privacy failures aren’t only about retention—they’re about contextual integrity (whether the information flow fits the norms of the context in which it’s used). The assistant can be “helpful” and still violate those norms.

What “contextual integrity” means when your assistant remembers you

The study uses contextual integrity—a privacy framework by Nissenbaum—to interpret disclosure as more than secrecy. In plain language: privacy isn’t just “don’t tell anyone.” It’s about whether information is appropriate to move from one place to another under certain rules.

In this study, that matters because an assistant with memory changes the information flow:
- to an assistant, you often volunteer disability information (because the assistant doesn’t ask)
- the assistant and the company behind it become two recipients at once
- memory changes a disclosure from temporary/task-scoped to persistent/global

So instead of asking “Did the assistant save it?” the research asks: Was it supposed to be there, for this purpose, for this recipient, in this context? That’s why the paper’s findings sound sometimes contradictory—because the norms differ depending on who you think you’re talking to.

How disabled people disclose disability to assistants: needs, not names—and sometimes diagnoses as shortcuts

One of the strongest patterns from the interviews is that participants often disclose disability in the form of functional needs rather than identity labels.

People often send task-scoped instructions, not disability “facts”

Out of the 12 participants:
- 6 initially described disclosure as something other than a diagnosis—they described an instruction (like “tell me in a format I can use”).
- 2 described very little direct disclosure (one due to low need in text-only interaction, another because they largely stopped using).
- 3 explicitly described a rule like “I only disclose what I need.”

This is what “need-based disclosure” looks like in practice:
- A participant may say they want accessible formatting (rather than “I have X”).
- Someone might request specific advice styles (“constructive, not critical”) tied to a mental health access need.
- A wheelchair user might ask for accessible outings without naming the condition—because the request itself is the needed context.

A key idea is that participants treat disclosure like instructions that unlock access, not like a full identity explanation.

When diagnoses do get mentioned, it’s often in medical tasks—or when people expect the label to “unlock knowledge”

Diagnoses showed up mainly in two situations:
1. Medical tasks (interpreting lab results, discussing medication, etc.).
2. When participants expected the assistant to already “know” something useful about the condition from training data—like a compressed shortcut for many constraints.

But the paper reports that this shortcut sometimes fails: participants did not stop naming diagnoses after mistakes; instead, they corrected the assistant. Still, diagnosis naming influenced what people chose to share overall, because some users reduced disclosure when the assistant’s behavior felt unableist or untrustworthy.

Memory complicates even “minimal disclosure”

There’s a subtle twist: disclosure isn’t only what people type. Some participants reported that the assistant picked up disability context indirectly (from their content, past messages, or patterns). That matters because it means disclosure can happen without a user feeling they “made a choice” to share.

Two recipients at once: why people judge “the assistant” and “the company” by different norms

The paper’s interviews revealed something that can’t be understood by looking only at the assistant’s responses. Participants were often evaluating two recipients simultaneously:
1. the agent (the assistant, as the conversation partner)
2. the platform/company (the entity storing, processing, and potentially reusing the data)

This is where participants split in a way that looks inconsistent until you realize they’re foregrounding different recipients.

Same norm, different recipient → opposite disclosure choices

The paper groups participants into two broad patterns (with a third variation explained later):

• Agent-focused participants often found disclosing to AI easier than disclosing to people.
  They described the assistant as non-judging, factual, and safe.

• Platform-focused participants often disclosed less to AI than to trusted humans.
  They emphasized privacy tradeoffs, worries about retention, and concerns about linking data to identity.

Here’s a structured view of the difference in how recipients shaped disclosure:

One participant added a third twist: they didn’t just see the company as the risk—they saw the assistant itself as capable of evaluating them negatively, based on early experiences where advice felt ableist.

Secondary use was judged differently than identity linkage

Even participants who worried about retention often said secondary use could be acceptable when it’s not tied to identity and serves collective purposes like research. In other words: “privacy isn’t always absolute secrecy.” It’s about what the data becomes and why.

When memory helps and when it “drifts”: over-anchoring, misattribution, and bleed across contexts

Cross-session memory changed everything. It gave some participants relief—but also created new privacy and accessibility risks.

The relief: fewer repeats, more accessibility continuity

7 participants welcomed memory because it reduced the burden of repeating access needs. Examples included:
- remembering constraints so dosing/health-related planning could stay consistent
- resurfacing “accessible ideas” later
- removing the need to re-explain what the user can’t do

For some, memory was not convenience—it was accessibility. One participant described the benefit specifically because their device setup couldn’t remember.

The cost: drift—disability info used where it doesn’t belong

The paper describes three drift patterns that users noticed:

1. Over-anchoring: a remembered detail becomes a repeated “last-line” suggestion even when irrelevant.
   Example: caregiver instructions get applied when caregivers aren’t present.

2. Misattribution: the assistant attaches someone else’s context to the participant.
   Example: mixing up which organization someone works for.

3. Bleed across personas: information meant for one context leaks into another.
   Example: nutrition persona constraints show up during a casual dinner plan.

A fourth issue (also noted) happens when the assistant fails to honor constraints even if it knows them—like recommending something a user said they can’t do.

Two practical tests: relevance and provenance

Participants used two judgment tools to decide whether resurfaced disability information felt appropriate:

• Relevance: Is it connected to the current task?
• Provenance: Does the assistant explain why it’s bringing it up?

Provenance showed up as especially important because it makes memory feel less like a random resurfacing and more like a traceable decision.

How users tried to regain control: boundary work beats toggles for many people

The most practical part of the paper might be this: participants did a lot of “boundary work” to keep information in the right context. And many of those strategies aren’t simple “turn memory off” moves—they’re structural work.

The common strategies they used

Across participants, the paper identifies five families of strategies:

1. Partitioning
   Separate accounts, separate assistants, separate contexts—so overlap is less likely.

2. Profiling (the opposite of partitioning)
   Build a stable profile so the assistant doesn’t need repeated reminders—sometimes still with scoping.

3. Auditing
   Ask the assistant what it remembers, then correct it. One participant described periodic “memory correction.”

4. Minimizing / channel control
   Choose devices or setups so that only the user can see or manage sensitive content.

5. Verifying and repairing
   Double-check accessibility-critical outputs with trusted people, or correct immediately and move on.

The paper notes these strategies are effective but expensive—especially because many memory controls are buried in settings, requiring time, clicks, or technical confidence.

Memory controls: per-message toggles were often not loved

In response to design probes (like:
- a “remember this” control per message
- personas with separate memories
- expiring memory after a chosen period
), participants’ reactions didn’t split neatly into “pro” and “anti.”

A key quote-level theme was that some people found per-message toggles unrealistic because relevance can’t always be predicted at the moment of capture. Since the real norm is about whether info is appropriate later, a “remember/don’t remember right now” toggle can’t fully solve the drift problem.

Instead, participants often wanted control in areas like:
- scope (where it applies)
- access (who can reach stored info)
- retention (how long it lasts)
- provenance (whether the assistant signals why it used stored info)

What they wanted instead (in plain terms)

Participants generally wanted an assistant that:
- reliably acts on stated access needs
- doesn’t treat “a disability label” as automatically permanent for every context
- provides signals like “this answer is based on what you said earlier”
- allows access/retention decisions without forcing complicated ongoing micromanagement

The paper’s design implications are grounded in this: control shouldn’t just be a toggle buried in settings—it should be part of the interaction and traceability.

Key Takeaways

• Disabled people often disclose disability as task-specific access needs, not identity labels or full diagnoses.
• Diagnoses are used strategically: mainly for medical tasks or when users expect the assistant to “decompress” the label into correct constraints.
• Memory changes the privacy problem from temporary/contextual to persistent/global—leading to real issues like over-anchoring, misattribution, and bleed.
• Participants judge whether resurfaced information is appropriate using two tests:
  • relevance to the current task
  • provenance (whether the assistant explains why it’s using stored info)
• People evaluate AI as two recipients at once: the assistant (agent norms) and the company/platform (institutional norms). This drives opposite disclosure choices depending on which recipient feels most salient.
• Many users prefer control over scope, access, retention, and provenance over per-message “remember this” toggles.
• Practical boundary work is already happening: partitioning accounts/personas, auditing memory, verifying outputs, and correcting on the fly—but these approaches are time- and energy-intensive.

If you want the longer, original story straight from the researchers, the paper behind these findings is here: https://arxiv.org/abs/2609.22720.