Memory-Aware Rewarding: Train LLMs to Use Your ChatGPT Memories

Memory-Aware Rewarding: Train LLMs to Use Your ChatGPT Memories

Introduction

If you’ve ever used a chat assistant that “remembers” things about you, you probably had the same thought: Shouldn’t that make the answers better? Intuitively, more context about a person feels like it ought to improve personalization.

New research from the original paper challenges that assumption in a pretty direct way. The authors study memory-aware reward modeling—basically, learning what kind of memory use actually matches what people prefer. Their headline result: having your memories doesn’t automatically lead to responses you’d rather receive, and sometimes it can make things worse.

They run an experiment with about 1,000 pairwise judgments from 87 participants. Each participant provides a memory profile built from their own ChatGPT activity, then judges side-by-side answers to the same queries—one answer generated with retrieved personal memories, and one generated without them. And importantly, the judgments are grounded in their own memory context, so this isn’t just “which response sounds better in general,” but “which response do I actually prefer when my memories are available.”

Why This Matters

This is significant right now because “memory” is rapidly becoming a default feature in assistant-style products. But most systems optimize memory quality using proxy metrics like “did the model retrieve what you stored?” or “can it compress and recall details?” Those are useful—yet they don’t answer the most user-relevant question: Does using memory improve the experience for the person whose information is being used?

Here’s a scenario where this research applies today. Imagine an assistant that remembers you like tech podcasts and your last job role. You come back and ask: “Give me a high-level overview of cybersecurity.” A memory-conditioned assistant might proactively tailor the explanation using your previous profession—sometimes narrowing too much, changing tone, or implying assumptions you didn’t intend. That can feel distracting or even “off,” even if the memory was retrieved accurately. This paper shows that such mismatches aren’t rare edge cases—they can drive systematic preference reversals.

It also builds on (and corrects) a pattern in earlier AI work. Prior research has improved long-term memory by focusing on retrieval, storage, recall, and personalization mechanisms. But the authors argue that accurate recall and successful preference inference aren’t the same thing as “appropriate memory use.” Their work treats memory use as a preference learning problem, and then tries to learn a reward function that aligns with human choices about whether memory should influence the response. That’s the key shift: evaluating memory as behavioral contribution rather than just information capability.

What the Researchers Actually Measured: Memory Use vs. User Preference (Not Just Memory Recall)

The study is carefully designed to isolate memory use from other differences. Participants built a personal memory profile using their own ChatGPT interactions. Then for a set of queries taken from their recent history, the researchers generated response pairs where the only meaningful difference was whether retrieved memories were provided to the model.

How they created the comparison pairs

For each participant:

• They assembled a candidate pool of 100 recent queries (from prior ChatGPT usage).
• For each query q, they retrieved a memory context M_q consisting of nine memory entries.
• They selected a response generator model at random from three: GPT-5.2, Gemini-3.1-Pro, and Claude-Opus-5.
• Then for the same query and same generator, they produced two responses:
  • one without retrieved memories
  • one with retrieved memories

So users judged: “Given this question and this retrieved personal context, which answer would I prefer?” not “Which memory system seems smarter?”

The three memory prompting regimes

Memory-conditioned responses weren’t generated with a single fixed style. The paper varies how strongly the model is encouraged to use memories:

• Conservative: use memories only when they would meaningfully improve the answer
• Ordinary: use memories when relevant and appropriate
• Aggressive: use personal context proactively whenever possible

This is important, because if memory helps only when used sparingly, then a blanket “always personalize” approach could backfire.

What participants judged—and how many data points they ended up with

Participants viewed the query and two responses (A/B side-by-side). They then chose how they felt on a five-point scale:

• A much better / A slightly better / about the same / B slightly better / B much better

They also allowed optional notes.

Raw data: 1,047 total judgments, including 972 preference judgments plus 75 test–retest repeats. After filtering for quality and removing unreliable cases (including repeats from training/evaluation), they retained 821 judgments from 87 participants—specifically:
- 673 strict preferences
- 148 ties

They also ran a small test–retest check: 10 participants returned 3 days later. Directional agreement (whether they preferred the same side regardless of how strong) was 68.0% across 75 comparisons, with stronger “much better” preferences more consistent (73.9% vs 58.6% for “slightly better”). For model testing, they focus on the “much better” subset to get cleaner preference signal.

The Surprising Results: Memory-Conditioned Answers Don’t Reliably Win

Now for the part that’s most likely to make you raise an eyebrow.

Under “ordinary” personalization, preference is basically split

In the main comparison setting (their screened pairs prioritize substantial, memory-grounded differences), under ordinary prompting:

• Memory-conditioned response preferred: 38.7%
• No-memory response preferred: 41.9%
• About the same: the remainder

So simply providing personal memories didn’t create a clear advantage.

Aggregating across prompting styles, preference for memory-conditioned answers drops as the prompting gets more aggressive:

Even the conservative approach only has a small descriptive advantage over no-memory responses. And under aggressive memory use, users strongly prefer the no-memory answer (49.3% vs 34.6% for memory-conditioned).

It’s not uniform across generators (so the “average” can hide patterns)

Another subtle point: memory preference depends on which base model produced the responses. The paper notes that across generators, the preference direction differs—for instance, under ordinary prompting the memory-conditioned response is preferred more often for GPT-5.2 and Claude-Opus-5, but the pattern reverses for Gemini-3.1-Pro.

Because of that, the combined results don’t mean “memory never helps.” Instead, they suggest something more realistic:

Memory helps only when the assistant uses it in a way that matches what the user expects for that specific request.

The first evaluation gap: automatic evaluators may reward the wrong behavior

The researchers also tested whether off-the-shelf evaluators can predict user preference.

They found that automatic evaluators:
- don’t agree well with users
- often favor memory-conditioned responses more often than users actually do

In other words, evaluators may be rewarded for “using memory” rather than “using memory appropriately.”

That mismatch becomes clearer with their accuracy numbers.

Can Automatic Judging Capture Human Preference? (Spoiler: Not Well)

They compare multiple automatic methods against human choices, using the same held-out comparisons.

Preference prediction accuracy (clear “much better” pairs)

The paper reports these accuracies:

A few takeaways jump out:

1. Direct LLM judging isn’t enough. Even with access to the query, the memories, and both candidate responses, it hovers close to the majority baseline.
2. Rubric prompting helps slightly, but still leaves a lot of disagreement with users.
3. The memory-aware reward model does much better, reaching 72.8%—and the authors note it approaches the measured test–retest consistency of 73.9% from their subset.

Another gap revealed: memory use changes what evaluators think is “good”

They also compare how often different systems select the memory-conditioned response. For example (directionally described in the paper):

• The direct judge and Skywork-Reward-V2 tend to pick memory-conditioned answers much more often than annotators do.
• Removing memories from the reward model input reverses its tendency—but accuracy remains near the majority baseline.

So the evaluators aren’t simply “bad at preference prediction.” They appear to be biased toward memory-conditioned responses when memories are provided, even when users don’t actually prefer them.

Enter the Memory-Aware Reward Model: Learn When Memory Should (and Shouldn’t) Influence Answers

This is where the paper becomes more constructive.

Instead of assuming that “relevant memory leads to preferred responses,” the authors build a reward model that explicitly conditions on memory and learns preference patterns from user judgments.

Standard reward modeling vs. memory-aware reward modeling

Traditional reward modeling scores response quality as something like:

• Rθ(q, y) — reward for response y given query q

But the paper changes the interface to make memory an explicit input:

• Rθ(q, y, M) — reward depends on query, response, and the memory context M

That sounds small, but it changes what the model can learn. It lets the reward model learn not only whether the response is high-quality, but also whether the response uses memory in ways that match user preferences.

A rubric-based, interpretable approach (not just a black box score)

They use a rubric idea: break the evaluation into multiple feature dimensions—some about general response quality, and others about memory usage behavior.

Step 1: Induce rubric dimensions from user judgments

They take preference-labeled training examples and ask an LLM to generate rubric dimensions that explain differences between “memory-conditioned” and “no-memory” responses. Then they consolidate near-duplicates and label each rubric dimension as:

• memory-dependent (the score needs the memory contents)
• memory-agnostic (the score can be computed without inspecting the memory)

In one example, they end up with 12 features total, including three memory-related features:
- memory relevance
- assumption proportionality
- unwarranted personal assumptions

…and nine response-quality framing/form features.

They also stress a key methodological point: rubric induction happens separately inside each cross-validation fold, using only that fold’s training participants.

Step 2: Learn how to weigh those features to predict preferences

Then they fit a regularized logistic model that combines feature signals into a probability that a user prefers the memory-conditioned response.

Importantly, this weighting is learned from actual human choices—so the model learns how people want criteria aggregated.

What mattered most (memory features do real work)

When they inspect the learned weights, the top influential features include:

• More headings in the memory-conditioned response (presentation-length effects)
• Relatively longer responses
• Memory relevance
• Assumption proportionality
• Unwarranted personal assumptions (negative weight—users dislike unjustified inferences)

Their interpretation is straightforward and user-centered:

People prefer memory-conditioned responses when the personal context is relevant and the resulting assumptions are justified—not when the assistant “fills in” personal details beyond the query.

Why combining memory signals with general quality signals improves prediction

They also run ablations:

• Memory only: 62.5% accuracy
• Response only: 67.0% accuracy
• Full model (memory + response features): 72.8% accuracy

So memory-aware evaluation isn’t simply “measure memory quality.” It’s more like: the assistant has two jobs—answer well, and use memory appropriately for this specific question.

What You Should Do With This (Practical Implications for Building Memory Assistants)

This work points to a practical shift if you’re building or evaluating memory-augmented LLM assistants.

1) Stop treating “memory retrieval success” as a proxy for user satisfaction

If your offline benchmarks say you retrieved relevant memories, you still might be failing the real user question: Should memory change this answer at all? This paper shows those don’t always align.

2) Evaluate “memory appropriateness,” not just memory presence

Their rubrics include memory relevance and unwarranted personal assumptions. That’s the kind of evaluation that catches over-personalization—where the model feels too narrow, too compliant, or too “aware” of things the user didn’t want invoked.

3) Train reward signals that learn from user preference about memory usage

The authors’ biggest win—72.8% preference prediction accuracy—suggests that learning reward functions tied to memory behavior is a better path than relying on generic LLM judges or pretrained reward models that may over-reward personalization.

4) Use confidence / selective prediction when you must choose

They also find that higher-confidence predictions are more accurate. For example, accuracy increases when restricting to top-confidence cases:
- 100% coverage: 72.8%
- 75% coverage: 76.2%
- 50% coverage: 83.7%
- 25% coverage: 88.8%

So if you’re deploying something like reranking or safety gating for “should we use memories here?”, confidence-based selection could be useful.

Key Takeaways

• Memory-conditioned responses do not consistently win user preference. In the screened comparisons, users preferred no-memory answers slightly more under ordinary prompting (38.7% vs 41.9%), and preference for memory declines further with more aggressive personalization.
• More aggressive “use memories whenever possible” can backfire. Under aggressive prompting, users strongly favored no-memory responses (49.3%) over memory-conditioned ones (34.6%).
• Automatic evaluators are misaligned with users. Direct LLM judging and pretrained reward models often select memory-conditioned responses far more frequently than humans do, with accuracy near the majority baseline.
• The paper’s solution works better: Memory-Aware Reward Model hits 72.8% accuracy. It outperforms majority baseline (55.0%), direct LLM judging (~57–58%), and rubric-based LLM judging (59.6%).
• Memory quality isn’t enough—memory appropriateness matters. The strongest learned signals include memory relevance and assumption proportionality, and a negative signal for unwarranted personal assumptions.
• Building better assistants means learning when memory should influence responses. The authors argue that future evaluation and post-training objectives should align with user preferences about how and whether memories should be used.

If you want memory features to feel genuinely helpful (and not just creepy or distracting), this paper is a strong reminder: the right memory at the wrong moment can make things worse—and your training signals should reflect that.