Pedagogical Fit Score for LLM Tutors: PSI in Action

Pedagogical Fit Score for LLM Tutors: PSI in Action

Introduction: “Correct” Doesn’t Always Mean “Helpful”

If you’ve ever used an AI tutor (or watched one in action), you’ve probably seen this problem: the response sounds confident, it might even be technically correct… but it still doesn’t teach the learner the right way, at the right time.

That’s exactly what new research from Barlog, Craig, and Peng digs into. The core idea is simple: an LLM tutor can give the “right fix” while still being pedagogically mistimed—assuming prerequisite knowledge the student doesn’t have, skipping the reasoning that builds long-term understanding, or introducing concepts out of sequence. In other words, correctness alone is not enough for tutoring quality.

This paper proposes the Pedagogical Suitability Index (PSI)—a theory-informed metric that scores how well a tutor response matches a learner’s current foundation, the course’s concept order, and the timing of instruction. And the authors don’t stop at evaluation: they use PSI as structured feedback to improve weak tutoring responses. Their results show measurable gains—most notably, an 82.3% improvement rate on selected weak cases after PSI-guided regeneration.

Why This Matters: Tutoring Needs “Instructional Timing,” Not Just Answers

This research feels especially relevant right now because we’re past the stage where “Can the model answer the question?” is the main bottleneck. Most strong LLMs can produce fluent, often correct explanations. The real bottleneck is instructional alignment: does the tutoring message match where the student is in their learning journey?

In real deployments, tutoring systems typically lack rich learner state (what concepts the student mastered, what they’re ready for, what was introduced last). PSI directly targets that missing piece: it evaluates responses based on learner readiness, prerequisite structure, and pacing—things a purely accuracy-based evaluator tends to miss.

Here’s a concrete scenario you could apply today: imagine a platform used in an intro programming course. A student asks, “Why did my code fail?” The AI provides an answer like “Your issue is due to how list comprehensions evaluate,” complete with a working example. It might be correct in general—but if the student hasn’t learned list comprehensions yet, that “help” can actually derail learning. PSI is designed to detect these kinds of mismatches (like temporal prerequisite violations and wrong knowledge calibration) and then push the model toward responses that scaffold the next step instead of jumping ahead.

This work also builds on earlier AI tutoring research, but with a sharper lens. Prior efforts often focus on correctness, readability, or knowledge tracing. PSI’s contribution is to unify multiple teaching-relevant dimensions into one composite score while keeping the ability to diagnose why a response is a poor fit.

What PSI Actually Measures: Six Pedagogical “Fit” Signals Combined

The Pedagogical Suitability Index (PSI) is a composite score made from six normalized sub-scores, each representing a different part of “good tutoring.” The authors use equal weights across the six components (w_i = 1/6 each), since there wasn’t a clear empirical basis for choosing different weights.

A helpful analogy: PSI is like a teacher’s rubric that checks multiple things at once—not just whether the final answer is correct, but whether the explanation builds the right foundation, step-by-step, in the right sequence.

The six PSI sub-scores (and what they’re trying to catch)

Below are the six components described in the paper:

• Knowledge Distance (SK_D): Is the response aiming at the learner’s “just-right” zone (their zone of proximal development)? If the response is too easy or too advanced relative to the student’s known concepts, PSI penalizes it.

• Temporal Violation (ST_V): Does the response introduce concepts before their prerequisites? The paper builds an explicit prerequisite graph and checks whether prerequisite ordering is respected.

• Scaffolding Density (SS_D): Does the response provide instruction-like support (worked examples, step-by-step guidance, hints, comprehension checks, analogies)? This is one of PSI’s most important and most responsive dimensions in their experiments.

• Ebbinghaus Forgetting (SE_F): Does the response account for retention timing of referenced concepts? In their implementation, this depends on scenario metadata rather than what the response content actually mentions.

• Cognitive Load (SC_L): Does the tutor add avoidable fluff, hedging, filler, or tangential material? PSI penalizes extraneous content.

• Bloom’s Alignment (SR_A): Is the cognitive level of the response aligned with the intended learning target (e.g., remember vs. analyze)? The paper uses proxy detection (keyword-based) rather than deep semantic classification.

Then the composite PSI score is basically the average of these six pieces after normalization. Importantly: PSI is treated as an operationalized, theory-informed benchmark, not a fully validated psychological measurement instrument.

How the paper scores responses across different kinds of student prompts

To test tutoring behavior under realistic noise, the authors used paired scenario prompts:
- Standard prompts, with complete student context.
- Defective prompts, designed to mimic common student communication failures.

They simulate eight defect categories, including:
- Too Short, Keyword Only
- Missing Context, Unclear Goal
- Code Dump, Vague Error
- Wrong Terminology, Multi-Issue

This matters because tutoring isn’t just about understanding a well-formed question—it’s also about what the model does when the learner’s prompt is incomplete or messy.

The Benchmark Setup: One Course, 30 Scenarios, 240 Evaluations

The experiments are anchored in CSCI 150 (Intro to Computer Science) at the University of Montana, a 15-week Python-based course that starts from basic computing concepts and steadily ramps up toward more abstract topics.

Curriculum progression used to create the testbed

The curriculum is structured roughly like this:
- Weeks 1–3: variables, hardware concepts
- Weeks 4–9: operators, conditionals, loops
- Weeks 10–15: functions, data structures, NLP, image processing

The authors explicitly use this progression so PSI can check concept ordering and timing rather than judging in a vacuum.

Scenario design: learner profiles, prerequisite maps, and realistic misconceptions

They created 30 structured tutoring scenarios (two per week), spanning Bloom levels 1–4:
- Remember (n=1)
- Understand (n=7)
- Apply (n=16)
- Analyze (n=6)

Each scenario includes:
- explicit learner profiles (what concepts the student knows)
- a prerequisite map (what must come first)
- a realistic student question reflecting common misconceptions

Two undergraduate students rated scenario realism and course fit:
- Realism mean 3.67 (SD 0.69)
- Course fit mean 4.10 (SD 0.60)
- inter-rater agreement within ±1 point for 83.3% of judgments

Only one scenario (S14) had realism below 2.5 on average, which suggests the benchmark broadly feels like real tutoring.

The four LLM tutors compared under the same pipeline

They evaluated four models (commercial + open-weight):
- ChatGPT (GPT-4.5)
- Gemini (Gemini 3)
- Gemma 4 (open-weight 27B)
- Qwen 3 (open-weight 235B-A2)

All models were prompted identically per scenario, including student context and learning objectives. Total evaluation count:
- 30 scenarios × 2 prompt types × 4 models = 240 responses

How LLM Tutors Compared: Baseline PSI Differences Were Modest, Not a Simple “Open vs Closed” Story

A common instinct is to rank models by general “quality” and assume open-weight vs closed will separate cleanly. This paper’s results push against that.

Baseline PSI results across models

Overall PSI scores ranged from about 0.557 to 0.638—a spread of 0.081, described as modest. Here’s the ranking they report in the tested setup:

Key point: the paper explicitly says open-weight and closed models did not show a clear separation in pedagogical fit. In fact, open-weight Qwen 3 outperformed closed Gemini.

Where the differences came from: scaffolding

The authors report that scaffolding density (SS_D) is the main discriminator across models.

They give an especially striking contrast:
- ChatGPT scaffolding density: 0.492
- Gemini scaffolding density: 0.087

That’s described as over 5× more scaffolding by the best-scaffolding model in their tested comparison.

Meanwhile, two components weren’t very helpful for discrimination:
- Forgetting (SE_F): identical across models because it depends on scenario metadata, not response text.
- Cognitive load (SC_L): near-saturated (≥ 0.996), so it doesn’t vary much in a useful way.

PSI declines as learning goals get harder

They also show PSI drops as the target Bloom level increases—across all models.

Cross-model mean PSI:
- around 0.60 at Remember
- around 0.56 at Analyze

This makes intuitive sense: as the target becomes more complex, maintaining prerequisite coherence and appropriate scaffolding becomes harder.

Prompt Robustness and PSI-Guided Improvement: The Real Win Is Measurable Repair

The most interesting part, honestly, is not the baseline comparison—it’s what PSI enables afterward.

PSI under defective prompts: stable overall, with sub-score trade-offs

When they switched from standard prompts to defective prompts, they found overall PSI didn’t change much:
- Δ ≈ -0.002 (essentially stable)

But sub-scores shifted, suggesting different “compensations” by models.

The paper highlights a specific trade-off pattern:
- Temporal ordering (ST_V) improved by about +0.059
- Knowledge distance (SK_D) decreased by about -0.041
- Scaffolding density (SS_D) decreased by about -0.043

So when the input is noisier, models may “default” to stronger prerequisite ordering, but struggle to calibrate learner-specific knowledge and provide the same level of scaffolding.

PSI-guided regeneration: fixing weak cases with targeted feedback

They then choose 62 weak-performing cases based primarily on low PSI under defective prompts. Regeneration is done using a special prompt that includes:
- original student context + question
- the original model response
- the PSI score and all six sub-scores
- a structured diagnosis of weaknesses
- an explicit checklist of improvements

This is the part that’s easiest to see as “tuition” for the model: the prompt becomes a teacher-like feedback plan, not just “try again.”

How many weak cases improved?

Results are strong:
- 51 of 62 cases improved
- improvement rate: 82.3%
- mean PSI gain: +0.049

Every model improved on average, though sample counts differed:
- ChatGPT: n=4 weak cases
- Gemini: n=23
- Gemma 4: n=22
- Qwen 3: n=13

This unequal sampling limits deep cross-model statistical claims, but it still supports the core conclusion: PSI-guided feedback reliably improves many weak responses.

What changed the most after PSI feedback? Scaffolding again

Looking at sub-score shifts after regeneration, the biggest driver was:
- Scaffolding density (SS_D) increased from 0.073 to 0.345
→ a gain of +0.272 (the largest single change in the study)

Other sub-scores improved too:
- Knowledge distance (SK_D) improved by about +0.039

But there were also trade-offs:
- Temporal violations (ST_V) declined by about -0.034
- Bloom alignment (SR_A) declined by about -0.055

This is a really important practical insight: if your improvement instructions focus heavily on scaffolding, the model may add lots of instructional content, but it can accidentally:
- disturb prerequisite ordering, and/or
- shift the cognitive level of the response

The authors interpret this as evidence that single-round feedback is most effective for concrete, instruction-like dimensions (like adding worked examples and checks), while dimensions requiring deeper instructional judgment may need multi-round balancing.

Do the Improvements Actually Look Better to Humans? A Focused Manual Check

Automation is helpful, but you still need to ask: do teachers agree this is improvement?

The authors ran a focused manual evaluation on the same 62 PSI-selected weak cases. An instructor familiar with the course materials reviewed:
- the Round 1 response
- the regenerated Round 2 response

They checked two criteria:
1. Meaningful Improvement: did the response become clearly better instructionally (not just higher PSI)?
2. Weaknesses Resolved: did the regenerated response fix the specific issues PSI identified?

Results:
- Weaknesses resolved: 82.3%, matching the automated improvement rate 51/62
- Meaningful improvement: 74.2%

The drop between these two metrics suggests that some PSI gains don’t always translate into visibly better teaching—especially when PSI is computed with proxy measures (like keyword-based Bloom detection) or when the feedback introduced scaffolding but introduced new mismatches elsewhere.

Still, the fact that “weakness resolved” aligns with PSI improvement is a strong sign that PSI is surfacing instructionally real problems, not random artifacts.

Limitations You Should Keep in Mind (Especially If You’d Use PSI in Production)

The paper is pretty upfront about where PSI is strong and where it’s still rough.

• Proxy-based scoring: several sub-scores use keyword/regex detection rather than deep semantic analysis. That can misclassify Bloom level or scaffolding markers.

• Two components aren’t discriminative in this setup:

  • SE_F is identical across models because it uses scenario metadata.
  • SC_L is near-saturated, so it doesn’t help separate models.

• Single-course benchmark: everything is built on CSCI 150 concept taxonomy and 30 scenarios. We don’t yet know how PSI transfers to other subjects (history, math, medicine training, etc.) without reworking the concept/prerequisite graph.

• Manual evaluation scope: only PSI-selected weak cases (62) were checked manually, and done by a single expert rater. No inter-rater reliability estimate is provided.

• Single feedback round: regeneration uses one structured feedback pass. Given the sub-score trade-offs, multi-round optimization could be necessary for better balance.

Key Takeaways

• Correctness isn’t enough for AI tutoring: PSI targets instructional fit—learner readiness, prerequisite order, and pacing—not just whether the answer works.
• PSI is measurable and improvable: across 240 evaluations, models showed modest baseline PSI differences (0.557 to 0.638), and PSI-guided regeneration improved 51/62 weak cases (82.3%).
• Scaffolding is the biggest lever: the strongest differentiator across models and the largest improvement driver after feedback was scaffolding density (SS_D increased from 0.073 to 0.345 in regeneration).
• Open vs closed models didn’t cleanly separate: in this study, Qwen 3 (open-weight) beat Gemini (closed) in baseline pedagogical fit, so model category alone isn’t the main story.
• Trade-offs are real: adding scaffolding can sometimes disrupt prerequisite ordering or cognitive-level alignment, so better multi-step refinement may be needed.
• Human checks broadly agree with PSI: “weaknesses resolved” matched the automated PSI improvement rate (82.3%), while “meaningful improvement” was lower (74.2%), suggesting PSI is strong for diagnosis but still needs refinement for “teaching quality” in every case.
• Practical implication today: if you’re building or evaluating an LLM tutor, prioritize curriculum-aware evaluation (prerequisites + learning stage) and use structured feedback that explicitly requests worked examples, step-by-step scaffolds, and checks—because that’s where measurable wins happened.

If you want, I can also turn PSI into a simple “rubric-style” checklist you could apply when reviewing tutoring responses manually (or when designing your own automated evaluator).