ED revisit screening with AI knowledge graphs to cut chart review

ED revisit screening with AI knowledge graphs to cut chart review
Emergency department bounceback reviews are a quality-improvement staple, but they’re also exhausting. Chart reviews take time, and “actionable findings” are often disappointingly rare—especially when teams only look at return visits within a tight window like 48–72 hours. That limitation can mean missing quality opportunities that happen just outside the cutoff.

New research from Handler et al. (arXiv:2609.10421) tackles this problem directly: how humans decide whether an ED revisit deserves deeper review, and whether an AI system can help screen more broadly without overwhelming reviewers. The study combines clinician-style judgment with GPT-4, then builds an algorithm using an LLM-informed knowledge graph to flag “potentially concerning” diagnosis pairs across a 1–14 day revisit window.

What makes this interesting isn’t just that AI is involved—it’s how it’s used. The paper doesn’t pretend LLMs can replace clinical reviewers right away. Instead, it explores where LLMs get it wrong (a lot, at least in this setup), what actually correlates with human concern, and how to turn that into a high-precision triage tool.

Why This Matters Now: Quality Review Is the Bottleneck, Not the Care

Quality programs in busy health systems are usually constrained by one thing: how much review capacity you have, not how much data you can collect. That’s why many programs default to short revisit windows—even though clinical failures don’t always show up neatly inside 48–72 hours. The research from Handler et al. is significant because it targets the bottleneck: choosing which cases to open in chart review when you can’t afford to look at everything.

Here’s a scenario where this could be applied today: imagine an ED quality team currently reviews a large number of bouncebacks that meet a strict criterion (say, 72 hours, then “admission next time”). Even then, they still face high workload and low yield. This research suggests a more efficient workflow could be possible by screening diagnosis pairs more intelligently—potentially catching concerning revisits around 6–7 days that would be excluded by a narrower window.

It also builds (and corrects) a trend in AI research. Prior work has used machine learning to predict bouncebacks or detect diagnostic error signals. LLM research has explored differential diagnosis generation and symptom-to-disease approaches. This study extends that theme by trying to cover three ideas at once using an LLM-derived structure: differential diagnosis fit, possible complications, and diagnostic “gravity.” The big “expert commentary” takeaway is: rather than relying on the LLM as an end-to-end judge, the authors use it to help create a constrained screening system—so the tool aims for fewer false alarms.

What the Researchers Actually Measured When “A Revisit Deserves Review” Is Unclear

The core challenge is that “quality” isn’t directly labeled in the data. So the paper created a proxy decision: given only the primary diagnosis at two ED visits, should the diagnosis pair trigger further assessment? That “should we look closer?” judgment is captured as a binary target variable.

The dataset: 99 diagnosis-pair revisit records inside 1–14 days

The study was exploratory and retrospective, using randomly selected ED visits from a multi-hospital US health system during 2022. Key inclusion constraints included:

• Adults 19–89 at the index visit
• Index disposition not among several non-reviewable categories (e.g., Admit, Transfer, Deceased)
• An ED revisit within 1–14 days to the same health system
• Revisit primary ICD-10 code having different first three characters from the index code
• Excluding certain diagnoses (e.g., not using Z53.21 “left without being seen” as primary diagnosis)

After filtering (and a small correction removed 1 case), the dataset included 99 diagnosis pairs.

Human raters didn’t see anything except diagnosis codes—and still had opinions

Two to three reviewers (board-certified emergency physicians and an urgent-care advanced practice provider) rated diagnosis pairs in a spreadsheet. Importantly: no visit narratives, vitals, labs, or imaging were provided—just the diagnoses.

They scored multiple dimensions:

• Diagnosis Membership Value (DMV): how “diagnostic” the primary diagnosis is (symptom/finding = lower)
• Medical Gravity Value (MGV): how bad the condition would be to the patient if they had a milder form vs a more severe form (0–100)
• Differential Inclusion Value (DIV): how much the revisit diagnosis belongs in the differential of the index diagnosis
• Complication Inclusion Value (CIV): how much the revisit diagnosis counts as a complication of the index diagnosis
• Target variable: whether further investigation seems warranted

In parallel, the team also queried GPT-4 for the same “target” decision.

One striking number: GPT-4 rated nearly all diagnosis pairs as warranting follow-up—94% (93 out of 99). Clinicians were much more selective: for example, among the three human raters, “yes” rates were 21.2%, 11.1%, and 7.1%. That mismatch matters, because it shows an LLM can easily become overly trigger-happy unless constrained.

Which Signals Actually Mattered to Clinicians (and Which Didn’t Hold Up)

A big part of the paper is figuring out what humans were really using, even though they were only looking at diagnoses. In the first modeling stage, the authors tried to predict the target variable using selected numeric features derived from the ratings.

Stage 1 (three human raters): revisit “gravity delta” and differential fit were the differentiators

The team tested predictors including DMV, DIV, the delta in “Typical” MGV between visits, and the days between visits.

Only two predictors were statistically significant in the adjusted multivariate model:

Meanwhile, the “days between visits” variable did not show significance (OR 0.96, p 0.606), and DMV was not significant.

Interpretation in plain English: clinicians seemed to be concerned when the diagnoses suggested either (1) a meaningful shift in expected severity, and/or (2) the revisit diagnosis still plausibly belongs in the first diagnosis’s differential—i.e., “this could be the same clinical story that wasn’t handled right.”

Stage 2 (two human raters): “complication thinking” helped, but only before adjustment

Later, the authors recognized that the revisit diagnosis might represent a complication rather than just a different differential possibility. Two raters assessed CIV (complication inclusion).

They created a composite called the Relationship Composite (RC):
- take the greater of DIV or CIV for each pair (a “bigger signal” approach)
- then test whether that composite predicts the target

In unadjusted analysis, RC was significant; but in adjusted analysis it wasn’t. The paper notes this likely reflects reduced power and exclusions when building the adjusted models.

The core takeaway: clinicians’ “should we look deeper?” decisions aren’t based on one simple rule. They look like a combination of diagnostic fit and expected severity—sometimes framed as differential mismatch, sometimes as complication plausibility.

GPT-4 vs Clinicians: Why the “Always Yes” Problem Matters

If you’re thinking “great, we can use GPT-4 directly,” the study offers a caution.

GPT-4 became an over-referral machine in this setup

In Stage 1, GPT-4 labeled 93.9% of pairs as warranting follow-up. Clinicians ranged from 7.1% to 21.2%. That implies extremely high false positive pressure—bad for workflow.

There’s also an important methodological clue: the authors report that prompt engineering was minimal and that they asked each visit pair as a separate prompt (not jointly). So GPT-4’s performance wasn’t treated as a finished product—more like a starting point showing the need for better structuring.

Light prompting can still miscalibrate clinical triage

This isn’t just a “GPT-4 is bad” headline. It’s a human-centered lesson: if a model is asked in an open-ended way to decide “should we review,” it may default to being cautious. In quality programs, caution becomes costly.

So the authors pivoted to building a different kind of AI support: not a free-form reviewer replacement, but a screening algorithm that’s optimized for high precision.

The Knowledge Graph Algorithm (KGA): Turning LLM Reasoning into a High-Precision Filter

This is the paper’s “most actionable” part: the authors developed an algorithm called the KGA (Knowledge Graph Algorithm) to automatically screen diagnosis pairs and flag only the most promising candidates for chart review.

How the KGA works: LLM-populated structure + gravity thresholds

The KGA depends on a knowledge graph populated by earlier work, built using open-source tooling (the paper cites Darth Vecdor as the platform). The knowledge graph includes diagnosis relationships deemed potentially concerning, and each diagnosis gets a Medical Gravity Index (MGI) in the graph.

Then, for new encounter pairs, the algorithm:
1. uses the graph to retrieve potentially concerning diagnosis-pair relationships
2. applies cutoffs based on revisit MGI (tested at thresholds like >=90 and >=100)
3. labels results:
- “positive” = warrants manual follow-up
- “negative” = doesn’t warrant manual follow-up

Performance: strong positive predictive value, but low sensitivity

In total, 28 out of 99 diagnosis pairs (28.3%) were flagged by at least one rater as “actual positives” in the dataset. The KGA’s positive predictive value (PPV) was high:

• At the lower MGI cutoff, 5/6 were true positives → 83% PPV
• At the higher cutoff, 4/4 were true positives → 100% PPV

Sensitivity wasn’t high: the paper reports the KGA’s sensitivity in the 14–18% range, which makes sense if you optimize for not overwhelming reviewers.

But here’s the workflow relevance: among the true positives, the KGA frequently identified revisits around 6–7 days that would be missed by a traditional 48–72-hour screening window. Specifically, it reports:
- At the lower cutoff: 4/5 (80%) of true positives were revisits at 6–7 days
- At the higher cutoff: 4/4 (100%) of true positives were revisits at 6–7 days

Plain-English analogy: Think of the ED review pipeline like a smoke alarm system. You don’t want it to ring for every smell (false positives), and you don’t need it to detect every possible smoke event (sensitivity). What you want is a reliable “smoke found” signal that’s actionable—especially for delays that happen after the usual window.

Where this beats the direct LLM approach

The direct GPT-4 approach produced huge numbers of “yes” outputs (94% positive). The KGA, by contrast, returns a smaller set with much higher PPV. That’s exactly what quality teams need: a way to expand review scope without exploding reviewer workload.

Practical Implications: What a Quality Team Could Do Next

The authors are clear: this is exploratory, with limitations like small sample size from a single health system and limited rater agreement. But the preliminary results still give a plausible near-term path.

1) Use knowledge-graph screening to expand beyond the 72-hour window

Because the KGA flagged many true positives at 6–7 days, it’s directly aligned with one of the biggest problems in bounceback review: missing clinically relevant failures outside the default window. If your team currently reviews only early revisits, this tool could extend that window while keeping PPV high.

2) Treat LLMs as infrastructure, not as final decision-makers

The paper’s “human + AI support” framing lands here: GPT-4 used alone wasn’t calibrated for triage. But LLM reasoning helped populate the structure that became a constrained algorithm. That’s a more reliable pattern for clinical workflows: use AI to build guardrails, then use guardrails to route decisions.

3) Validate with richer inputs and workflow endpoints

The paper notes future work should consider broader inputs (like disposition) and validate against additional ground truth. In quality settings, the most important “endpoint” isn’t just PPV—it’s whether flagged cases lead to meaningful changes (and how often reviewers have to open charts unnecessarily).

Key Takeaways

• Clinician “revisit review” decisions—made using only primary diagnosis codes—were most associated with:
  • changes in expected severity (MGV delta)
  • and how well the revisit diagnosis fits the initial differential (DIV)
• GPT-4 in this setup over-flagged: it rated 94% of diagnosis pairs as needing follow-up, far higher than human raters (about 7–21%).
• The study built a KGA (knowledge graph algorithm) that uses LLM-populated relationships plus gravity thresholds to screen diagnosis pairs.
• The KGA showed high precision:
  • 83% PPV at the lower cutoff (5/6 true positives)
  • 100% PPV at the higher cutoff (4/4 true positives)
• A key workflow win: many KGA true positives occurred at 6–7 days, meaning it could catch concerning revisits that typical 48–72-hour review windows miss.
• The results are preliminary (small sample, limited rater agreement), but they support a practical direction: use LLM-derived structures for high-precision screening, not unbounded “judge everything” outputs.

If you want, I can also rewrite this as a “what to implement in your ED QA program” checklist (inputs, thresholds, and a suggested pilot design) based strictly on what the preprint reports.