A Measurement-Error Checklist for AI-Assisted Literature Reviews

A Measurement-Error Checklist for AI-Assisted Literature Reviews

Introduction

If you’ve been using generative AI (especially LLMs) to speed up literature reviews, you’ve probably asked a simple question: “How trustworthy are the outputs really?” A new paper from Jeffrey D. Michler, Kieran Douglas, and Anna Josephson digs into this in a surprisingly principled way—by treating LLM-based literature review pipelines as a measurement system, not just a text generator. The research is based on new work published on arXiv.

The big idea is: when an LLM reads papers and extracts structured information (like “Does this paper use rainfall as an instrumental variable?” or “What endogenous variable is being instrumented?”), it’s producing measurements—and measurements can be wrong in specific, consequential ways. The authors don’t just measure “how often the model is accurate.” They investigate how different types of errors affect what researchers conclude about the literature, especially at the fine-grained level where LLMs are supposed to shine.

Why This Matters

This matters right now because AI-assisted reviews are becoming the default workflow in lots of labs and research teams—often with a quiet assumption that “good validation scores” automatically translate into “credible conclusions.” But this paper shows that’s not guaranteed. An LLM can look strong on standard metrics while still producing downstream results that shift depending on how the model was set up (prompting style, retrieval, fine-tuning).

A concrete scenario: imagine you’re on a team building a living database of econometric identification strategies for a meta-study—say, collecting every paper that uses rainfall instruments. You’d likely validate your extraction pipeline on a held-out set, see high precision/recall, and then confidently analyze patterns like “Which endogenous variables are most often instrumented?” or “Are rainfall instruments concentrated in certain mechanisms?” This paper warns that those “paper-level” patterns can vary a lot depending on the extraction system, even when the models all pass validation.

And it’s a meaningful build on earlier AI research that often focuses on extraction quality in isolation—metrics like accuracy and semantic similarity. What this work adds is the missing bridge: measurement error → estimand distortion. The authors explicitly treat LLM pipelines as different “measurement systems,” and then ask whether your substantive claims are robust when you swap the measurement system.

How the Paper Treats LLMs: Your Model Is a Measurement Device

The authors frame LLM-assisted literature review as a measurement problem. Think of your pipeline like a tool that takes an input (a paper PDF) and outputs numbers/labels (structured metadata) that you later analyze.

If your tool were perfect, it would always return the correct label. But the authors assume you don’t get that luxury. Instead, they classify errors in a way that mirrors the kinds of mistakes humans make in coding, just at scale:

• Type I error: reporting a concept that isn’t there
• Type II error: missing a concept that is there
• Semantic errors: detecting the concept is present but getting the meaning wrong (e.g., wrong variable name, wrong role in the model)

To connect this to inference, the paper distinguishes:
- Task-level measurement error (how the LLM performs at classification/extraction)
- Estimand distortion (how measurement error changes the conclusions you build from that data)
- Measurement-system dispersion (how sensitive your conclusions are to the specific pipeline you used)

In other words: the question isn’t only “Is the model right?” but “Does it matter which way it’s wrong?”

A test case that’s hard on purpose: rainfall as an IV

They apply this framework to economics papers that use rainfall as an instrumental variable. This turns out to be a great stress test because the “important information” isn’t always cleanly located in one place. Mentions of instruments and roles can show up in:
- equations
- tables
- footnotes
- different sections
- with inconsistent terminology
- and sometimes only implicitly defined

So the model must do more than keyword spotting—it must infer econometric roles from context.

The Three LLM Pipelines: Zero-Shot vs RAG vs Fine-Tuning

The study compares three implementations of ChatGPT-4.1 (via OpenAI’s API), all applied to the same document-processing workflow. They build structured outputs through a hierarchical prompting sequence (binary questions first, then “extract the name” only if the answer is yes).

Here’s the comparison:

The validation setup: human labels plus a staged evaluation

They assemble a candidate corpus of 4,191 economic papers (the paper text indicates 4,191/4,1914,191 due to formatting, but the core figure is 4,191 as the candidate size) and create a human-labeled reference sample of 439 papers. (Again, the formatting shows 439439, but the reference sample is explicitly described as 439 and used as the validation foundation.)

Across three LLM systems, they benchmark on a shared held-out evaluation set (a common model-labeled test subset) and then run each system on the full corpus to see how the extracted “rainfall-IV literature” differs.

A key implementation choice: they don’t just pick one “best” model and then stop. They push all three datasets through the same downstream analysis, so the comparison isn’t just about output similarity—it’s about whether the literature conclusions are stable.

What Standard Performance Metrics Miss: Accuracy Hides the Error Shape

One of the most useful parts of this paper is that it shows why typical model performance reporting can be misleading.

Binary classification looks great across systems

For binary tasks (e.g., “is the paper empirical?” “does it have an endogeneity problem?” “does it use an IV?” “does it use rainfall as an IV?”), all three implementations perform very well.

The paper reports that in their held-out evaluation sample of 888 papers (formatted inconsistently as 8888, but described as 8,888 evaluation items/records), each system misclassifies 12 or fewer papers for every binary query. That’s why accuracy/precision/f1 look strong.

But here’s the trap: high accuracy can come from very different mixes of false positives and false negatives.

Same accuracy, different mistake modes

The authors highlight a classic mismatch:
- RAG tends to produce more false positives
- Zero-shot tends to produce more false negatives
- SFT produces a more balanced error profile

So if you only report “accuracy,” you’re not learning whether the pipeline is:
- over-including papers (“false alarms”), or
- under-including them (“misses”)

And those choices matter when you build a literature-level dataset.

String extraction is where things get messy

When the task requires contextual interpretation—like extracting dependent variables, endogenous variables, instrument names, and rainfall instrument names—semantic performance drops and diverges between systems.

The study uses sentence-embedding similarity to score semantic alignment between extracted strings and human targets. They find:
- Titles and DOIs are recovered with high similarity (title ~0.97, DOI > 0.87)
- Context-dependent economic concepts have lower similarity and more variation
- SFT improves semantic similarity, with example ranges like:
- dependent-variable extraction: mean around 0.73 for SFT, versus 0.49 for zero-shot and 0.47 for RAG
- Even when means look close, distributions differ (e.g., RAG can have some high-quality outputs but also a meaningful chunk near zero)

So the deeper message is: correct detection ≠ correct extraction. The model might say “rainfall IV exists,” but still get the instrument or the economic role wrong.

Do These Differences Change What We Learn About the Rainfall-IV Literature?

Now we get to the “so what?” section.

The authors construct Sankey-style relationship maps from the extracted data. They focus first on whether systems agree on which papers use rainfall as an IV, then on how they disagree about which rainfall measures instrument which endogenous variables.

Paper inclusion: systems disagree a lot

In their common model-labeled corpus of 3,840 papers (formatted as 3,8403,840 but stated as 3,840), the number flagged as using rainfall IV differs:

• Zero-shot: 160 papers
• RAG: 168 papers
• SFT: 187 papers

They then compare overlap:
- All three agree on 137 papers (63%)
- Exactly two agree on 23 papers (11%)
- Only one system identifies the paper as rainfall-IV in 58 cases (27%)

They also report Jaccard overlap (set overlap measure):
- zero-shot vs RAG: 0.79
- zero-shot vs SFT: 0.70
- RAG vs SFT: 0.70

So: even after “high validation performance,” about a third of the papers identified by at least one system differ on rainfall-IV status.

Relationship-level disagreements persist too

Even when all systems agree a paper uses rainfall IV, they still disagree on what instrument is used and what endogenous variable is instrumented. After strict normalization and exact-string matching, they find:
- same endogenous string for 37 papers (27%)
- same rainfall-IV string for 24 papers (18%)
- same rainfall-IV/endogenous pair for 8 papers (6%)

That doesn’t mean “everything is wrong.” It means the fine-grained coding is measurement-system sensitive.

Why “coarser claims” are more stable

Interestingly, when the analysis zooms out—from paper-level relationships to broader descriptions of the literature—the disagreements become less consequential.

They argue this is estimand-dependent:
- Paper-level or relationship-level claims are sensitive to measurement system choice.
- Broad literature-level claims (like “rainfall instruments many socioeconomic processes”) are more stable.

This is one of the paper’s most important practical points: the value of LLM extraction depends on what you’re trying to prove.

Practical Implications: A Validation Rule That Actually Matches Your Claim

If you take one thing from this paper, make it this:

“Validation should match the resolution of the claim.”

They’re basically saying: model metrics tell you whether the system reproduces labels, but not whether residual errors distort the specific estimand you care about.

So if your claim is:
- “Which papers use rainfall IV?” → you need paper-level robustness
- “Which instruments and mechanisms show up most often?” → you need relationship-level robustness
- “Broadly, rainfall-IV studies raise concerns about exclusion restrictions?” → robustness is more likely (and LLMs may be less necessary)

In fact, the authors note an uncomfortable tradeoff:
- LLMs shine at fine-grained characterization
- but measurement error is most consequential at fine-grained resolution
- so you must explicitly test whether your fine-grained conclusions survive switching measurement systems

Key Takeaways

• LLMs in literature reviews behave like measurement devices. Errors aren’t just “hallucinations”—they propagate like measurement error into the conclusions you make.
• High binary classification accuracy isn’t enough. RAG, zero-shot, and SFT can all look accurate while producing different error patterns (false positives vs false negatives).
• Semantic extraction is where performance meaningfully diverges. Titles/DOIs are easy; dependent/endogenous/instrument extraction is substantially harder, with SFT usually best but still imperfect.
• Paper inclusion can change a lot by measurement system. In the rainfall-IV test case, systems disagreed on rainfall-IV status for ~27% of papers identified by at least one implementation.
• Fine-grained relationship claims are sensitive; broad claims are more stable. Disagreement “washes out” when you coarsen the estimand, but the places LLMs add value (detail) are also where measurement error matters most.
• Future workflow advice: don’t stop at reporting accuracy/precision. Run robustness checks where you swap measurement systems (different prompting/RAG/fine-tuning setups) and verify that your substantive estimand stays the same.

If you want, tell me what kind of literature review you’re doing (field, level of detail, and the structured fields you’re extracting), and I’ll suggest a practical validation/robustness checklist tailored to your use case.