ChatGPT vs Expert Reviewers: How Reliable Is PDF-Based Paper Scoring?

ChatGPT vs Expert Reviewers: How Reliable Is PDF-Based Paper Scoring?

Introduction

If you’ve ever wondered whether ChatGPT can replace expert judgment when evaluating journal articles, you’re not alone. The big promise is simple: you feed in a paper, it spits out an “overall quality” score—fast. The practical question is harder: does it actually match how reliable human expert reviewers are?

New research from Thelwall and Ali tackles exactly that. Based on new work published as a preprint in 2026, the study compares ChatGPT-5.4 scoring quality (using both PDF input and title/abstract input) against expert reviewer scores—and in one dataset, even compares ChatGPT against individual reviewers directly rather than just an averaged panel opinion. That last detail matters because “panel averages” can hide a lot of the messiness that shows up in real reviewing.

Why This Matters

Here’s the reason this study feels important right now: AI tooling is racing ahead, but universities and research managers are still stuck deciding what to trust. If you’re using AI outputs as part of a system—screening submissions, triaging quality, ranking papers, or guiding reviewer effort—you need to know whether AI is merely confident-sounding or whether it’s actually useful.

This research is the kind of evidence you want when you’re making decisions like:

• A department wants to rank a backlog of submitted manuscripts for internal triage before sending them to a full review panel.
• A journal editor wants a fast “first pass” to decide which papers are worth deeper human time.
• A research office is calibrating reviewer rubrics across fields and wants a consistent secondary signal.

And crucially, this paper doesn’t just ask “does ChatGPT correlate with experts?” It asks two more revealing questions:
1) Is ChatGPT as reliable as individual expert reviewers?
2) Does providing PDFs improve scoring accuracy compared to just titles and abstracts?

It also connects to prior AI evaluation work that found LLMs often give ratings that are too narrow (rarely very low) and don’t always capture the nuanced “is this limitation fatal?” judgment humans make. This new study adds a practical twist: even when ChatGPT talks more about PDFs, that doesn’t necessarily translate into better predicted quality scores.

What the Researchers Actually Measured (and Why It’s Not “Fair” but Still Useful)

The study is built around a pretty realistic (but ethically sensitive) data setup. In the UK, research quality is assessed via the Research Excellence Framework (REF), and universities run internal “mini-REF” reviews to decide what to submit nationally. Those internal scores are confidential—but they’re produced by field experts, which makes them a good proxy for something like “ground truth” quality ranking.

Because the national REF scores aren’t publicly available, the researchers used University of Sheffield internal departmental mini-REF scores:

• UoA3: Allied Health Professions, Dentistry, Nursing and Pharmacy

  • 98 journal articles
  • 2–5 individual reviewer scores for each article (from internal experts)
  • This is the only dataset where they could compare ChatGPT directly to individual reviewer scores for the same papers.

• UoA13: Architecture, Built Environment and Planning

  • 44 journal articles
  • departmental agreed scores only (not individual reviewer scores)

• UoA34: Communication, Cultural and Media Studies, Library and Information Management

  • 58 journal articles
  • departmental agreed scores only

Inputs to ChatGPT: Title/Abstract vs PDF (and what else was done)

The researchers downloaded each published article as a PDF and created an anonymous PDF by removing metadata (authors, journal name, affiliations, citations details, etc.). They also created:

• a title/abstract-only version
• a plain-text version (not used as the headline comparison in the main results, but part of the input processing approach)
• a full PDF input for the “deepest” test

Then they scored everything using ChatGPT-5.4 with structured prompts designed to mimic the UK internal scoring logic (including 1* to 4* style ratings, plus fractional scores).

What “reliability” means here

There’s no true objective quality score for any paper, so reliability had to be approximated using rank comparisons:

• For ranking, the paper uses Spearman correlation (and bootstrapped confidence intervals).
• When comparing ChatGPT to reviewers, they assume humans roughly approximate the “true” ranking, and then ask whether ChatGPT correlates with humans as strongly as humans correlate with each other.

That “human-human baseline” is especially important for the individual-reviewer comparison in UoA3.

How ChatGPT Compared to Individual Reviewers (UoA3): Close, and Possibly Better

The core question in UoA3 was:

RQ1: Is ChatGPT-5.4 just as good at ranking papers as individual expert reviewers?

They compared:
- human reviewer score ranks from two different individual reviewers (the first two reviewers, to avoid bias from later reviewers being called in only for borderline cases)
- ChatGPT-5.4 ranks computed from 30 score predictions per article (see method details below)

The “human vs human” baseline

In UoA3, the study reports that the Spearman correlation between the first-two reviewer scores is about half as strong as the correlation between reviewer scores and ChatGPT scores.

That’s a hint that—at least under this setup—ChatGPT might be closer to the “true” ranking than a single human reviewer set is.

Here’s the relevant comparison summary from their Table 2 (random reviewer sets are shown too):

What does “almost all statistically significant positive” mean here?

The researchers found that correlations between ChatGPT and expert rankings were mostly statistically significantly positive, but the differences between the correlations were often not statistically significant.

So the honest interpretation is:

• ChatGPT doesn’t dramatically outperform humans in a way that’s clearly proven statistically.
• But there is a weakly suggestive pattern that ChatGPT-5.4 could be more reliable than individual reviewers for UoA3.

Also notable: ChatGPT was extremely consistent with itself across input types—PDF vs full text didn’t change ranks much. That leads into the PDF question.

Does Feeding PDFs Improve the Scores? Surprisingly, Not Really

The study’s second research question was:

RQ2: Can ChatGPT predict article quality from PDFs better than from titles and abstracts?

They looked at correlations between:
- ChatGPT ranks and departmental agreed scores
- across input types: title/abstract vs PDF (and also full text in some analyses)
- across three UoAs

What they found

Broadly:
- In UoA3 and UoA34, most ChatGPT variants produced correlations that were statistically significantly positive with agreed scores.
- In UoA13, correlations were often not significantly different from 0, though the authors caution that the sample size is small (44 papers), making it hard to detect subtle effects.

The “input depth” result: more information didn’t consistently help

Here’s the key takeaway: there was no clear overall tendency that more input (title/abstract → full text → PDF) improved score prediction.

In fact, the direction sometimes flipped depending on the field:

• For UoA3, title/abstract correlations were higher than PDF/full-text.
• For UoA13, PDF/full-text did better than title/abstract.
• For UoA34, there was little difference.

So even though PDFs are where humans get the full story, ChatGPT-5.4 didn’t reliably become more accurate once it had PDFs.

Comparison table: does “more text” help?

Bigger models help a bit—but not by much

They also tested multiple ChatGPT-5.4 variants (“full, mini, nano”). The larger versions generally did better, but the differences were not large and usually not statistically significant. One notable point: many of the correlations for the largest model were statistically significant (with one exception).

So if you’re thinking “should I pay extra for a bigger model?”—the evidence here says maybe, but don’t expect miracles.

ChatGPT Sounds More “Detailed” on PDFs—But That Doesn’t Improve Scoring

This is where the study gets especially interesting (and a little alarming for anyone using AI-generated commentary for decision-making).

RQ3: Do ChatGPT-5.4 reports on PDFs differ from its reports on titles/abstracts?

To analyze report text differences, they used Word Association Thematic Analysis (WATA)—basically a statistically guided way to detect which words/themes appear more often between two sets of generated reports.

What differed?

The PDF-based reports:
- contained more detailed evaluations
- used more distinctive positive and negative evaluative terms
- mentioned full-text-like elements (including things like ethical approval)
- referenced sections/contents such as procedural detail, tables, and other manuscript-specific clues

Meanwhile, the title/abstract reports were:
- more hedged (“seems,” “apparent,” “appears”)
- more likely to include “insufficient information” style caveats
- generally less specific

So yes—ChatGPT’s PDF comments are more confident and more detailed.

But here’s the catch

Even though the PDF reports look more like “real reviewing,” the researchers found that this didn’t translate into improved score predictions.

That’s the “misleading” part the authors emphasize: a reader might think, “Wow, it is critiquing methods, limitations, recruitment bias—so the score must be better.” But the ranking accuracy didn’t improve in a consistent way.

Examples of how the tone changes

The paper provides example PDF-based evaluative statements like:

• “The methods section gives a broad description… but it lacks enough procedural detail to fully assess robustness.”
• “[ ] was not based on validated constructs… weakens measurement reliability…”
• “Survey recruitment… introduces selection bias.”

Title/abstract-based examples were more hedged or general:

• “… appears to be capable of informing…”
• “… appears to offer…”
• “This is a sound, useful, and well-conceived [study]”

Tables and figures: ChatGPT talks about them, but doesn’t seem to “read” them deeply

The study notes:
- “figure” mentions were far more common in PDF reports (99 individual reports) than title/abstract reports (23)
- “table” mentions were common in PDF reports (271) and rare in title/abstract reports (4)

But importantly, they report that ChatGPT typically evaluated captions and meta text, not the actual figure content. In many cases, the figure/table content wasn’t interpreted beyond what captions revealed—so the “PDF advantage” looks partially like structure-aware narration rather than deeper evidence extraction.

So… Should You Use ChatGPT Scores as a Reviewer Substitute?

This study’s bottom line is nuanced:

• ChatGPT can rank academic papers with a moderate ability, and it correlates positively with expert scores/ranks in multiple fields.
• Entering PDFs doesn’t reliably make the ranking more accurate, even though the generated commentary sounds more “expert.”
• In UoA3, there’s weak suggestion that ChatGPT-based scoring might be more reliable than individual reviewers, but differences are mostly not clearly significant.

Practical implications for today

If you’re building a workflow right now, a cautious but useful pattern emerges:

Good uses (lower risk):
- formative feedback (“here’s what might be missing”)
- calibration or discussion support for human reviewers
- triage signals when speed matters and errors are tolerable
- large-scale ranking where you care about ordering rather than exact “truth”

High-stakes uses (don’t automate fully):
- using scores alone for REF submissions/selection
- promotion, recruitment, and funding decisions without human oversight
- any situation where the difference between “world-leading” and “internationally excellent” materially changes outcomes

And if your managers are the type who trust “more detailed critique = better evaluation,” this paper is basically a warning label: don’t confuse detailed PDF commentary with better scoring accuracy. That’s not what the evidence supports here.

Key Takeaways

• ChatGPT-5.4 can rank published journal articles for research quality with generally positive correlation to expert assessments across multiple fields.
• In UoA3 (98 articles), ChatGPT’s rank alignment with expert scoring is closer than individual reviewer sets are to each other, with only weakly suggestive evidence that it may be more reliable than individual reviewers (differences often not statistically significant).
• PDF input doesn’t consistently improve score prediction over title/abstract input. For some fields title/abstract even performed slightly better.
• ChatGPT’s PDF reports are more detailed and more specific, with more positive/negative evaluative language—but that detail did not translate into better ranking accuracy.
• The study suggests you should treat PDF-based LLM commentary as plausible analysis, not necessarily genuinely deeper evaluation.
• Use AI as a support signal, not a replacement, especially for consequential decisions like selection for REF, hiring, or funding.

If you want, tell me your use case (journal triage, internal ranking, grant pre-screening, reviewer assistance, etc.) and I’ll suggest a workflow that fits the evidence—without over-trusting the model’s “confident-sounding” PDF critique.