Scientific writing is drifting faster after ChatGPT—here’s the evidence

Scientific writing is drifting faster after ChatGPT—here’s the evidence

Introduction

Scientific writing doesn’t sit still. Each new wave of papers borrows patterns from what came before—then quietly nudges the norms that the next generation will copy. With new tools like large language models (LLMs) now being used for drafting, revising, and polishing, the question is no longer just “Do AI tools leave traces?” It’s also: “Are those traces reshaping the way scientific prose evolves over time?”

New research from Hanna Hubarava and Benjamin V. Ineichen tackles exactly that. Instead of simply comparing word frequencies before vs. after LLMs became widely available, the authors ask whether the trajectories—the rate and direction of linguistic change—shifted around the public release of ChatGPT in November 2022. They analyze more than 600,000 scientific-preprint abstracts across multiple fields, looking at how vocabulary, punctuation, readability, and syntax patterns moved before and after the breakpoint.

What they find is striking: post-2022 changes aren’t limited to a handful of “obvious” AI-associated words. Across major corpora, the research reports sustained movement toward longer words, lower formula-based readability, and syntactic changes like reduced maximum dependency-tree depth and less variation in sentence-level structure. And importantly, the timing isn’t uniform—some features surge and then recede, while others keep accelerating.

Why This Matters

This matters right now because scientific abstracts are effectively the “front door” to research. If the style of that front door is changing, then everything downstream changes too: search relevance, how quickly readers infer claims, how systematically reviewers scan papers, and even how future models learn what “good scientific prose” looks like.

Here’s a concrete scenario you can picture immediately: imagine you’re running a large lab’s internal “paper-quality” pipeline—something like a style gatekeeper that checks readability, clarity, and structural consistency before drafts go out. After November 2022, the baseline for what counts as “standard” scientific abstract language might have shifted. If your system assumes stability in features like sentence depth, word length, or readability score ranges, it could start flagging perfectly normal post-2022 writing styles as “worse,” or worse, it could fail to notice genuinely confusing drafts because the norm has drifted.

This study also builds on earlier AI text research that often framed the problem like a detective story: detect whether a particular document looks machine-written. That’s useful, but it can miss the bigger picture. Hubarava and Ineichen don’t try to label individual papers. They look at population-level drift—a different question, but arguably more helpful for understanding how norms evolve when everyone has access to powerful writing support. Their emphasis on trajectories (not just marker frequencies) also helps resolve an annoying puzzle from previous work: some “AI-like” words spike and later drop, which makes single-time measurements misleading.

How the Researchers Tested “Drift” Instead of “Before vs. After”

The core move in this paper is methodological: don’t just ask whether a feature became more common after ChatGPT. Ask whether the slope of change changed after the intervention point.

The data: four big corpora, two text scopes

The authors analyze scientific preprints, because timestamps are available and the writing is at a compressed stage (abstracts and titles). They focus primarily on abstracts, with titles used as a secondary comparison.

They use:
- arXiv (January 2015 through April 2026), split into three non-overlapping domains:
- Artificial Intelligence: 403,247 records (cs.CL, cs.AI, cs.LG)
- Quantitative Biology: 38,424 records (all q-bio categories)
- Statistics: 115,255 records (all stat categories)
- medRxiv (June 2019 through April 2026): 82,011 records

Across domains, that’s 638,937 records total, with 309,950 dated through 2022 and 328,987 from 2023 onward. Importantly, the analysis is done separately within each corpus so that a huge domain can’t drown out others.

The intervention: November 2022 as an observational breakpoint

The paper treats November 2022—the public release of conversational LLMs like ChatGPT—as an observational breakpoint. This is not a randomized experiment, so the authors are careful: they cannot prove that LLM use caused the changes. But they can test whether linguistic trajectories look different after that point, and then compare across fields and topics.

What they measured: lexical, punctuation, readability, and syntax structure

They track features in four families for each abstract (and partially for titles):
1. Lexical markers (including “excess vocabulary” style signals and expressive verbs/adjectives like across, align, and discourse markers)
2. Punctuation patterns (notably — em-dashes, colons, semicolons)
3. Readability metrics, using standard “formula-based” scores like:
- Flesch Reading Ease (FRE)
- Flesch-Kincaid Grade Level (FKGL)
- Dale-Chall, SMOG, ARI, and Gunning Fog
4. Dependency-based syntax, using parse-tree properties (dependency depth, dependency-length stats, coordination patterns, entropy of dependency roles, plus directed dependency-label bigrams)

To avoid treating titles as “missing features,” titles with sparse feature sets are handled carefully (missing rather than treated as null effects).

The key model: interrupted time-series regression (ITS)

Their statistical engine is interrupted time-series regression. In plain terms: they fit a line to each feature’s monthly trajectory before November 2022, then check whether the line’s slope and level change after the breakpoint.

You can think of it like checking a cyclist’s speedometer:
- Before November 2022, you estimate the usual speed increase/decrease.
- After November 2022, you test whether the cyclist started speeding up, slowing down, or changing direction in a sustained way.

They also compute a standardized version of slope change so features can be compared despite different variances.

One useful nuance: because ITS is good at finding sustained changes, they also run a complementary “counterfactual excess” check for temporary departures (especially over 2023–2024). That helps avoid the trap where you see a spike in a marker but miss the fact it quickly reverses.

What Changed After 2022? The “Big Patterns” Across Fields

Here’s the headline: post-2022 drift shows up across multiple feature families and multiple scientific domains. Not everything moves the same way, but the direction of some shifts is remarkably consistent.

Lexical drift: more certain-sounding structure—and some markers surge then fade

Across all four corpora, the authors find that overall aggregate marker-word frequency has a positive slope change.

They report standardized slope changes (relative to each feature’s pre-2022 variability), including:
- medRxiv: positive change (from smaller baseline slope to larger)
- arXiv AI: one of the stronger cases, with consistent positive shifts

A standout example is the word across, which increases in slope in every corpus:
- Standardized changes include 6.10 in AI, 3.88 in quantitative biology, 3.10 in statistics, and 3.10 in medRxiv (all with corrected significance q < .001).

They also look at expressive verbs and category-level discourse-ish markers. For instance:
- align shows positive slope change across all four corpora (standardized changes including 3.14 in AI and 2.71 in medRxiv, again all with corrected significance).
- Not every “salient AI-signal word” keeps rising.

Some words behave like a wave: they climb sharply, then recede. The paper mentions cases like:
- delve rising and then recoding back downward—so single-point “before/after frequency” studies would miss or misinterpret the story.

They interpret this as a reminder that “marker words” can be unstable: editing habits, human adaptation, and evolving model prompting strategies can all reshape which words people choose.

Punctuation drift: em-dashes rise almost everywhere, but colons accelerate more specifically

Punctuation is where the paper shows the drift is real, but not uniform.

A strong consistent signal is em-dash (—) usage, with positive slope change in all four corpora:
- Largest standardized effect in medRxiv: 2.84 (and smaller but significant effects in arXiv domains in the range 1.45 to 2.35, all q < .001).

For other punctuation:
- colons show significant acceleration across all arXiv corpora
- semicolons vary more by domain

So the punctuation story is: some signals shift in a coordinated way, others don’t.

Readability drift: abstracts trend toward longer words and lower formula readability

This is one of the clearest cross-domain results.

Across all four abstract corpora, readability measures indicate a shift toward greater formula-based complexity after 2022—again with the strongest effects in arXiv.

For example, FKGL shows positive slope change (i.e., higher grade level over time):
- Standardized FKGL slope changes of 3.44 (AI), 2.79 (quantitative biology), 1.85 (statistics), and 0.66 (medRxiv) (all q < .001)

FRE, which goes the opposite direction (higher score = easier text), shows negative slope changes:
- Standardized FRE slope changes of about −3.50 (AI), −3.41 (quant. bio), −2.56 (stats), and −0.87 (medRxiv) (all q < .001)

The “why” inside the readability formulas: word length does most of the damage

They don’t stop at saying “readability got worse.” They decompose readability formulas into components.

The biggest driver is longer words, not longer sentences:
- For FKGL, syllables-per-word accounts for most of the slope change:
- 81.7% in AI
- 93.9% in quantitative biology
- 80.2% in statistics
- 53.9% in medRxiv
- Sentence length contributes a smaller share and often contributes in the opposite direction in medRxiv.

For ARI, characters-per-word explains the overwhelming portion of the absolute component change (roughly 70.1% to 99.2%, depending on corpus).

So: abstracts are drifting toward lexically heavier phrasing—more syllable weight, more word complexity—rather than simply packing more words into sentences.

Syntax drift: shallower parse trees and less depth variation

Now for the structure.

The dependency-based analysis finds that:
- maximum dependency-tree depth has a negative slope change in every corpus
- standardized changes include −2.13 (AI), −1.31 (quant. bio), −1.69 (stats), and −1.88 (medRxiv) (all q < .001)

They also see declines in:
- variation in dependency-tree depth
- dependency-role entropy (though smaller)

That’s important because it suggests a kind of structural regularization: not necessarily “more simple” in a reader-friendly sense, but less structurally diverse in how information gets packed into sentence relationships.

They also observe patterns in how determiners attach to prepositional-object heads: the frequency drops significantly in each corpus. Meanwhile, the ratio of certain kinds of noun-phrase modification increases (more compact pre-nominal structure relative to prepositional elaboration).

Rhetorical drift: not one universal shift toward “more confident” writing

A common fear (and a common narrative online) is that AI tools make everything sound more assertive. This paper complicates that.

They find:
- hedging declines in arXiv (all three domains) but increases in medRxiv
- certainty accelerates significantly in statistics and medRxiv, but not in AI or quantitative biology after correction
- emphasis markers change most strongly in medRxiv (standardized change 2.47, q < .001)

So, the drift isn’t a single rhetorical direction across science. It’s uneven.

Are These Changes Confined to LLM-Heavy Topics? Not Entirely.

The authors also ask whether the drift is just an echo of topic growth—like if AI papers exploded and dragged the corpus along with them.

They run topic-stratified analyses using embeddings and clustering (with all-MiniLM-L6-v2 and BERTopic). Topics aren’t perfectly aligned across corpora (since topic modeling differs by domain and run), but the approach lets them check whether the drift appears broadly within each corpus rather than only in one fast-growing pocket.

Their conclusion:
- The largest effects often show up in arXiv AI topics associated with deep learning and LLMs, but
- similar directional shifts appear across non-LLM topics and also in quantitative biology, statistics, and medicine
- medRxiv shows broad but generally weaker distribution of effects across topics

So the drift looks like a broader change in scientific-preprint abstract language, not only a localized “AI writing genre” effect.

Why Titles and Abstracts Don’t Always Match (And Why That Matters)

One subtle result: the paper explicitly warns that titles can’t stand in for abstracts.

When they compare scopes:
- agreement in direction of slope change ranges from ~51% in quantitative biology to ~74% in AI
- some features match across corpora (e.g., across, colon usage, maximum tree depth)
- but other features can even diverge in direction between titles and abstracts (they mention em-dash behavior as an example)

Readability also differs by scope—for instance, FRE can move in one direction in abstracts and another in titles (notably for medRxiv).

Practical implication: if you’re building automated checks or monitoring systems, don’t assume that “the drift in abstracts” automatically means “the drift in everything else.” Titles are constrained by length and purpose; abstracts are where nuance lives.

Also, the authors’ core point holds: you should monitor language drift as a multidimensional, evolving system, not as a fixed list of suspicious markers.

So… Did LLMs Cause This?

The honest answer is: the study shows post-2022 drift, not a causal LLM effect.

Why not causality? Because:
- November 2022 is an observational breakpoint, not a randomized treatment
- adoption of LLM-assisted writing likely ramped up gradually and unevenly
- many concurrent forces could affect writing (editorial practices, machine translation, broader style trends, research-community norm shifts)

The authors also note that they did placebo-style checks (moving intervention dates earlier) and that November 2022 produced the largest absolute slope change in most representative feature–corpus comparisons.

Still, the safest interpretation is:
- the language of scientific preprint abstracts began shifting in a way that coincides with widespread public LLM availability
- but the mechanism could include LLM drafting, editing, translation support, human imitation, or downstream influence on what future models learn

And that’s actually valuable, because it reframes the question away from “AI detection” toward “how writing norms evolve under tool pressure.”

Key Takeaways

• Scientific abstract language drift is measurable: after November 2022, multiple linguistic properties change in their rate and direction, not just in isolated frequencies.
• Longer words are a major driver of reduced readability: formula-based readability worsens mainly because syllables-per-word (FKGL) and characters-per-word (ARI) move upward.
• Syntax shifts toward shallower structure: maximum dependency-tree depth declines across all four corpora, along with reduced depth variation.
• Lexical and punctuation changes are real but uneven:
  • some markers accelerate persistently (e.g., across, align)
  • others rise and then recede (e.g., delve)
  • em-dash usage increases across all corpora; other punctuation varies.
• Rhetorical effects are not one-size-fits-all: hedging declines in arXiv but increases in medRxiv; certainty and emphasis shift differently by domain.
• Topic growth isn’t the only explanation: while arXiv AI/LLM-heavy topics show strong effects, similar directions appear elsewhere too.
• Titles and abstracts drift differently: you can’t reliably infer abstract changes from title changes alone.
• Practical implication for tooling: if you’re monitoring clarity/readability norms, update baselines—because “normal scientific writing” isn’t stable anymore.
• What this means for the future: language drift is multidimensional; focusing on a single “AI marker list” will miss structural and trajectory-level shifts.