SlopShape: Detecting AI Blog “Slop” by Structure, Not Words

SlopShape: Detecting AI Blog “Slop” by Structure, Not Words

If you’ve ever tried to spot AI-written content in the wild, you’ve probably run into the same frustrating reality: word-level detectors can be insanely good… until someone paraphrases the text. New research from the original paper (published as SlopShape) tackles that weakness head-on by asking a different question: instead of hunting for suspicious wording, can we detect the shape of the writing?

The core idea is pretty intuitive: AI-generated commercial blog posts don’t just use different words—they tend to follow a recognizable structure—how they announce their purpose, in what order they present points, how evidence shows up, how the voice sounds, and how they “close the loop.” And here’s the surprising part: the researchers show you can detect AI posts with essentially the same accuracy even after rewording (using the same model that generated the text). That makes this approach much more practical for the real world, where “humanizer” tools are cheap and widely used.

This blog post breaks down what SlopShape did, what it found (including the headline numbers), and why it matters for SEO today and Generative Engine Optimization (GEO) tomorrow—especially as search engines increasingly rely on LLMs when choosing what to surface.

Why This Matters for SEO, GEO, and “AI-Search” Reality Right Now

The timing couldn’t be more relevant. Search used to be mainly about ranking algorithms. Now it’s also about what LLMs do when they read the page. That changes the game from “optimize for keywords” to “optimize for the way models interpret and summarize content.”

So a company doesn’t just need a page that ranks—it needs a page that survives LLM evaluation. Meanwhile, the cost to generate a plausible blog post has dropped toward zero, and low-quality AI text has become a flood problem. The paper references the “slop” framing and notes that industry estimates suggest AI involvement is extremely common in new web content—especially in the commercial sector where blogs are directly tied to traffic and revenue.

Here’s a concrete scenario where this research could be applied today:
- You run a content intake pipeline for a platform or marketplace.
- You see “company blog posts” arriving at scale, and you suspect a chunk are AI-generated variants made to game AI answers.
- You want to reduce the distribution of low-value pages before they mess with user experience (and before they skew your retrieval or recommendation quality).

Word-level detectors are brittle here. The moment someone paraphrases, they lose signal. SlopShape’s structural approach is designed for that exact moment: it’s meant to detect the underlying template-like writing pattern that’s harder to rewrite without losing coherence.

Also, this work builds on a bigger research arc: earlier studies (like StoryScope, also discussed in the SlopShape paper) showed that AI fiction can be detected from narrative structure rather than style. SlopShape essentially asks: if this is true for fiction, can it transfer to commercial writing, where incentives and writing patterns differ? Their results suggest yes—and in a way that’s especially useful for GEO.

What the Researchers Actually Measured: The “Tidy, Self-Announcing” Blog Shape

SlopShape measures writing structure using an instrument built from an 11-dimension commercial template schema. Think of it like a checklist for blog posts—but not the kind you could fill in manually. It’s derived bottom-up from real company blogs, then used to score each post against structural options.

The 11 dimensions include things like:
- Purpose (what the post is trying to accomplish)
- Audience
- Structure & flow (how the post unfolds)
- Explanation (how concepts are clarified)
- Evidence (what counts as proof, citations, examples, etc.)
- Voices (the rhetorical stance / “who’s talking like what”)
- Actionability (how directly it turns into next steps)
- Commercial integration (how product/business ties appear)
- Timeliness (how it frames recency)
- Page format
- Writing style (kept separate so the model can focus on structure)

A key methodological detail: the paper explicitly strips writing-style features out of the main structural detection signal. That’s what lets them claim they’re detecting structure, not just “AI-ish prose.”

The dataset setup: human blogs vs AI mirrors at scale

The study compares:
- 2,250 human blog posts from 268 company domains (snapshots captured pre-ChatGPT release, dated roughly 2008–2022)
- against 11,250 AI-generated mirror posts created from the same brief content

That’s 5 AI models generating a mirror for each human prompt—so the mirrors are paired by intent/brief:
- gpt-5.4
- claude-sonnet-4.6
- gemini-3-flash
- deepseek-v3.2
- kimi-k2.5

The mirrors are generated from inferred “content briefs” derived from the human post (anonymized to avoid publisher identity leaking into the brief).

The paper also mentions multiple safeguards:
- a memorization check based on 13-gram overlap (flagged only 0.19% of human/mirror pairs),
- filters for suspicious future entities (only 0.08% of posts had mentions that “shouldn’t exist” yet),
- and a strict split by company domain so one company can’t appear across train/test.

How SlopShape Finds Structure: A 214-Feature Instrument Scored by an LLM

SlopShape doesn’t start from a handful of hand-crafted rules. It reverse-engineers features the way researchers in this area often do: discover structural divergences between human and AI versions, then turn those divergences into measurable options.

From schema to features: templates, divergence notes, and a quality gate

Their pipeline (based on the same methodology lineage as StoryScope) has multiple stages, but the most important parts are:

1. Template extraction
   For each post, the system produces a JSON-like template describing values across the 11 dimensions.

2. Cross-source comparisons (feature discovery)
   They take a set of held-out prompts (100 prompts / 600 posts), show the templates side-by-side (human vs five AI mirrors), and ask an LLM to note where sources diverge.

3. Turn divergence notes into candidate features
   They generate 457 candidate features, then apply a quality gate that demands:

   • answerable from text alone,
   • locatable evidence,
   • self-explanatory option wording,
   • and no “compound” constructions.

4. Deduplicate and filter unreliable features
   Candidate features are clustered to remove duplicates, then a reliability filter removes features that are degenerate or unstable.

That pipeline lands at a final instrument of:
- 214 total features
- 187 structural features
- 27 style features

To be clear: the main results come from structural features alone, not style.

Reliability and validation: human agreement matters

Because these measurements are LLM-generated, the paper validates the scoring with a “gold annotation session.”

They report:
- human-human kappa: 0.928
- human-model kappa: 0.946

Both are above the paper’s pre-set acceptance threshold of 0.60. That’s a strong signal that the instrument isn’t just matching some LLM’s quirks—it tracks what careful humans agree on.

The Detection Results That Matter: 98.0% Macro-F1 from Structure Alone

Now to the headline numbers. The paper evaluates models using macro-F1 (balanced across human and AI classes). They emphasize macro-F1 because the dataset is imbalanced in a way where naive “always AI” accuracy would look deceptively high.

Word-level vs structural: structure gets close, even when words get swapped

Here’s the key comparison described in the results section. (All evaluated on held-out companies.)

The paper also reports a very telling detail:
- word-level detectors are near perfect on unedited text (stylometry and related baselines can hit essentially 100),
- but structural detection is doing something different: it remains robust under rewriting (next section).

Attribution: structural signatures can even identify which AI model made it

They also train a model to identify source among six possibilities (human + five AI models). Performance is measured with macro-F1 and accuracy.

Reported result:
- 79.3% of posts attributed to the correct source
- against a chance rate of 16.7%

The paper notes human posts are identified almost perfectly, and the main attribution errors are AI posts being attributed to the wrong AI model.

This is important: if the structural signature isn’t just “AI vs human” but also “AI model vs AI model,” then it suggests a kind of shared underlying production pattern that varies subtly across systems.

Rewording Attack Test: Structural Detection Stays Put (While Style Falls Apart)

This is where SlopShape really earns its credibility.

The paper tests robustness by taking every AI post in the test split and rewording it using the same model that generated the original. They follow an editing protocol from prior work (span-by-span editing targeting AI writing artifacts), while trying to preserve claims and links.

They also quantify that the attack is substantial:
- on average 73% of 13-word sequences no longer appear verbatim,
- and 15.6% of measured structural feature answers change for a post.

So: words change a lot, and some structural features change too. Yet the structural classifier’s performance barely budges.

The rewording robustness results

In other words:
- structural detection is essentially unchanged (98.0 → 98.1),
- while style detection drops as expected under a wording attack.

The paper also notes that only 5 of 1,450 reworded AI posts were misclassified as human-written.

This is a big deal because it targets the most common failure mode of “AI detectors”: paraphrase / humanizer tools. Structural patterns are harder to rewrite while keeping the same meaning and still passing coherence tests.

What “Tidy, Self-Announcing” Means in Practice (And Why Humans Don’t Write Like That)

SlopShape doesn’t just say “AI has structure.” It describes what the structure looks like.

Using feature importance analysis (SHAP), they find the most predictive features often correspond to an “argument-structure” sequence:
- promise payoff in the title
e.g., “How to Cut Onboarding Time in Half.”
- state the thesis and announce flow before the first section
e.g., “In this post, we’ll cover why onboarding stalls, three fixes…”
- close with a restatement / summary
e.g., “In short, structured onboarding saves time.”

The researchers call this pattern the “tidy, self-announcing blog post.”

Rare configurations: humans spread out, AI clusters up

They also measure rarity in structural feature space: posts whose structure few neighbors share score as “rare.” Their finding:
- human posts occupy rarer structural configurations,
- AI posts cluster into common structural regions.

They report an average rarity percentile gap:
- mean human rarity percentile 0.838
- mean AI rarity percentile 0.435
- with Cohen’s d = 1.83

And in the extreme tail:
- among posts above the 99th percentile rarity threshold, they find 149 human vs 4 AI (based on their test-only basis referenced in the paper).

So the story isn’t “AI always uses one structure.” It’s “AI tends to use a small number of highly reusable structures,” while human writing explores more variety.

Key Takeaways

• AI-generated commercial blog posts have a structural signature detectable from information order, evidence presentation, and voice—not just wording.
• SlopShape builds an LLM-scored 214-feature instrument (187 structural + 27 style) derived from an 11-dimension commercial template schema.
• Structural features alone detect AI vs human at 98.0 macro-F1 on held-out companies.
• Word-level detectors are near-perfect on unedited text, but structural detection stays basically unchanged under rewording attacks (98.0 → 98.1).
• The structural signal can also attribute which AI model wrote the post: 79.3% correct source (chance is 16.7%).
• The most common AI pattern is a “tidy, self-announcing blog post”: payoff promised early, thesis and flow announced before the first section, and a closing restatement.
• Humans occupy rarer structural configurations, while AI models cluster into a narrower set of “safe templates.”

If you want to pressure-test content quality systems, especially in a world where “AI text” gets reworded on purpose, this is the kind of approach that’s built for the next round of adversarial editing.

The researchers also release their pipeline and artifacts publicly (with post-level data gated under a non-commercial research agreement), following the disclosure and reproducibility approach described in the paper itself.