Can Humans Spot Compositional AI Image Bugs? New Study

Can Humans Spot Compositional AI Image Bugs? New Study

AI image generators are getting scary-good—until the moment you ask for something with multiple entities, multiple attributes, and messy spatial relationships (like hands holding objects, people interacting closely, or tricky occlusions). That’s exactly the theme of new research from the original paper: when composition doesn’t add up, humans can often detect defects in the images, and those defects aren’t random. They’re systematic.

In this study, the researchers dug into how people find those compositional failures. They curated 651 reference images across categories like people, hands, objects, and scenes, then used carefully edited prompts (from ChatGPT output) to generate AI images with three different text-to-image (T2I) models. After that, 29 participants reviewed the generated images and labeled the defect types and where they showed up—collecting 18,906 total defect annotations. The result is a new dataset called CO-AID (COmpositional AI-generated Image Defect), designed to benchmark defect identification and even help improve generation quality.

What’s especially interesting: the paper doesn’t just ask “Are images bad?” It asks “Are humans consistently able to localize and categorize compositional defects?” And then it tests whether training a model on that human-labeled defect data can predict defects and guide repair.

Why This Research Is a Big Deal Right Now (and Where You’ll Feel It Today)

The timing is rough in a very practical way. AI-generated images are already being used for ads, concept art, marketing mockups, UI visuals, and content pipelines. But many of those real workflows need composition: a product in-hand, a character interacting with props, a scene with multiple objects, a “team photo” with correct placements and believable interactions.

Here’s the problem: compositional prompts are exactly where today’s best T2I models tend to slip. The failure isn’t just “low quality”—it can be plausible-looking but wrong in specific structural ways, like hands with the wrong number of fingers, objects partially teleporting through limbs, or perspective/occlusion relationships that don’t make sense. If you’re relying on AI images for production—even as rough drafts—those defects can become expensive to catch late.

This research matters because it turns human intuition into something measurable. Instead of only scoring faithfulness globally (like “does the prompt match the picture?”), it builds fine-grained local defect labels—the kind you need if you want automated QA, defect-aware editing, or repair tools that know where to look.

And it builds on previous work in a key way: earlier benchmarks often focused on global preference or overall compositional faithfulness. This paper pushes toward local, category-aware defect detection, specifically for compositional complexity. That’s a step toward making AI generation pipelines more robust, not just prettier.

What Counts as a “Compositional Defect” in AI-Generated Images?

Let’s ground the idea of “composition” in something concrete. The paper doesn’t treat composition as a vague art principle. Instead, it uses a workable definition: an image is “highly compositional” when the prompt and scene simultaneously involve:

• Multiple entities (e.g., person + hand + object)
• Multiple attributes (e.g., posture, appearance, state)
• Multi-dimensional spatial relations (positions, perspective, occlusion)
• Complex interactions (physical contact, binding between objects and body parts)

If the model gets those relationships wrong, you get compositional defects.

The analogy: “Spot the wrong LEGO piece”

Think of a complex LEGO build instruction. The model can produce a figure with nice colors and decent lighting—but if a brick is rotated, clipped in the wrong place, or the hand is connected to the wrong part, the whole build becomes suspect. Humans don’t just judge “quality”; they notice structural mismatch at specific regions.

In AI terms, those mismatches often cluster into predictable defect types:
- unnatural-looking visual styling (global)
- blur/missing detail (global)
- abnormal text (global)
- commonsense violations (global)
- and—crucially—local failures like hand structure, body/limb issues, face errors, hair/fur problems, and object anomalies.

The researchers designed their study to capture exactly that “spot the specific wrong piece” behavior.

How the Dataset Was Built: Prompts, Models, and 29 Human Annotators

A dataset like this has to start with the hard part: generating images that are truly compositional, not just “an object” or “a single attribute.” The paper builds CO-AID in a very deliberate way.

Step 1: 651 reference images with complex composition

They manually selected 651 reference images from two sources (Pixels and Unsplash), divided into:

• People: 166
• Hand: 161
• Object: 161 (further split into animal, machine, LEGO, food, grocery)
• Scene: 163 (split into modern, natural, abstract patterns/text)

This category split matters because hands and multi-object interactions are where compositional structure tends to break most visibly.

Step 2: ChatGPT-based prompt editing to emphasize composition

For each reference image, they first used ChatGPT to generate a textual description, then manually refined it so the prompt explicitly included:

1. Multiple entities + attributes (appearance, posture, state)
2. Spatial relations (relative positions, occlusion)
3. Interactions (contact, hand-object relationships, etc.)

The goal wasn’t just similarity to the image—it was to force the prompt to carry compositional constraints that models might fail to satisfy.

Step 3: Generate images with three T2I models

They ran the 651 prompts through three T2I models:

• Midjourney
• Imagen
• FLUX

These were chosen after a preliminary comparison using 80 prompts, where they selected the models producing higher-quality compositional outputs.

Step 4: Human study—global vs local defect labeling

The experiment design is pretty rigorous:

• Pilot study: 40 images with 15 participants
• Main study: 600 images with 29 participants
• Each participant spent about 10 hours over two weeks
• The 600 images were split into four batches to reduce fatigue
• They repeated 12 images to measure reliability
• Total collected annotations: 18,906
  • 6,626 global-level annotations
  • 12,280 local-level annotations

Participants reported defects in a flexible way:
- If they saw no defect → “No noticeable defect”
- If defects were too hard to localize or too numerous → global-level reasons
- If defects were few → they could click and mark local locations with defect categories

They used predefined global defect reasons (like unnatural style, blur/missing detail, abnormal text, commonsense violations) plus composition-heavy global categories such as:
- large number of abnormal entities (attribute binding failures)
- large number of anomalous inter-entity interactions
- large number of spatial relationship anomalies (illogical perspective/occlusion)

Local defect categories were grouped into:
- face
- hair/fur
- hand
- body
- object

Finally, they removed outliers based on engagement time, self-consistency on repeated images, and conformity to the majority. They excluded 6 participants and used data from 23 participants for analysis.

What the Humans Found: Defect Patterns Vary a Lot by Model and Category

One of the most revealing parts of the paper is how defect-free images don’t distribute evenly across categories—and how different models fail in different ways.

Defect-free outcomes: people and hands are especially hard

When they label images as defect-free (based on majority human reports), results show:

• Only 18 defect-free images for people and hands combined
• Versus 136 defect-free images for objects and scenes combined

So despite overall image quality trends in the field, humans still perceive that complex human structure and hand-related composition is where T2I models struggle hardest.

Model comparison: Imagen vs Midjourney vs FLUX

Here’s how the three models stacked up in defect-free generation and defect type tendencies (as summarized by the paper’s figures and descriptions):

That tradeoff is kind of the story of composition failures: you can optimize for visual plausibility or for specific constraint satisfaction, but models don’t always get both at once.

Can Training on CO-AID Actually Improve Defect Detection and Repair?

This paper doesn’t stop at “here’s a dataset.” It tests whether CO-AID is useful in practice.

Predicting defects: local maps from human annotations

For a proof-of-concept, they used the locations of human-annotated local defects to create “ground truth defect maps.” Then they fine-tuned a model:

• Base: TranSalNet (trained for attention saliency prediction)
• Fine-tuned on CO-AID defect map supervision
• Compared to GPT-Image-1 defect localization outputs

The key qualitative finding: the fine-tuned TranSalNet produced more human-like defect localization patterns than GPT-Image-1 in their examples.

Guiding repair: using predicted defect maps as “where to fix”

They also tested a defect-guided repair approach:

1. Feed the AI image and defect map into GPT-Image-1
2. Let the defect map act as spatial guidance for what needs correction

Their sample results showed that using predicted defect maps could lead to noticeable quality improvement, because the repair tool had a better sense of which region to treat as problematic.

This is a pretty meaningful step: it suggests CO-AID can help turn composition defects from something humans complain about into something systems can actively target.

And if you want to see the dataset itself, the authors provide it here: https://github.com/Future-IQA/CO-AID

Key Takeaways

Key Takeaways

• Compositional prompts break models in consistent ways—and humans can reliably detect those failures, including where they happen.
• The study collected CO-AID, built from 651 compositional prompts and 651 generated images across Midjourney, Imagen, and FLUX, labeled by 29 participants.
• People and hands are the hardest categories: only 18 defect-free images for people+hands combined, compared to 136 for objects+scenes.
• Different models fail differently:
  • Imagen does better on defect-free counts (nearly 70) but more often triggers global “CG/unrealistic” judgments.
  • Midjourney tends to produce more realistic images but shows more local detail defects (notably 72 images labeled local defect).
  • FLUX underperforms for people/hand/scene (best on object).
• Training with CO-AID can improve defect prediction and can support defect-guided repair, using predicted defect maps to target fixes.

If you’re building with T2I—whether you’re doing design review, automating QA, or building editing tools—this dataset is a useful stepping stone. It moves evaluation from “does the image match the prompt overall?” to “can we locate and categorize what’s wrong when composition gets tricky?” And that’s the kind of capability you’ll need for real-world production workflows.