Twin face recognition just got a reality check—new CTTS research

Twin face recognition just got a reality check—new CTTS research

Introduction

If you’ve ever looked at a pair of monozygotic (“identical”) twins and thought, “Wow, same DNA, same face,” you’re not wrong—but you’re also missing the punchline. New research from Zang, Wu, Sharma, and Bowyer (based on the original paper) shows that modern face recognition systems still struggle with twins in ways that are surprisingly specific—and that dataset design is a huge part of the story.

The study centers on something called the Celeb Twins Test Set (CTTS-80). It’s a large, carefully organized benchmark made from web-scraped celebrity twin images: 80 sets of twins, 21,120 image pairs total, evaluated like classic face verification tests (think LFW-style). What makes CTTS special is that it includes metadata for whether twins have distinguishing skin marks (like moles or scars) and whether some pairs look like mirror twins (left-handed vs right-handed symmetry).

And here’s the twist: even though more than half of the twins in CTTS have skin marks that should help humans, the researchers find that current deep CNN face matchers basically don’t use those marks (and similarly don’t exploit facial asymmetry/mirror effects in a meaningful way). That mismatch between what humans notice and what AI learns is where a lot of the “twin recognition” mystery comes from.

Why This Matters

Twin face recognition isn’t just a niche academic question anymore—it’s a practical concern for systems that rely on identity verification in the real world, like regulated logins, age-restricted services, or identity checks in systems where people can plausibly have an “almost identical” look-alike. Twins are one of the toughest edge cases because the background assumptions behind face embeddings—“faces differ enough naturally”—start to break down.

CTTS matters right now because it’s one of the first large twin datasets designed to let researchers test hypotheses about specific human-discriminative cues (skin marks and mirror-like asymmetry). This makes the benchmark more than a scoreboard. It becomes a diagnostic tool: what should the model learn, and what is it actually learning?

And it builds on earlier AI research in an interesting way. Prior work on face recognition for twins often focused on classical features or manual landmark approaches, and it frequently suggested skin marks could be key. But modern face recognition pipelines use deep embeddings trained on large face datasets—not twins-focused data. That means the model’s “attention” is shaped by training distribution and augmentation strategies, not by what humans naturally use. This paper basically asks: Are today’s popular architectures learning the right kinds of differences for twins—or just the easiest ones available in typical training? Their results strongly suggest the latter.

What the Researchers Actually Measured: CTTS-80’s scale and setup

To understand the paper’s conclusions, you need to see how they measured performance.

CTTS-80 is built like standard face verification benchmarks. For each twin set, the researchers select images, crop faces to a normalized size (112×112 pixels), and then form pairs labeled as:

• Same-person (“positive”) pairs: two images of the same twin
• Different-person (“negative”) pairs: one image from each twin

Dataset size: big enough to be useful, structured enough to be fair

CTTS-80 contains 80 twin sets. With 12 images selected per person, each twin set yields:
- 132 positive pairs
- 144 negative pairs

But for evaluation, they keep it balanced by randomly selecting 132 negative pairs. For folds:
- Each fold includes 8 twin sets
- Each fold therefore has 264 image pairs per twin set × 8 = 2,112 pairs
- With 10 folds, total pairs = 21,120

That’s where the paper’s “why this matters” claim becomes concrete: CTTS-80 is over 3.5× larger than several widely used face verification benchmarks (including LFW, CPLFW, CALFW, CFP-FP, AgeDB-30, Hadrian, Eclipse, and ND-Twins—each around ~6,000 pairs in the comparison).

Evaluation method: LFW-style cross-validation

The researchers use 10-fold cross-validation and report average accuracy across folds. Importantly, the folds are twin-set-disjoint—meaning the specific twin set being tested isn’t involved in choosing the threshold. That reduces “optimistic bias” that can happen if pair distributions leak across folds.

Performance baseline: modern matchers get decent accuracy—but not “solved”

They evaluate 4 face matchers—in 12 instances total—trained on combinations of common training sets. The headline result: current deep CNN matchers achieve over 76% accuracy on CTTS in classifying same-person vs different-person image pairs (CTTS positive/negative).

To be clear, this is not “perfect twins recognition.” But it’s also not random guessing: 50% would be what you’d expect if two twins looked literally identical to the model. So the model does encode some differences.

The crucial question becomes: which differences? That’s where the rest of the paper zeroes in.

The biology angle: why “identical” doesn’t mean “pixel-identical”

The paper starts with a biological reminder that’s easy to gloss over: monozygotic twins come from one embryo splitting into two, so DNA is “the same” at the start—but later life introduces differences.

Mirror twins: not just trivia, but a symmetry shift

A subset of MZ twins are mirror twins, where certain physical handedness traits are reversed (left vs right), such as:
- handedness (e.g., left-handed vs right-handed)
- hair whorl direction
- dental left-right patterns

Mirror twins are estimated at roughly 1 in 4 MZ twins. The paper also points out a subtle research reality: there’s no DNA test that cleanly tells you whether twins are mirror twins. So they use indirect labeling where possible—for CTTS they identify 7 twin sets where one is left-handed and the other right-handed.

Facial differences increase with age

Even for non-mirror twins, differences accumulate:
- skin marks: moles, freckles, scars
- environmental variation
- potentially even facial landmark differences as faces change over time

Age also affects NIST evaluation results previously—similar take: as twins get older, faces can become easier to distinguish.

So when you evaluate twin recognition, you’re evaluating a moving target: twins are “same starting point, different trajectories,” and you want a dataset that captures that.

Do modern matchers actually use skin marks?

This is the paper’s most concrete “does the model do what we think it does?” experiment.

CTTS-80 includes skin mark metadata: 43 out of 80 twin sets have skin marks visible in enough images to potentially distinguish the twins.

So the researchers ask:

If humans use visible skin marks, do deep CNN embeddings also reflect them?

Their test: erase the skin mark and see if embeddings change

They do two key experiments using an example set (Tamera and Tia Mowry):

1. Skin-mark-erased vs original matching (embedding impact test)
   They edit only the skin-mark area in one twin’s images and then measure whether distance distributions shift.

2. Original twin vs edited same-twin pairing (identity consistency test)
   They match original images to skin-mark-erased versions of the same person. If embeddings rely heavily on skin marks, then distances should become noticeably smaller/larger in ways that separate positive/negative pairs.

Result: almost no effect

The distance distributions don’t meaningfully change when skin marks are erased. When they directly compare:
- positive pairs: both images either have the skin mark or both don’t
- negative pairs: one image has the mark, the other doesn’t

…the paper finds that the positive/negative distances nearly fully overlap, except for a small number of negative pairs clustered near zero distance.

They also observe this pattern across additional CTTS examples (like Hassan twins and Kaczynski twins). The conclusion is direct: the presence of the visible skin mark has almost no effect on the embedding for these matchers.

Why would skin marks “not matter” to a deep embedding?

It’s tempting to think the model just “can’t see” marks—but the issue looks more like training behavior.

The paper suggests multiple reasons, and they’re pretty intuitive once you think about augmentation:

1) Twins are rare in training data

Face recognition training sets contain hundreds of thousands or millions of identities, but MZ twins will be a tiny fraction of identities. So the model doesn’t repeatedly learn “this exact kind of within-pair difference matters.”

2) Horizontal flipping can scramble positional cues

Many training pipelines use horizontal flip augmentation. If a discriminative skin mark appears on the left for one twin in real life, flipping means it appears on the right sometimes during training. The model may learn that positional skin-mark evidence is unreliable, so it effectively downweights it.

3) Test-time embedding averaging may cancel positional signals

Some evaluation protocols average embeddings from the original and horizontally flipped images. If positional cues were important, this could blur them back into “noise.”

Their “asymmetry-aware” retraining doesn’t fix it (much)

To probe this, they retrain an ArcFace ResNet-100 variant with horizontal flip augmentation turned off, and compute accuracy using only the original-image embedding (an “asymmetry-aware” setup).

CTTS accuracy with the flip removed is:
- 78.58% ± 3.7

That’s not statistically significantly different from the original model’s CTTS accuracy. And distribution overlap for the example twin set doesn’t improve dramatically either.

Interestingly, when they look at the 7 known mirror twin sets, overlap changes sometimes increase, sometimes decrease, and for some pairs the overlap decreases meaningfully. The paper interprets this cautiously: maybe only some mirror twins have facial asymmetries that are actually visually usable, and most don’t—meaning it’s an underexplored area even for humans.

If you want the practical takeaway: simply “turn off flips and train” doesn’t automatically teach the model to use skin marks or mirror asymmetry. The model needs targeted learning signals, not just less augmentation.

Why this matters for the next frontier: could generative AI create twins for training?

The paper doesn’t stop at critique—they also test a speculative route: using generative AI to create images of imagined monozygotic twins to increase representation in face recognition training.

Their idea: if you can synthesize twin variation, you could teach embeddings

They frame a blunt “data coverage” question: how many twin identities would you need in training for a model to become better at twins?

They use a rule-of-thumb from earlier face recognition work targeting specific conditions (like sunglasses or facial hair): performance improves when ~20% of identities in training are targeted to the condition.

If you applied that to twins, they estimate:
- for WebFace12M / WebFace42M-scale training sets, you’d need tens of thousands to a couple hundred thousand twin sets
That’s far beyond what real twin data collection can realistically supply.

So they test whether generative AI can help by fabricating non-existent twins.

Their experiment: Grok vs ChatGPT vs Gemini

They prompt each model to generate a pair of “not real” monozygotic twins with variations in:
- hairstyle
- facial expression
- age
(and they use the same prompt pattern for each tool)

Then they evaluate by measuring distance distributions for:
- positive pairs (same imagined twin)
- negative pairs (different imagined twins)

Result: current tools don’t generate “real twins” variability correctly

The paper reports major issues:

• Grok: some images look implausible, and critically the distance distributions don’t reflect expected identity differences. Same-person distances start too high and positive/negative distributions are heavily overlapped—suggesting the generated twins are too identical, not “twin-different.”
• ChatGPT: images look more human-plausible and pose/age variation is somewhat better, but positive/negative distributions are still effectively fully overlapped.
• Gemini: it often refuses to generate images matching the prompt constraints, citing technical limitations around producing “perfect identical geometry” across variations.

The broader implication is sharp: generative AI tools don’t currently model the biologically realistic notion of “identical DNA but non-identical faces over time.” They may produce something closer to “two copies of the same face,” which defeats the purpose for training twin discrimination.

And that reveals a research gap: we don’t yet have a well-established characterization of what range and type of facial differences are typical for real MZ twins in a way that could be used as a recipe for synthesis.

The paper even ties back to earlier work: training could bias a generative model toward “perfect twins,” potentially worsening the problem rather than fixing it.

Where twin recognition goes from here: dataset design + learning signals

So what’s the path forward after CTTS and these experiments?

1) Stop treating twins as just another test category

With CTTS, the field can start investigating “twin-specific features” more methodically. For example, skin marks might be helpful, but deep embeddings aren’t picking them up automatically at 112×112. That suggests:
- either new model architectures that explicitly incorporate facial-mark cues
- or training strategies that increase the chance models see meaningful twin differences often enough

2) Improve the representation of twin variation

Some CTTS twin sets have well-separated positive/negative distance distributions; others don’t. The paper links low performance to things like:
- occlusions (caps, heavy hair covering)
- pose variation
- atypical expressions

This hints that part of “twin difficulty” is capture conditions. If researchers collect images that expose marks and asymmetries more consistently, separation might improve—suggesting a pathway for better data collection protocols, not only better models.

3) Revisit what “asymmetry” means for real faces

Mirror twins aren’t guaranteed to have visible facial asymmetry. So the goal shouldn’t be “always use asymmetry.” It should be: measure when asymmetry exists, quantify its visual effect, then teach models those cases.

CTTS is a starting point precisely because it includes metadata for skin marks and known mirror-twin indicators. The field needs more of that.

4) Use generative AI carefully—or not at all (yet) for this task

The paper’s generative AI results are a warning label: tools today likely generate too-identical twins or fail to produce consistent identity-preserving variation.

Still, the door isn’t shut. The paper’s more interesting challenge is:
- can we build generative models that learn realistic “twin variation distributions” from real data?
- could we generate images grounded to an existing person’s visual identity while simulating plausible twin differences?

That’s a research frontier, not a done deal.

For the original paper’s broader framing and experimental details, see again: https://arxiv.org/abs/2609.01141.

Key Takeaways

• CTTS-80 is a purpose-built twin benchmark: 80 celebrity twin sets, 21,120 image pairs, with metadata for skin marks and indicators of mirror twins (left/right-handed).
• Current deep CNN matchers get >76% accuracy on CTTS, meaning they detect differences between twins—but not necessarily the differences humans rely on.
• Skin marks barely affect deep embeddings in these matchers: when the skin-mark region is erased, positive/negative distance distributions change very little.
• Turning off horizontal flip augmentation (“asymmetry-aware” training) does not meaningfully improve CTTS accuracy, suggesting the learning problem isn’t solved by augmentation tweaks alone.
• Generative AI tools (Grok, ChatGPT, Gemini) don’t yet create synthetic twin images with the correct “not-quite-the-same” variability needed to separate positive and negative pairs; their outputs tend to produce twins that are too identical or fail constraints.
• Practical next steps for researchers: improve twin-specific training signals and data protocols (visibility, pose/occlusion control), and study mirror-twin facial asymmetry as an empirical phenomenon—not an assumption.