Object-Aware Video Captions Without Retraining (ProCap)

Object-Aware Video Captions Without Retraining (ProCap)
ProCap fixes missing details in video captions—no fine-tuning.

Introduction

If you’ve ever used automated video captioning for search, accessibility, or safety, you’ve probably seen the same two failure modes over and over: captions that sound fluent but miss important objects, and captions that mention things that aren’t actually there. That’s not just an annoyance—on real workflows it can mislead users, confuse retrieval, or even trigger the wrong downstream decision.

New research from the ProCap paper tackles this directly. The core idea is a prominence-guided, iterative post-hoc rectification method: instead of retraining a heavy vision-language model, ProCap uses detected objects to rank what matters most and then refines the caption in multiple rounds until it’s more complete and less hallucinated. It’s designed to be model-agnostic—you don’t modify the underlying captioner’s parameters.

What makes ProCap different from earlier “training-free” fixes is how it corrects captions. Prior object-grounded approaches often do a single correction pass and treat every detected object as equally important. ProCap goes further by (1) computing an object’s “prominence” using spatial saliency, temporal persistence, and interaction dynamics, and (2) running an iterative prompt-driven loop that progressively injects missing high-prominence entities into the caption.

In experiments on MSVD and MSR-VTT, ProCap improved perceived completeness by up to 48% and reduced hallucination/inconsistency by up to 45% compared to a strong pretrained captioning baseline—without retraining and without reference captions.

Why This Matters

This kind of work is especially relevant right now because we’re all stuck in a trade-off: modern captioning models are getting more fluent thanks to large pretraining, but they remain generative—meaning they’re optimized to produce text that reads well, not text that is guaranteed to be fully faithful to what’s visually present.

That mismatch shows up in practical deployments. Imagine an accessibility tool describing what’s happening in a video for a user who can’t see it. If the model consistently omits a key object (say, a “bike” approaching a street scene), the caption becomes less useful even if it’s grammatically perfect. Or take video retrieval: if a system misses the “dog” even once, that video may never be found in a “dog” search.

ProCap matters because it attacks the structural weakness behind omissions: earlier rectification methods don’t know which detected objects to prioritize, and they don’t have a multi-round way to keep closing the semantic gaps. ProCap’s prominence-aware ranking is like giving the captioner a prioritized checklist—what should be mentioned first, and what’s likely to be central to the story?—then repeatedly re-checking the output against that checklist.

It also builds on the earlier training-free rectification line (the paper builds on ReCap) while addressing its key limitation: ReCap performs a correction that’s effectively single-shot and not narratively prioritized. ProCap’s contribution is to make rectification both iterative and importance-aware—which is exactly the combo you’d want when a caption generator is prone to omission.

A specific scenario where this could be used today: large-scale video libraries (news, sports, education) where you can’t afford to retrain models but you still need captions that support filtering, search, and accessibility. ProCap’s “plug-and-play” framing is designed for that: the caption backbone stays frozen; the rectification layer can be swapped in.

How ProCap Improves Captions Without Retraining

ProCap is built as a modular pipeline layered on top of an existing video captioning model (the paper reuses mPLUG as the underlying backbone, consistent with the earlier ReCap protocol). The pipeline has three major stages:

1. Frame-level rectification (ground captions against detected objects)
2. Prominence scoring to rank detected objects by importance
3. Iterative caption refinement to inject missing high-prominence objects over multiple rounds

The key design constraint is: no retraining of the backbone. ProCap is “post-hoc,” meaning it fixes outputs after the fact—exactly what you’d want in production when retraining is expensive or risky.

Stage 1: Start from rectified frame-level captions

The paper retains ReCap’s first stage: for selected key frames, a captioning module generates captions, and an object detector is used to ensure the text is grounded in what’s visually detected.

This helps reduce obvious errors: if the detector doesn’t see an object, the rectifier can guide the caption not to claim it. But the paper points out the core problem: this grounding doesn’t guarantee completeness, because the downstream summarization step can still omit semantically important objects—and it can’t prioritize which ones matter most.

Stage 2: Rank objects using “prominence” (appearance + persistence + dynamics)

Once the system has detections, it computes a prominence score for each object instance across the video. The intuition is simple even if the formula is more involved: an object is prominent if it’s (a) visually big enough to catch attention, (b) present long enough to be meaningful, and (c) not just sitting there—it has some relational/motion relevance.

Concretely, ProCap decomposes prominence into three normalized components:

• Appearance score (Ai): how much image area the object occupies (averaged over frames where it appears)
• Presence score (Pi): temporal persistence—fraction of frames where the object is present
• Dynamics score (Di): how the object’s spatial relationship changes with other objects over time (a proxy for interaction/relative motion)

Then it fuses them into a single prominence score, using a “probabilistic union-inspired” formulation (the paper interprets this in terms of inclusion–exclusion-like contributions and joint effects).

Here’s the practical takeaway: it prevents tiny-but-frequent background detections from overshadowing the big story elements. The paper’s example contrasts a person vs. a dining table: both might have low dynamics, but the person gets much higher prominence due to larger appearance ratio and longer persistence.

A quick analogy: ProCap turns detections into a “priority story list”

Think of the detected objects as raw notes from a camera. A normal captioner might turn those notes into a sentence in a way that accidentally drops the most important line. ProCap turns notes into a ranked story outline:

• Big/persistent/interactive objects move to the top
• Less relevant background objects sink down
• The system uses that outline to repeatedly check what’s missing from the current caption

That outline is what enables the next stage.

The Iterative Caption Refinement Loop (Where the Magic Actually Happens)

Many prior rectification approaches do something like: “Detect objects → correct once → output final caption.” ProCap instead runs a feedback loop. It treats captioning like a conversation with the video evidence rather than a single write-and-forget step.

Iteration 0: Generate a base caption from aggregated frame descriptions

After frame-level processing, ProCap aggregates the frame captions and uses an LLM to generate an initial short, fluent video caption. The prompt includes constraints like keeping it concise (often 10–12 words) and removing redundant or inconsistent statements.

This initial caption can still be incomplete—that’s expected. The point is to create a starting point that’s then improved.

Iteration r>0: Detect missing objects and re-write the caption

For each subsequent iteration, ProCap:

1. Performs a lexical grounding check: which prominent objects appear to be missing from the current caption?
2. Builds a new prompt that explicitly includes those missing objects.
3. Asks the LLM to update the caption, encouraging insertion of missing entities while maintaining coherence and brevity.

This continues for up to R rounds or until the system believes the semantic gap is closed. The paper explicitly aims to preserve already-correct content while selectively enriching what’s missing.

A concrete example from the paper

For a hospital video:

• Iteration 0: “A newborn baby is seen inside a hospital room.”
• Iteration 1 (+ mother): “A mother cradles her newborn baby inside a hospital room.”
• Iteration 2 (Final + dining table): “A mother cradles her newborn baby while sitting at a table in the hospital.”

Notice how it doesn’t just dump all detected objects. It incrementally adds entities that become relevant and contextually consistent.

Why iterative prompting beats single-pass correction

If you do a single correction pass, you only catch omissions once—and you have no structured mechanism to re-check after the caption changes. ProCap’s loop gives you that re-check ability.

In practical terms: as the caption gets enriched, the system can see what’s still absent relative to the ranked prominent objects. That’s how it tends to improve completeness without letting hallucination explode.

Prompt Design That Keeps Refinement Faithful

A big question with any LLM-based post-processing: won’t the model just invent stuff again, especially when you ask it to “improve” the caption?

ProCap addresses this with a structured prompt strategy that repeatedly enforces grounding constraints. The paper uses multiple prompt templates (short/medium/long variants plus an object insertion clause), but they share a common instruction schema.

The shared instruction schema includes four constraints, summarized in the paper as:

1. Identify non-redundant core events
2. Generate a compact, fluent summary (typically 1–2 sentences)
3. Include only objects present in the captions or the object list
4. Avoid hallucination or external inference

The intent is to use LLMs where they’re strong (rewriting, consolidation, language fluency) while keeping them on a leash with explicit object grounding rules.

And because ProCap is model-agnostic, this prompt layer can sit on top of different caption backbones without modifying their internal weights—one more reason it’s attractive for real deployments.

What the Experiments Showed (Automatic + Human + Qualitative)

ProCap is evaluated on two well-known video captioning benchmarks:

• MSR-VTT (10,000 clips; 10–30 seconds; standard split: 6,513 train / 497 val / 2,990 test)
• MSVD (1,970 clips; ~10 seconds; standard split: 1,200 train / 100 val / 670 test)

A key evaluation choice: instead of relying only on BLEU/METEOR/CIDEr (lexical overlap metrics that can miss faithfulness), the paper uses object-grounded metrics adapted from ReCap:

• Temporal Completeness: how many significant objects (filtered by temporal persistence threshold τ) are mentioned in the caption
• Temporal Inconsistency: the complementary omission-focused failure—proportion of significant objects that were present in the video but missing from the caption

Automatic metrics: improvements in object coverage and reduced inconsistency

The paper reports relative improvements against baselines for both short and long prompt variants (and also the final iterative variant). While the table isn’t reproduced in the text we have here, the key reported findings include:

• ProCap variants beat the unrectified base caption significantly on both datasets.
• The short variant on MSVD even slightly edges out mPLUG-2 in completeness (+0.43%) while reducing inconsistency (-2.75%).
• On remaining variant-baseline pairs, automatic coverage and inconsistency are broadly comparable to mPLUG-2, but they remain much better than the unrectified baseline.

The bigger story becomes clearer in human evaluation, where people can judge whether the caption actually includes the important parts.

Human study: 110 annotators rating completeness and inconsistency

The paper runs a human-centered survey:

• 110 participants, self-paced through a web interface
• 100 clips selected via stratified random sampling (pre-filtered to avoid degenerate cases)
• Each participant rates caption variants on a 5-point Likert scale:
  • Completeness (higher is better)
  • Inconsistency (lower is better)

They also took care to reduce bias: captions were shown under blinded randomized labels (“Caption 1–5”), with both video order and caption order randomized. The study followed ethical guidelines from IIT Kharagpur.

Reported headline numbers (the ones you actually care about)

Compared to the base and mPLUG-2 baselines, ProCap shows:

• Completeness gains by up to 48%
• Inconsistency reductions by up to 45%
• The paper notes the best human results come from the long variant, with improvements in completeness up to 64.61% over the base caption and 47.60% over mPLUG-2, while reducing inconsistency by 42.07% (vs base) and 45.10% (vs mPLUG-2).

Also important: the final iterative variant trades off a bit of peak object coverage for more conservative, cleaner output. In human terms, it still improves substantially:
- Completeness: +52.67% vs base, +36.90% vs mPLUG-2
- Inconsistency: -26.20% vs base, -30.07% vs mPLUG-2

That trade-off makes sense: the iterative loop can add missing prominent objects, but prompts can also enforce tighter brevity and coherence, which can lower the chance of “almost right but wrong” additions.

Qualitative comparisons: why LLMs hallucinate and ProCap stays grounded

The paper includes qualitative examples comparing ProCap against ChatGPT and Gemini. Their observations are consistent with what you’d expect:

• General LLMs often use learned world knowledge and associations.
• That can produce captions that are fluent but unsupported, introducing specific entities/actions that the video never shows.

Examples mentioned in the paper include:
- Gemini describing a documentary about Boko Haram and naming individuals not directly observable in the video.
- Another model attributing a specific player in a match when the footage only shows a generic match context.
- ChatGPT adding spectators and traditional attire when the visible content is mainly children playing outdoors.

ProCap’s refinement is designed to stay faithful because the object list and prominence ranking act as constraints on what can be injected into the caption.

A Simple Comparison: ProCap vs. Baselines (What’s Different, in Practice)

Here’s the practical comparison between the key approaches the paper builds around—focused on what readers can expect at deployment time.

If you’re deciding between these in a real pipeline, ProCap’s “no retraining + iterative prioritization” combo is the differentiator: it doesn’t just enforce correctness—it aims for completeness.

Key Takeaways

• ProCap improves video caption faithfulness without retraining. It’s a post-hoc, model-agnostic rectification layer built on top of a frozen caption backbone.
• It fixes a real weakness in earlier training-free rectifiers: single-pass correction that treats all detected objects as equally important.
• Prominence scoring is the prioritization engine. It ranks objects using appearance (spatial saliency), presence (temporal persistence), and dynamics (relational/motion cues).
• Iterative refinement matters. ProCap updates the caption over multiple rounds by explicitly identifying missing high-prominence objects.
• Human evaluation results are strong: up to 48% completeness improvement and up to 45% inconsistency/hallucination reduction compared to a strong baseline (mPLUG-2), with 110 participants rating 100 clips.
• Practical applications are immediate: accessibility tooling, video search/retrieval, and any workflow where missing or fake objects can mislead users.
• The approach complements modern LLM fluency. ProCap uses LLMs for rewriting, but grounds what gets inserted using detected objects and prominence-based prioritization—reducing the “sounds plausible” hallucination issue.

If you want to dig deeper, the code and pipeline for ProCap are public here: https://github.com/Debjyoti-Adhikary/ProCap