Detailed Localized Captioning Eval

---
name: detailed-localized-captioning-eval
description: Evaluates a model's ability to generate detailed, region-specific descriptions for images and videos, ranging from keywords to multi-sentence captions. It probes fine-grained visual grounding, attribute recognition, and hallucination resistance by comparing generated text against reference captions or using attribute-level positive/negative judgments. Use when the user wants to benchmark on DLC-Bench, LVIS, PACO, Flickr30k Entities, Ref-L4, HC-STVG, VideoRefer-Bench-D, or asks about evaluating this task. Reports positive accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.16072
  bibtex_key: lian2025describeanything
  confidence: high
---

# detailed-localized-captioning-eval

> Describe Anything: Detailed Localized Image and Video Captioning — Lian et al. (2025) (arXiv:2504.16072, 2025)

## What this evaluates

Evaluates a model's ability to generate detailed, region-specific descriptions for images and videos, ranging from keywords to multi-sentence captions. It probes fine-grained visual grounding, attribute recognition, and hallucination resistance by comparing generated text against reference captions or using attribute-level positive/negative judgments.

## Datasets

- **DLC-Bench** — total ?; splits: test (-1)
- **LVIS** — total ?; splits: test (-1)
- **PACO** — total ?; splits: test (-1)
- **Flickr30k Entities** — total ?; splits: test (-1)
- **Ref-L4** — total ?; splits: test (-1)
- **HC-STVG** — total ?; splits: test (-1)
- **VideoRefer-Bench-D** — total ?; splits: test (-1)

## Metrics

- `positive accuracy` **(primary)** — range: [0, 1]
  - Fraction of test instances where the generated description correctly includes all specified positive attributes for the target region.
- `negative accuracy` — range: [0, 1]
  - Fraction of test instances where the generated description correctly excludes all specified negative (forbidden) attributes.
- `average accuracy` — range: [0, 1]
  - Arithmetic mean of positive accuracy and negative accuracy.
- `semantic IoU` — range: percent
  - Intersection over union of predicted and ground-truth semantic region masks.
- `semantic similarity` — range: percent
  - Cosine similarity between predicted and ground-truth semantic embeddings of the region.
- `BLEU@4` — range: other
  - N-gram precision up to 4-grams with brevity penalty, standard for machine translation and captioning.
- `METEOR` — range: other
  - Harmonic mean of unigram precision and recall, incorporating stemming and synonymy matching.
- `ROUGE-L` — range: other
  - F-measure based on the longest common subsequence between prediction and reference.
- `CIDEr` — range: other
  - TF-IDF weighted n-gram similarity that downweights common words and rewards consensus among references.
- `SPICE` — range: other
  - Semantic Proposition Image Caption Evaluation score based on scene graph matching.

## Input / output format

**Input**: Image or video frame(s) with a specified region (bounding box, mask, or focal crop) and a text prompt requesting a description at a certain granularity (keyword, phrase, or detailed).

**Output**: Text description of the specified region, ranging from a single keyword/phrase to multiple sentences.

## Scoring recipe

```python
def score_dlc_bench(predictions, gold):
    pos_correct = sum(1 for p, g in zip(predictions, gold) if matches_attributes(p, g.positive))
    neg_correct = sum(1 for p, g in zip(predictions, gold) if not contains_forbidden(p, g.negative))
    pos_acc = pos_correct / len(predictions)
    neg_acc = neg_correct / len(predictions)
    return {'positive_accuracy': pos_acc, 'negative_accuracy': neg_acc, 'average_accuracy': (pos_acc + neg_acc) / 2}

def score_captioning(predictions, gold):
    return {
        'BLEU@4': compute_bleu(predictions, gold, n=4),
        'METEOR': compute_meteor(predictions, gold),
        'ROUGE-L': compute_rouge(predictions, gold, 'rougeL'),
        'CIDEr': compute_cider(predictions, gold),
        'SPICE': compute_spice(predictions, gold)
    }
```

## Common pitfalls

- Reference captions often omit valid details, causing models to be unfairly penalized on hallucination detection sub-tasks when they generate correct but unmentioned information.
- Using only local crops without cross-attention to global context severely degrades region-specific accuracy by losing scene semantics.
- Mixing prompt granularities (keyword vs. detailed) during evaluation inflates or deflates scores; prompts must strictly match the benchmark's required output length.

## Evidence (verbatim from paper)

> In the PACO benchmark, a challenging benchmark that includes both full objects and parts in complex scenes and requires the model to decide whether the region is an object or a part, our method achieves 73.2% semantic IoU and 84.2% semantic similarity, outperforming the previous best by 23.2% and 8.5% respectively.

## Citation

```bibtex
@misc{lian2025describeanything,
  title={Describe Anything: Detailed Localized Image and Video Captioning},
  author={Lian et al. (2025)},
  year={2025},
  note={arXiv:2504.16072}
}
```

- arXiv: 2504.16072