Cops Ref Eval

---
name: cops-ref-eval
description: Probes a model's ability to perform compositional visual reasoning and ground referring expressions in the presence of semantically similar distractors. It evaluates whether models can parse complex logical structures and distinguish fine-grained visual differences rather than relying on statistical biases. Use when the user wants to benchmark on Cops-Ref, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2003.00403
  bibtex_key: chen2020copsref
  confidence: high
---

# cops-ref-eval

> Cops-Ref: A new Dataset and Task on Compositional Referring Expression Comprehension — Chen et al. (2020) (arXiv:2003.00403, 2020)

## What this evaluates

Probes a model's ability to perform compositional visual reasoning and ground referring expressions in the presence of semantically similar distractors. It evaluates whether models can parse complex logical structures and distinguish fine-grained visual differences rather than relying on statistical biases.

## Datasets

- **Cops-Ref** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Calculated as the percentage of expressions for which the model correctly selects the target region from the candidate set.

## Input / output format

**Input**: A natural language referring expression and a set of candidate images containing the target and semantically similar distractors.

**Output**: A single selected image/region proposal from the candidate set.

## Scoring recipe

```python
correct = 0
for expr, candidates, gt_idx in dataset:
    scores = model.score(expr, candidates)
    pred_idx = argmax(scores)
    if pred_idx == gt_idx:
        correct += 1
accuracy = (correct / len(dataset)) * 100
```

## Common pitfalls

- Models often exploit dataset statistics (e.g., frequent object categories) rather than true compositional reasoning, especially in the 'WithoutDist' setting.
- Performance drops dramatically when distractors share the same category/attributes as the target, revealing over-reliance on object/attribute recognition over relational reasoning.
- Evaluation must be reported across specific distractor configurations (Full, DiffCat, Cat, Cat&attr, Cat&cat) as results are highly sensitive to the type of distractors present.

## Evidence (verbatim from paper)

> Table 3 reports the accuracy of all the baselines and the proposed MattNet-Mine. We consider it as a correct comprehension if the model successfully chooses the proposal pointed by the expression among all the proposals extracted from the similar image set.

## Citation

```bibtex
@misc{chen2020copsref,
  title={Cops-Ref: A new Dataset and Task on Compositional Referring Expression Comprehension},
  author={Chen et al. (2020)},
  year={2020},
  note={arXiv:2003.00403}
}
```

- arXiv: 2003.00403