Pbench Eval

---
name: pbench-eval
description: Evaluates a model's ability to perform referring expression segmentation across five hierarchical levels of semantic complexity, from basic object recognition to fine-grained attribute binding, OCR-based disambiguation, spatial layout understanding, and relational interactions. It also stress-tests long-context generation and instance stability in crowded scenes with high object counts. Use when the user wants to benchmark on PBench, or asks about evaluating this task. Reports per-level performance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.27365
  bibtex_key: bevli2026falconperception
  confidence: high
---

# pbench-eval

> Falcon Perception — Bevli et al. (2026) (arXiv:2603.27365, 2026)

## What this evaluates

Evaluates a model's ability to perform referring expression segmentation across five hierarchical levels of semantic complexity, from basic object recognition to fine-grained attribute binding, OCR-based disambiguation, spatial layout understanding, and relational interactions. It also stress-tests long-context generation and instance stability in crowded scenes with high object counts.

## Datasets

- **PBench** — total 5400; splits: test (5400); repo https://github.com/tiiuae/Falcon-Perception

## Metrics

- `per-level performance` **(primary)** — range: [0, 1]
  - Compute Intersection over Union (IoU) between the predicted mask and ground-truth mask. A prediction is correct if IoU ≥ 0.5. Accuracy is calculated per complexity level (0–4) and as an overall average to produce a capability profile.

## Input / output format

**Input**: A single image paired with a natural language referring expression prompt (e.g., 'red car', 'Diet Coke bottle', 'person holding umbrella').

**Output**: A sequence of structured predictions per instance: coordinates, size, and segmentation mask, generated autoregressively via a chain-of-perception decoding interface.

## Scoring recipe

```python
def score(predictions, gold_masks, levels, threshold=0.5):
    correct = {l: 0 for l in range(5)}
    total = {l: 0 for l in range(5)}
    for pred, gt, lvl in zip(predictions, gold_masks, levels):
        iou = compute_iou(pred, gt)
        if iou >= threshold:
            correct[lvl] += 1
        total[lvl] += 1
    per_level_acc = {l: correct[l]/total[l] for l in range(5)}
    overall_acc = sum(correct.values()) / sum(total.values())
    return per_level_acc, overall_acc
```

## Common pitfalls

- Existing benchmarks conflate spatial ambiguity, complex prompts, and world knowledge into a single score, masking specific failure modes.
- Models may suffer from instance duplication, drift, or 'object not found' prior collapse when generating masks for crowded scenes (K > 150).
- Reporting only an aggregate score hides capability gaps; the benchmark requires per-level profiling to diagnose weaknesses (e.g., OCR vs. spatial grounding).

## Evidence (verbatim from paper)

> We therefore report per-level performance in addition to an overall average, yielding a capability profile that reveals where a model fails (e.g., collapsing at OCR or relations).

## Citation

```bibtex
@misc{bevli2026falconperception,
  title={Falcon Perception},
  author={Bevli et al. (2026)},
  year={2026},
  note={arXiv:2603.27365}
}
```

- arXiv: 2603.27365