Ref Avs Eval

---
name: ref-avs-eval
description: Evaluates a model's ability to segment objects in audio-visual videos based on natural language referring expressions. It tests both seen categories and generalization to unseen categories, as well as handling null references where no object exists. Use when the user wants to benchmark on Ref-AVS Dataset, or asks about evaluating this task. Reports Jaccard Index ($\mathcal{J}$).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.05018
  bibtex_key: song2026paperorchestra
  confidence: medium
---

# ref-avs-eval

> PaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing — Song et al. (2026) (arXiv:2604.05018, 2026)

## What this evaluates

Evaluates a model's ability to segment objects in audio-visual videos based on natural language referring expressions. It tests both seen categories and generalization to unseen categories, as well as handling null references where no object exists.

## Datasets

- **Ref-AVS Dataset** — total 4000; splits: train (2908), val (276), test (818)

## Metrics

- `Jaccard Index ($\mathcal{J}$)` **(primary)** — range: [0, 1]
  - Intersection over union of predicted and ground truth binary masks.
- `F-score ($\mathcal{F}$)` — range: [0, 1]
  - Harmonic mean of precision and recall computed on binary mask predictions.
- `S` — range: other
  - Square root of the ratio of predicted mask area to background area: $S = \sqrt{\text{predicted mask area/background area}}$. Lower values indicate better performance.

## Input / output format

**Input**: 10-second video frames, corresponding audio, and a natural language text expression referring to a target object.

**Output**: Pixel-level segmentation mask for the target object.

## Scoring recipe

```python
def compute_j(pred, gt):
    inter = (pred & gt).sum()
    union = (pred | gt).sum()
    return inter / union if union > 0 else 0.0

def compute_f(pred, gt):
    tp = (pred & gt).sum()
    fp = (pred & ~gt).sum()
    fn = (~pred & gt).sum()
    p = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    r = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * p * r / (p + r) if (p + r) > 0 else 0.0

def compute_s(pred, gt):
    pred_area = pred.sum()
    bg_area = gt.size - gt.sum()
    return np.sqrt(pred_area / bg_area) if bg_area > 0 else 0.0
```

## Common pitfalls

- The S metric is only evaluated on the Null subset, not the Seen/Unseen subsets.
- Lower S values indicate better performance, which is inverse to standard accuracy metrics.
- The test set is strictly partitioned into Seen, Unseen, and Null subsets, requiring separate evaluation rather than a single aggregate score.

## Evidence (verbatim from paper)

> We employed the Jaccard Index $(\mathcal{J})$ and F-score $(\mathcal{F})$ for the Seen and Unseen subsets. Null Subset Metric: We employed the metric $S$, which measures the ratio of the predicted mask area to the background area ( $S = \sqrt{\text{predicted mask area/background area}}$ ). A lower $S$ value indicates better performance (less incorrect segmentation).

## Citation

```bibtex
@misc{song2026paperorchestra,
  title={PaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing},
  author={Song et al. (2026)},
  year={2026},
  note={arXiv:2604.05018}
}
```

- arXiv: 2604.05018