Foreseaqa Eval

---
name: foreseaqa-eval
description: Evaluates a model's ability to perform temporally grounded, multimodal video understanding in surveillance settings. It probes precise temporal localization, identity-based search, and complex reasoning over long videos using text-only or image+text queries. Use when the user wants to benchmark on ForeSeaQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.22872
  bibtex_key: park2026foresea
  confidence: high
---

# foreseaqa-eval

> ForeSea: AI Forensic Search with Multi-modal Queries for Video Surveillance — Park et al. (2026) (arXiv:2603.22872, 2026)

## What this evaluates

Evaluates a model's ability to perform temporally grounded, multimodal video understanding in surveillance settings. It probes precise temporal localization, identity-based search, and complex reasoning over long videos using text-only or image+text queries.

## Datasets

- **ForeSeaQA** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions.
- `temporal localization IoU` — range: [0, 1]
  - Intersection-over-union between the predicted and ground-truth time intervals, averaged over all questions.

## Input / output format

**Input**: Long surveillance videos paired with either text-only or multimodal (image+text) queries.

**Output**: A multiple-choice answer selection and a predicted temporal time interval (start and end timestamps).

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    correct = sum(1 for p, g in zip(predictions['answer'], gold['answer']) if p == g)
    accuracy = correct / len(predictions['answer'])
    ious = []
    for p, g in zip(predictions['interval'], gold['interval']):
        inter = max(0, min(p[1], g[1]) - max(p[0], g[0]))
        union = max(p[1], g[1]) - min(p[0], g[0])
        ious.append(inter / union if union > 0 else 0)
    iou = sum(ious) / len(ious)
    return accuracy, iou
```

## Common pitfalls

- Models often achieve reasonable multiple-choice accuracy but produce low temporal IoU, indicating answers are inferred from global video context rather than grounded evidence.
- Multimodal queries (image+text) are consistently harder than text-only queries, exposing a gap in joint reasoning over reference images and long videos.
- Counting tasks are particularly challenging, with models frequently under-counting occurrences and failing to follow the required output format.

## Evidence (verbatim from paper)

> We report *accuracy* (percentage of correctly answered multiple-choice questions) and temporal localization *IoU* (intersection-over-union between the predicted and ground-truth time intervals, averaged over all questions) as the two primary metrics.

## Citation

```bibtex
@misc{park2026foresea,
  title={ForeSea: AI Forensic Search with Multi-modal Queries for Video Surveillance},
  author={Park et al. (2026)},
  year={2026},
  note={arXiv:2603.22872}
}
```

- arXiv: 2603.22872