Vilco Bench Eval

---
name: vilco-bench-eval
description: Evaluates video-language continual learning by testing a model's ability to retain episodic memories across streaming, long-duration videos without catastrophic forgetting. It probes cross-modal inference and temporal localization across three non-classification tasks: moment queries, natural language queries, and visual queries. Use when the user wants to benchmark on ViLCo-Bench, or asks about evaluating this task. Reports Average Recall@k (IoU=m).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.13123
  bibtex_key: tang2024vilcobench
  confidence: high
---

# vilco-bench-eval

> ViLCo-Bench: VIdeo Language COntinual learning Benchmark — Tang et al. (2024) (arXiv:2406.13123, 2024)

## What this evaluates

Evaluates video-language continual learning by testing a model's ability to retain episodic memories across streaming, long-duration videos without catastrophic forgetting. It probes cross-modal inference and temporal localization across three non-classification tasks: moment queries, natural language queries, and visual queries.

## Datasets

- **ViLCo-Bench** — total ?; splits: test (-1); repo https://github.com/cruiseresearchgroup/ViLCo

## Metrics

- `Average Recall@k (IoU=m)` **(primary)** — range: percent
  - Cumulative average recall across a sequence of tasks: P_i = (1/i) * sum_{j=1}^{i} p_{i,j}, where p_{i,j} is the recall on task i after training on j tasks. Top-k predictions are ranked, and a prediction is correct if its Intersection over Union (IoU) with the ground truth exceeds threshold m. Reported for k in {1,5} and m in {0.3,0.5}.
- `temporal AP (tAP)` — range: percent
  - Average precision computed over a range of IoU thresholds for temporal localization predictions in the Visual Query task. Measures the overlap and distance between predicted and ground-truth bounding boxes.
- `Backward Forgetting (BwF)` — range: percent
  - BwF_i = (1/(i-1)) * sum_{j=1}^{i-1}(p_{j,j} - p_{i,j}), measuring the average performance drop on previous task j after learning new task i. Lower values indicate less catastrophic forgetting.

## Input / output format

**Input**: A long egocentric video (average 10 minutes) paired with a text query (action category, natural language question, or object description) or a visual query frame.

**Output**: For MQ/NLQ: predicted temporal window (start and end timestamps). For VQ: predicted 2D bounding box coordinates and object probability.

## Scoring recipe

```python
def compute_recall_at_k(pred_windows, gt_windows, k=1, iou_thresh=0.5):
    correct = 0
    for gt in gt_windows:
        ious = [compute_iou(p, gt) for p in pred_windows[:k]]
        if max(ious) >= iou_thresh:
            correct += 1
    return correct / len(gt_windows) * 100

def compute_avg_performance(p_matrix, current_task_idx):
    return sum(p_matrix[current_task_idx][:current_task_idx+1]) / (current_task_idx + 1)

def compute_backward_forgetting(p_matrix, current_task_idx):
    drops = [p_matrix[j][j] - p_matrix[j][current_task_idx] for j in range(current_task_idx)]
    return sum(drops) / len(drops) if drops else 0
```

## Common pitfalls

- Label overlap across tasks is common in egocentric videos; strict partitioning is required to prevent data leakage between sub-tasks.
- IoU thresholds (0.3 and 0.5) drastically change recall scores; metrics must be reported separately for each threshold.
- Long video duration (avg 10 mins) makes frame-level rehearsal buffers memory-inefficient; models must use compact prompt/key representations for long-term memory.

## Evidence (verbatim from paper)

> In NLQ and MQ tasks, we adopt average recall@k (IoU=m) as the performance metric, where we select top k={1,5}. This metric presents the percentage of query sentences that appear in the top-k predictions with IoU larger than the threshold m={0.3,0.5}. For the VQ task, we leverage temporal AP (tAP) as the performance metric which measures the distance between the predictions and ground-truth localizations. Again we calculate the average of tAP over the previous tasks.(2) Memory stability metrics: Following [[47]], we also consider Backward Forgetting (BwF) to evaluate the performance of CL models. BwF measures the influence caused by learning task i on the performance of the model in remembering previous tasks. BwF_i is calculated as follows: BwF_i = 1/(i-1) sum_{j=1}^{i-1}(p_{j,j} - p_{i,j}).

## Citation

```bibtex
@misc{tang2024vilcobench,
  title={ViLCo-Bench: VIdeo Language COntinual learning Benchmark},
  author={Tang et al. (2024)},
  year={2024},
  note={arXiv:2406.13123}
}
```

- arXiv: 2406.13123