Vggsound Continual Eval

---
name: vggsound-continual-eval
description: Evaluates a model's ability to perform continual audio-visual classification across sequential tasks without catastrophic forgetting. It measures how well the model retains performance on previously learned categories while learning new ones, across audio, visual, and cross-modal fusion settings. Use when the user wants to benchmark on VGGSound-Instruments, VGGSound-100, VGG-Sound Source, or asks about evaluating this task. Reports Average accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2309.05281
  bibtex_key: mo2023cign
  confidence: high
---

# vggsound-continual-eval

> Class-Incremental Grouping Network for Continual Audio-Visual Learning — Mo et al. (2023) (arXiv:2309.05281, 2023)

## What this evaluates

Evaluates a model's ability to perform continual audio-visual classification across sequential tasks without catastrophic forgetting. It measures how well the model retains performance on previously learned categories while learning new ones, across audio, visual, and cross-modal fusion settings.

## Datasets

- **VGGSound-Instruments** — total 32000; splits: train (-1), val (-1), test (-1)
- **VGGSound-100** — total 97000; splits: train (-1), val (-1), test (-1)
- **VGG-Sound Source** — total 150000; splits: train (-1), val (-1), test (-1)

## Metrics

- `Average accuracy` **(primary)** — range: percent
  - The mean classification accuracy across all T sequential tasks. Calculated as the sum of per-task test accuracies divided by T.
- `Forgetting` — range: percent
  - Measures the performance drop on previously learned tasks after training on subsequent tasks. Typically computed as the average decrease in accuracy from each task's peak performance to its final accuracy.

## Input / output format

**Input**: Paired video clips (10s duration, frames resized to 224×224) and audio segments (3s duration, 22050Hz sample rate converted to 257×300 log spectrograms via STFT). Each instance corresponds to a single audio-visual category.

**Output**: Class label predictions for three modalities: audio-only, visual-only, and audio-visual fusion.

## Scoring recipe

```python
def compute_metrics(task_accuracies, initial_accuracies):
    # task_accuracies: list of final test accuracies per task
    # initial_accuracies: list of peak accuracies achieved per task during training
    avg_acc = sum(task_accuracies) / len(task_accuracies)
    forgetting = sum(max(0, init_acc - curr_acc) for init_acc, curr_acc in zip(initial_accuracies, task_accuracies)) / len(task_accuracies)
    return avg_acc, forgetting
```

## Common pitfalls

- The paper does not specify the exact mathematical formula for 'Forgetting'; standard continual learning literature uses either average drop or max drop, so implementations may vary and must be explicitly stated.
- The evaluation strictly uses a fixed rehearsal buffer of 50 audio-visual pairs per category from old tasks. Omitting or resizing this buffer will invalidate comparisons with the reported baselines.
- The setting is class-incremental with separate test sets per task, meaning the model must predict absolute class labels without task IDs. Confusing this with task-incremental evaluation will inflate results.

## Evidence (verbatim from paper)

> Following the prior work, we use the class-incremental setting of T=4 sequential classification tasks with equal sizes of categories, where each task has a separate test set. With the common metrics in previous methods, we apply Average accuracy and Forgetting for comprehensive evaluation. Higher Average accuracy is better, and lower Forgetting is better.

## Citation

```bibtex
@misc{mo2023cign,
  title={Class-Incremental Grouping Network for Continual Audio-Visual Learning},
  author={Mo et al. (2023)},
  year={2023},
  note={arXiv:2309.05281}
}
```

- arXiv: 2309.05281