Clove Clvqa Eval

---
name: clove-clvqa-eval
description: Evaluates continual learning capabilities on Visual Question Answering (CLVQA) by measuring how well a model retains knowledge from previous tasks while learning new ones across scene-incremental and function-incremental settings. Use when the user wants to benchmark on CLOVE, or asks about evaluating this task. Reports average accuracy (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2208.12037
  bibtex_key: lei2022symbolicreplay
  confidence: high
---

# clove-clvqa-eval

> Symbolic Replay: Scene Graph as Prompt for Continual Learning on VQA Task — Lei et al. (2022) (arXiv:2208.12037, 2022)

## What this evaluates

Evaluates continual learning capabilities on Visual Question Answering (CLVQA) by measuring how well a model retains knowledge from previous tasks while learning new ones across scene-incremental and function-incremental settings.

## Datasets

- **CLOVE** — total ?; splits: train (-1), test (-1); repo https://github.com/showlab/CLVQA

## Metrics

- `average accuracy (%)` **(primary)** — range: percent
  - Accuracy is the percentage of correctly predicted answers. The final reported metric is the average accuracy across all tasks in the sequence and the 6 randomly sampled task orders.

## Input / output format

**Input**: An image and a natural language question (VQA prompt)

**Output**: A natural language answer (text generation, maximum 12 decoding steps)

## Scoring recipe

```python
def compute_cl_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return (correct / len(golds)) * 100

# Evaluation is performed only at the final iteration of the last task in the sequence.
# Results are averaged across 6 randomly sampled task orders for both CLOVE-scene and CLOVE-function settings.
```

## Common pitfalls

- Models are evaluated exclusively at the last iteration of the final task, not continuously during training.
- Task orders are randomized (6 permutations sampled), so performance varies significantly by order; always report the average across orders.
- CLOVE has two distinct incremental settings (scene and function) that must be evaluated and reported separately.

## Evidence (verbatim from paper)

> We randomly sample 6 task orders from all possible permutations for evaluation. ... Besides, for both settings, we set γ=1.5 for SGP and report the average accuracy. We use models at last iteration of last task for testing. (Table 2 caption: Summary of average accuracy(%) for different methods under six task orders in CLOVE-scene and CLOVE-function respectively.)

## Citation

```bibtex
@misc{lei2022symbolicreplay,
  title={Symbolic Replay: Scene Graph as Prompt for Continual Learning on VQA Task},
  author={Lei et al. (2022)},
  year={2022},
  note={arXiv:2208.12037}
}
```

- arXiv: 2208.12037