Ccc Eval

---
name: ccc-eval
description: Evaluates continual semi-supervised learning on crowd counting by measuring how well a model adapts to evolving unlabeled data streams across sequential sessions. Use when the user wants to benchmark on Continual Crowd Counting (CCC), or asks about evaluating this task. Reports Mean Absolute Error (MAE).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2110.14613
  bibtex_key: shahbaz2021cssl
  confidence: high
---

# ccc-eval

> International Workshop on Continual Semi-Supervised Learning: Introduction, Benchmarks and Baselines — Shahbaz et al. (2021) (arXiv:2110.14613, 2021)

## What this evaluates

Evaluates continual semi-supervised learning on crowd counting by measuring how well a model adapts to evolving unlabeled data streams across sequential sessions.

## Datasets

- **Continual Crowd Counting (CCC)** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `Mean Absolute Error (MAE)` **(primary)** — range: [0, inf)
  - Average absolute difference between predicted and true crowd counts across all test instances: MAE = (1/N) * sum(|y_pred - y_true|).

## Input / output format

**Input**: Sequential crowd counting image streams divided into sessions, containing unlabeled validation and test folds for incremental self-training.

**Output**: Predicted crowd count/density for each image in the stream.

## Scoring recipe

```python
def compute_mae(predictions, gold):
    if len(predictions) != len(gold):
        raise ValueError('Length mismatch')
    errors = [abs(p - g) for p, g in zip(predictions, gold)]
    return sum(errors) / len(errors)
```

## Common pitfalls

- Comparing MAE values directly to standard batch-trained crowd counting papers without noting the different training protocol (400 vs 800 supervised images).
- Treating validation and test folds as a single contiguous stream when the protocol specifies separate incremental updates.
- Assuming incremental updates always improve performance; the paper notes updates are not always effective on validation streams.

## Evidence (verbatim from paper)

> Table 2 shows a quantitative analysis of the performance of the fine-tuned supervised model (sup) and two incrementally updated models (upd) on the validation and the test split, respectively, using the mean absolute error (MAE) metric.

## Citation

```bibtex
@misc{shahbaz2021cssl,
  title={International Workshop on Continual Semi-Supervised Learning: Introduction, Benchmarks and Baselines},
  author={Shahbaz et al. (2021)},
  year={2021},
  note={arXiv:2110.14613}
}
```

- arXiv: 2110.14613