Ilsvrc Eval

---
name: ilsvrc-eval
description: Evaluates large-scale visual recognition capabilities across three core tasks: image classification, single-object localization, and object detection. It probes a model's ability to categorize, localize, and detect objects across 1,000 diverse categories using a dataset of approximately 1 million images. Use when the user wants to benchmark on ILSVRC, or asks about evaluating this task. Reports classification error.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1409.0575
  bibtex_key: [REDACTED]
  confidence: high
---

# ilsvrc-eval

> ImageNet Large Scale Visual Recognition Challenge — Olga Russakovsky et al. (arXiv:1409.0575, 2014)

## What this evaluates

Evaluates large-scale visual recognition capabilities across three core tasks: image classification, single-object localization, and object detection. It probes a model's ability to categorize, localize, and detect objects across 1,000 diverse categories using a dataset of approximately 1 million images.

## Datasets

- **ILSVRC** — total 1000000; splits: train (-1), val (-1), test (-1)

## Metrics

- `classification error` **(primary)** — range: percent
  - Computed as 1 minus the top-k accuracy (typically top-5 in early years, top-1 later). Measures the fraction of images where the ground-truth class is not among the top predicted classes.
- `mean average precision` — range: [0, 1]
  - The average of the area under the precision-recall curve computed per object class, then averaged across all 200 categories in the detection task.

## Input / output format

**Input**: RGB images of varying resolutions, with ground-truth annotations including class labels, bounding box coordinates, and object presence indicators.

**Output**: For classification: a ranked list of predicted class labels with confidence scores. For localization/detection: a set of predicted bounding boxes with associated class labels and confidence scores.

## Scoring recipe

```python
def compute_classification_error(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if g in p[:5])
    return 1.0 - (correct / len(gold_labels))

def compute_mAP(predictions, gold_boxes, classes):
    aps = []
    for cls in classes:
        precisions, recalls = compute_pr_curve(predictions, gold_boxes, cls)
        aps.append(trapezoidal_rule(precisions, recalls))
    return sum(aps) / len(aps)
```

## Common pitfalls

- Starting in 2014, teams were allowed to use external training data, creating separate 'provided data' and 'external data' tracks that must not be mixed when reporting results.
- Single-object localization requires predicting exactly one bounding box per image, whereas object detection requires predicting all instances of multiple classes per image.
- Early ILSVRC years primarily reported top-5 classification error, while later years shifted focus to top-1 error, requiring careful year-specific metric alignment.

## Evidence (verbatim from paper)

> The ILSVRC dataset and the competition has allowed significant algorithmic advances in large-scale image recognition and retrieval. ... Significant progress has been made in just one year: image classification error was almost halved since ILSVRC2013 and object detection mean average precision almost doubled compared to ILSVRC2013.

## Citation

```bibtex
@misc{russakovsky2014ilsvrc,
  title={ImageNet Large Scale Visual Recognition Challenge},
  author={Olga Russakovsky et al.},
  year={2014},
  note={arXiv:1409.0575}
}
```

- arXiv: 1409.0575