Test Accuracy

---
name: test-accuracy
description: Evaluates the convergence speed and final test performance of distributed synchronous versus asynchronous stochastic gradient descent algorithms. It probes whether backup workers in synchronous training can mitigate stragglers without degrading accuracy due to gradient staleness. Use when the user has predictions and gold and needs to compute test accuracy.
metadata:
  skill_kind: metric
  source_arxiv: 1604.00981
  bibtex_key: [REDACTED]
  confidence: high
---

# test-accuracy

> Revisiting Distributed Synchronous SGD — Chen et al. (2016) (arXiv:1604.00981, 2016)

## What this evaluates

Evaluates the convergence speed and final test performance of distributed synchronous versus asynchronous stochastic gradient descent algorithms. It probes whether backup workers in synchronous training can mitigate stragglers without degrading accuracy due to gradient staleness.

## Datasets

- **ImageNet** — total ?; splits: train (-1), test (-1)
- **CIFAR-10** — total ?; splits: train (-1), test (-1)

## Metrics

- `test accuracy` **(primary)** — range: percent
  - Percentage of correctly classified images on the validation/test set. Computed as (correct predictions / total predictions) * 100.
- `epochs to converge` — range: int
  - Number of training epochs required to reach a predefined target test precision threshold.
- `time to converge` — range: hours
  - Wall-clock time in hours required to reach the target test precision or NLL threshold.
- `negative log likelihood (NLL)` — range: other
  - Test negative log likelihood for the PixelCNN generative model, where lower values indicate better density estimation.

## Input / output format

**Input**: Distributed training of Inception on ImageNet and PixelCNN on CIFAR-10 using TensorFlow, with varying total workers (N) and backup workers (b) from 1 to 212.

**Output**: Convergence curves and final reported values for test precision/accuracy, epochs to reach target precision, wall-clock time to converge, and test NLL.

## Scoring recipe

```python
def compute_metrics(predictions, gold, target_precision=0.77):
    accuracy = (predictions == gold).mean() * 100
    epochs = count_epochs_until(accuracy >= target_precision)
    time = wall_clock_time_until(accuracy >= target_precision)
    return accuracy, epochs, time
```

## Common pitfalls

- Focusing only on early-phase convergence speed can be misleading, as faster convergence may lead to poorer local optima with lower final accuracy.
- Comparing epoch counts across synchronous and asynchronous methods without accounting for the fact that an epoch represents different amounts of computation in each setting.

## Evidence (verbatim from paper)

> We are interested in two metrics of comparison for our empirical validation:
(1) test error or accuracy, and (2) speed of convergence.

## Citation

```bibtex
@misc{chen2016revisiting,
  title={Revisiting Distributed Synchronous SGD},
  author={Chen et al. (2016)},
  year={2016},
  note={arXiv:1604.00981}
}
```

- arXiv: 1604.00981