Leaf Federated Eval

---
name: leaf-federated-eval
description: Evaluates federated learning algorithms under realistic constraints including device-level data skew, heterogeneous data distributions, and communication bottlenecks. It measures model accuracy after federated training across multiple simulated devices. Use when the user wants to benchmark on Shakespeare, Sent140, FEMNIST, CelebA, Synthetic, Reddit, or asks about evaluating this task. Reports AccuracyTop1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1812.01097
  bibtex_key: caldas2018leaf
  confidence: high
---

# leaf-federated-eval

> LEAF: A Benchmark for Federated Settings — Caldas et al. (2018) (arXiv:1812.01097, 2018)

## What this evaluates

Evaluates federated learning algorithms under realistic constraints including device-level data skew, heterogeneous data distributions, and communication bottlenecks. It measures model accuracy after federated training across multiple simulated devices.

## Datasets

- **Shakespeare** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/TalwalkarLab/leaf
- **Sent140** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/TalwalkarLab/leaf
- **FEMNIST** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/TalwalkarLab/leaf
- **CelebA** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/TalwalkarLab/leaf
- **Synthetic** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/TalwalkarLab/leaf
- **Reddit** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/TalwalkarLab/leaf

## Metrics

- `AccuracyTop1` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted tokens or classes out of the total number of predictions on the test set. For text tasks, predictions of unknown and padding tokens are explicitly counted as incorrect.

## Input / output format

**Input**: Per user/device, sequences of tokens (text) or images (FEMNIST/CelebA) or feature vectors (Synthetic). Data is partitioned per user with a 60/20/20 train/val/test split.

**Output**: Predicted class label or token for each input instance.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)
```

## Common pitfalls

- Data is inherently non-IID and partitioned per user, not globally shuffled.
- Evaluation must be performed on held-out test sets per user or a global test set, never on training data.
- Federated protocols require tracking communication rounds and client sampling ratios, not just final model accuracy.

## Evidence (verbatim from paper)

> We evaluate using AccuracyTop1. ... evaluate using AccuracyTop1 and consider all predictions of the unknown and padding tokens as incorrect.

## Citation

```bibtex
@misc{caldas2018leaf,
  title={LEAF: A Benchmark for Federated Settings},
  author={Caldas et al. (2018)},
  year={2018},
  note={arXiv:1812.01097}
}
```

- arXiv: 1812.01097