Neural Gpu Algorithmic Eval

---
name: neural-gpu-algorithmic-eval
description: Evaluates a model's ability to learn algorithmic rules (arithmetic, sequence transformation) from short training sequences and generalize them to arbitrarily long inputs without explicit programming or architectural changes. Use when the user wants to benchmark on Neural GPU Algorithmic Tasks, or asks about evaluating this task. Reports fully_correct_output_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1511.08228
  bibtex_key: kaiser2015neural
  confidence: high
---

# neural-gpu-algorithmic-eval

> Neural GPUs Learn Algorithms — Kaiser et al. (2015) (arXiv:1511.08228, 2015)

## What this evaluates

Evaluates a model's ability to learn algorithmic rules (arithmetic, sequence transformation) from short training sequences and generalize them to arbitrarily long inputs without explicit programming or architectural changes.

## Datasets

- **Neural GPU Algorithmic Tasks** — total ?; splits: test (-1); repo https://github.com/tensorflow/models/tree/master/neural_gpu

## Metrics

- `fully_correct_output_rate` **(primary)** — range: [0, 1]
  - Fraction of test cases where every single bit of the model's output sequence exactly matches the ground truth sequence. No partial credit is awarded.

## Input / output format

**Input**: A sequence of discrete symbols: binary digits (0, 1), an operator (+ or ·), and a padding symbol (PAD). For non-arithmetic tasks, a sequence of bits possibly with padding.

**Output**: A sequence of binary digits (0s and 1s) representing the computed result or transformed sequence.

## Scoring recipe

```python
correct_count = 0
total_count = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct_count += 1
    total_count += 1
return correct_count / total_count
```

## Common pitfalls

- Generalization is evaluated on lengths far exceeding training length (e.g., train on 20-bit, test on 2000-bit).
- The metric is strict exact-match; partial correctness on any bit yields a score of 0 for that instance.
- Tasks are procedurally generated on-the-fly rather than drawn from a static corpus.

## Evidence (verbatim from paper)

> We measure also the rate of fully correct output sequences and report the results in Table 1. The table shows the fraction of test cases for which every single bit of the model’s output is correct.

## Citation

```bibtex
@misc{kaiser2015neural,
  title={Neural GPUs Learn Algorithms},
  author={Kaiser et al. (2015)},
  year={2015},
  note={arXiv:1511.08228}
}
```

- arXiv: 1511.08228