Tanda Augmentation Eval

---
name: tanda-augmentation-eval
description: Evaluates whether automatically composing domain-specific data augmentation transformations improves end-task classification performance. It probes the ability of a learned sequence model to generate effective augmentation pipelines compared to heuristic or random baselines across image and text domains. Use when the user wants to benchmark on MNIST, CIFAR-10, ACE (Employment relation extraction), DDSM (Mammography), or asks about evaluating this task. Reports test set accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1709.01643
  bibtex_key: ratner2017tanda
  confidence: high
---

# tanda-augmentation-eval

> Learning to Compose Domain-Specific Transformations for Data Augmentation — Ratner et al. (2017) (arXiv:1709.01643, 2017)

## What this evaluates

Evaluates whether automatically composing domain-specific data augmentation transformations improves end-task classification performance. It probes the ability of a learned sequence model to generate effective augmentation pipelines compared to heuristic or random baselines across image and text domains.

## Datasets

- **MNIST** — total ?; splits: test (-1); HF `mnist`
- **CIFAR-10** — total ?; splits: test (-1); HF `cifar10`
- **ACE (Employment relation extraction)** — total ?; splits: test (-1)
- **DDSM (Mammography)** — total 1506; splits: test (-1)

## Metrics

- `test set accuracy` **(primary)** — range: percent
  - Percentage of correctly classified instances in the test set. Computed as (number of correct predictions) / (total test instances).
- `F1 score` — range: [0, 1]
  - Harmonic mean of precision and recall for the positive class, used specifically for the ACE relation extraction task.

## Input / output format

**Input**: Labeled training data (subsampled), unlabeled data for generator training, and held-out test set. The end classifier receives augmented training instances generated by composing transformation functions.

**Output**: Class labels predicted by the end classifier for each test instance.

## Scoring recipe

```python
def compute_metric(predictions, gold, task):
    if task == 'ACE':
        tp = sum(1 for p, g in zip(predictions, gold) if p == g == 'positive')
        fp = sum(1 for p, g in zip(predictions, gold) if p == 'positive' and g != 'positive')
        fn = sum(1 for p, g in zip(predictions, gold) if p != 'positive' and g == 'positive')
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    return sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)
```

## Common pitfalls

- Using the full labeled dataset for augmentation training instead of the specified unlabeled subset.
- Failing to apply the task-specific sequence lengths (L=5 for ACE, L=7 for DDSM+DS, L=10 for others) calibrated during initial experiments.
- Comparing against baselines without including the same random cropping/Basic augmentation step used by the proposed method.

## Evidence (verbatim from paper)

> We present these results in Table 2, where we report test set accuracy (or F1 score for ACE), and use a random subsample of the available labeled training data.

## Citation

```bibtex
@misc{ratner2017tanda,
  title={Learning to Compose Domain-Specific Transformations for Data Augmentation},
  author={Ratner et al. (2017)},
  year={2017},
  note={arXiv:1709.01643}
}
```

- arXiv: 1709.01643