Coda Eval

---
name: coda-eval
description: Evaluates a training-free, constraint-based data augmentation framework for low-resource NLP. It probes whether synthetically augmented data improves downstream performance across sequence classification, intent classification, named entity recognition, and question answering tasks compared to gold-only and other augmentation baselines. Use when the user wants to benchmark on Huffpost, Yahoo, OTS, ATIS, Massive, ConLL-2003, OntoNotes-5.0, EBMNLP, BC2GM, SQuAD, NewsQA, or asks about evaluating this task. Reports micro-average F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.00415
  bibtex_key: evuru2024coda
  confidence: high
---

# coda-eval

> CoDa: Constrained Generation based Data Augmentation for Low-Resource NLP — Evuru et al. (2024) (arXiv:2404.00415, 2024)

## What this evaluates

Evaluates a training-free, constraint-based data augmentation framework for low-resource NLP. It probes whether synthetically augmented data improves downstream performance across sequence classification, intent classification, named entity recognition, and question answering tasks compared to gold-only and other augmentation baselines.

## Datasets

- **Huffpost** — total ?; splits: train (-1)
- **Yahoo** — total ?; splits: train (-1)
- **OTS** — total ?; splits: train (-1)
- **ATIS** — total ?; splits: train (-1)
- **Massive** — total ?; splits: train (-1)
- **ConLL-2003** — total ?; splits: train (-1)
- **OntoNotes-5.0** — total ?; splits: train (-1)
- **EBMNLP** — total ?; splits: train (-1)
- **BC2GM** — total ?; splits: train (-1)
- **SQuAD** — total ?; splits: train (-1)
- **NewsQA** — total ?; splits: train (-1)

## Metrics

- `micro-average F1 score` **(primary)** — range: [0, 1]
  - Aggregates true positives, false positives, and false negatives across all classes before computing precision and recall. F1 is calculated as 2 * (precision * recall) / (precision + recall).

## Input / output format

**Input**: Text instances for downstream NLU tasks (sequence classification, intent classification, NER, QA) with low-resource training splits (e.g., 50, 100, 200, 500, or 1000 examples).

**Output**: Predicted class labels, entity tags, or answers corresponding to the input instances.

## Scoring recipe

```python
tp, fp, fn = 0, 0, 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold: tp += 1
    else: fp += 1; fn += 1
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
micro_f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
return micro_f1  # averaged over 3 runs × 3 seeds
```

## Common pitfalls

- Reporting macro-F1 instead of the specified micro-average F1.
- Failing to average results across the 3 runs and 3 random seeds as mandated.
- Using full-dataset splits instead of the specified low-resource training sizes (50, 100, 200, 500, 1000).

## Evidence (verbatim from paper)

> We report the micro-average F1 score averaged across 3 runs for 3 random seeds.

## Citation

```bibtex
@misc{evuru2024coda,
  title={CoDa: Constrained Generation based Data Augmentation for Low-Resource NLP},
  author={Evuru et al. (2024)},
  year={2024},
  note={arXiv:2404.00415}
}
```

- arXiv: 2404.00415