Few Shot Nlg Eval

---
name: few-shot-nlg-eval
description: Evaluates parameter-efficient fine-tuning methods for few-shot natural language generation from structured data (knowledge graphs and semantic representations) to text. It probes the model's ability to adapt to data-scarce regimes while preserving generation fluency and factual alignment with the source structure. Use when the user wants to benchmark on WebNLG 2020, E2E, DART, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2309.06759
  bibtex_key: hu2023scaled
  confidence: high
---

# few-shot-nlg-eval

> Scaled Prompt-Tuning for Few-Shot Natural Language Generation — Ting Hu et al. (2023) (arXiv:2309.06759, 2023)

## What this evaluates

Evaluates parameter-efficient fine-tuning methods for few-shot natural language generation from structured data (knowledge graphs and semantic representations) to text. It probes the model's ability to adapt to data-scarce regimes while preserving generation fluency and factual alignment with the source structure.

## Datasets

- **WebNLG 2020** — total ?; splits: train (-1), dev (-1), test (-1)
- **E2E** — total ?; splits: train (-1), dev (-1), test (-1)
- **DART** — total ?; splits: train (-1), dev (-1), test (-1)

## Metrics

- `BLEU` **(primary)** — range: [0, 1]
  - Standard n-gram overlap metric between generated text and reference. Higher is better.
- `METEOR` — range: [0, 1]
  - Metric for Evaluation of Translation with Explicit ORdering. Higher is better.
- `TER` — range: [0, 1]
  - Translation Edit Ratio: measures the number of edits needed to change the hypothesis to the reference. Lower is better.
- `ROUGE-L` — range: [0, 1]
  - Longest common subsequence metric. Higher is better.

## Input / output format

**Input**: Linearized structured data (triples or slot-value pairs) with delimiter tokens (<S>, <P>, <O> for KGs; <S>, <V> for MRs).

**Output**: Natural language sentence.

## Scoring recipe

```python
def compute_bleu(hypotheses, references):
    # hypotheses and references are lists of strings
    # Uses standard n-gram precision with brevity penalty
    return nltk.translate.bleu_score.corpus_bleu(references, hypotheses)
```

## Common pitfalls

- TER is lower the better, unlike other metrics which are higher the better.
- Results are averaged over 9 runs (3 seeds × 3 sampling iterations) per few-shot scenario.
- Prefix-Tuning degrades significantly in extremely few-shot regimes on WebNLG and DART.

## Evidence (verbatim from paper)

> Experiments are conducted on three NLG datasets: WebNLG 2020, E2E, and DART. Since they contain a large number of instances, we sample subsets of instances from each dataset for few-shot tuning. The sampling process is implemented three times for each few-shot scenario. ... For each dataset, we employ the metrics provided in the benchmark for evaluation. WebNLG 2020 applies BLEU, METEOR, chrF++, TER, BERTScore, and BLEURT. E2E uses BLEU,NIST, METEOR, ROUGE-L, and CIDEr. DART employs BLEU, METEOR, TER, BERTScore, MoverScore, and BLEURT. ... Considering the training instability of few-shot learning, we conduct experiments three times with distinct random seeds on each sampled subset. Eventually, we showcase the average evaluation results of nine experiments in each few-shot case below if not specially claimed.

## Citation

```bibtex
@misc{hu2023scaled,
  title={Scaled Prompt-Tuning for Few-Shot Natural Language Generation},
  author={Ting Hu et al. (2023)},
  year={2023},
  note={arXiv:2309.06759}
}
```

- arXiv: 2309.06759