Zest Eval

---
name: zest-eval
description: Evaluates a model's ability to understand and generalize across unseen NLP tasks based solely on task descriptions, rather than few-shot examples. It probes systematic generalization across variations like paraphrasing, composition, semantic flips, and output structure changes. Use when the user wants to benchmark on ZEST, or asks about evaluating this task. Reports Mean.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2011.08115
  bibtex_key: weller2020learning
  confidence: medium
---

# zest-eval

> Learning from Task Descriptions — Weller et al. (2020) (arXiv:2011.08115, 2020)

## What this evaluates

Evaluates a model's ability to understand and generalize across unseen NLP tasks based solely on task descriptions, rather than few-shot examples. It probes systematic generalization across variations like paraphrasing, composition, semantic flips, and output structure changes.

## Datasets

- **ZEST** — total ?; splits: dev (-1), test (-1)

## Metrics

- `Mean` **(primary)** — range: percent
  - Average score across all instances in a split.
- `C@75` — range: percent
  - Performance metric at the 75% threshold (likely accuracy or coverage at top-75 predictions).
- `C@90` — range: percent
  - Performance metric at the 90% threshold (likely accuracy or coverage at top-90 predictions).

## Input / output format

**Input**: Task description formulated as a question paired with a passage.

**Output**: Model prediction (exact format not specified in text).

## Scoring recipe

```python
Compute mean score across instances.
Calculate C@75 and C@90 by evaluating predictions at the 75th and 90th percentile thresholds respectively.
Report per generalization type and overall.
```

## Common pitfalls

- Metrics are stratified by generalization type (Base, Paraphrase, etc.), so reporting only overall scores hides systematic failures.
- C@75 and C@90 thresholds are not explicitly defined in the provided text, requiring careful interpretation.
- Human performance is reported separately and significantly outperforms models, indicating a large capability gap.

## Evidence (verbatim from paper)

> Table 7: Detailed T5 results for ZEST with multi-task training.  

<table><tr><td rowspan=2>Generalization Type</td><td colspan=3>Dev</td><td colspan=3>Test</td></tr><tr><td>Mean</td><td>C@75</td><td>C@90</td>...

## Citation

```bibtex
@misc{weller2020learning,
  title={Learning from Task Descriptions},
  author={Weller et al. (2020)},
  year={2020},
  note={arXiv:2011.08115}
}
```

- arXiv: 2011.08115