Decanlp Eval

---
name: decanlp-eval
description: This benchmark evaluates a model's ability to perform multitask learning across ten diverse natural language processing tasks by framing them as a unified question-answering problem. It probes zero-shot generalization, domain adaptation, and the effectiveness of anti-curriculum training strategies without relying on task-specific modules. Use when the user wants to benchmark on decaNLP, or asks about evaluating this task. Reports decaScore.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1806.08730
  bibtex_key: mccann2018decanlp
  confidence: high
---

# decanlp-eval

> The Natural Language Decathlon: Multitask Learning as Question Answering — McCann et al. (2018) (arXiv:1806.08730, 2018)

## What this evaluates

This benchmark evaluates a model's ability to perform multitask learning across ten diverse natural language processing tasks by framing them as a unified question-answering problem. It probes zero-shot generalization, domain adaptation, and the effectiveness of anti-curriculum training strategies without relying on task-specific modules.

## Datasets

- **decaNLP** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `decaScore` **(primary)** — range: score
  - Composite benchmark score aggregating performance across the 10 diverse NLP tasks. Exact aggregation details are provided in Appendix D.

## Input / output format

**Input**: Question and context provided as input sequences. For baselines, they are concatenated into a single sequence; for the MQAN model, they are processed as separate sequences with a coattention mechanism.

**Output**: Generated answer text, produced via a multi-pointer-generator decoder that dynamically selects between generating tokens from an external vocabulary, pointing to the question, or pointing to the context.

## Scoring recipe

```python
def compute_score(predictions, gold, task_type):
    if task_type in ['SST', 'MNLI', 'MWSC', 'SNLI', 'Amazon', 'Yelp']:
        return 1.0 if predictions == gold else 0.0
    elif task_type in ['SQuAD', 'QA-SRL']:
        return nF1_score(predictions, gold)
    elif task_type == 'WikiSQL':
        return lfem_score(predictions, gold)
    # decaScore aggregates these task-specific scores across all 10 tasks
```

## Common pitfalls

- Assuming the model uses task-specific heads or modules; the paper explicitly evaluates a unified architecture without task-specific parameters.
- Misinterpreting the input format for baselines vs. MQAN; baselines concatenate question and context, while MQAN uses separate sequences with coattention, which significantly affects the model's ability to copy tokens.
- Overlooking that decaScore is a composite metric; evaluating on a single task does not reflect the full benchmark performance.

## Evidence (verbatim from paper)

> In Table 2, validation metrics reveal that the S2S model does not perform well on SQuAD. On WikiSQL, it obtains a much higher score than prior sequence-to-sequence baselines [Zhong et al., 2017], but it is low compared to MQAN (+QPtr) and the other baselines. Augmenting the S2S model with self-attentive (w/ SAtt) encoder and decoder layers Vaswani et al. [2017], as detailed in C, increases the model's capacity to integrate information from both context and question. This improves performance on SQuAD by 20 nF1, QA-SRL by 4 nF1, and WikiSQL by 12 LFEM.

## Citation

```bibtex
@misc{mccann2018decanlp,
  title={The Natural Language Decathlon: Multitask Learning as Question Answering},
  author={McCann et al. (2018)},
  year={2018},
  note={arXiv:1806.08730}
}
```

- arXiv: 1806.08730