General Nlu Eval

---
name: general-nlu-eval
description: Evaluates whether integrating external knowledge (textual descriptions or embeddings) improves performance on general natural language understanding tasks compared to baseline pre-trained language models. It probes the model's ability to leverage external semantic information to enhance representation learning and decision-making across classification, regression, and sequence labeling benchmarks. Use when the user wants to benchmark on GLUE, Penn Treebank, CoNLL-2003, or asks about evaluating this task. Reports Accuracy / F1 / Pearson correlation / Matthew's correlation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2109.00563
  bibtex_key: xu2021doesknowledgehelp
  confidence: high
---

# general-nlu-eval

> Does Knowledge Help General NLU? An Empirical Study — Xu et al. (2021) (arXiv:2109.00563, 2021)

## What this evaluates

Evaluates whether integrating external knowledge (textual descriptions or embeddings) improves performance on general natural language understanding tasks compared to baseline pre-trained language models. It probes the model's ability to leverage external semantic information to enhance representation learning and decision-making across classification, regression, and sequence labeling benchmarks.

## Datasets

- **GLUE, Penn Treebank, CoNLL-2003** — total ?; splits: train (-1), val (-1); HF `glue, ptb, conll2003`

## Metrics

- `Accuracy / F1 / Pearson correlation / Matthew's correlation` **(primary)** — range: [0, 1] | percent
  - Task-specific metrics: Accuracy for classification tasks, F1 for sequence labeling, Pearson correlation for STS-B, and Matthew's correlation for CoLA. Scores are averaged across tasks for overall comparison.

## Input / output format

**Input**: Sentence or token sequence for classification/regression tasks; token sequence for sequence labeling tasks.

**Output**: Task-specific prediction: class label, regression value, or token-level sequence tags.

## Scoring recipe

```python
def compute_metric(preds, golds, task):
    if task in ['SST-2', 'MNLI', 'QQP', 'QNLI', 'MRPC', 'RTE']:
        return accuracy(golds, preds)
    elif task == 'CoLA':
        return matthews_corrcoef(golds, preds)
    elif task == 'STS-B':
        return pearson_corr(golds, preds)
    elif task in ['POS', 'NER']:
        return f1_score(golds, preds, average='micro')
```

## Common pitfalls

- Reports medians over 5 random seeds, not means.
- Evaluation is performed on the development set rather than the official held-out test set.
- Task-specific metrics are used: Accuracy for classification, Matthew's correlation for CoLA, Pearson correlation for STS-B, and F1 for sequence labeling.

## Evidence (verbatim from paper)

> Metrics: Matt. corr., Acc., Pear. corr., F1. We report the median of results on the development set over five fixed random seeds for all tasks.

## Citation

```bibtex
@misc{xu2021doesknowledgehelp,
  title={Does Knowledge Help General NLU? An Empirical Study},
  author={Xu et al. (2021)},
  year={2021},
  note={arXiv:2109.00563}
}
```

- arXiv: 2109.00563