Bloom Empirical Eval

---
name: bloom-empirical-eval
description: Evaluates BLOOM model variants against BERT-style and GPT-style baselines across diverse NLP tasks including text classification, question answering, zero/few-shot learning, multilingual transfer, and text generation. Use when the user wants to benchmark on GLUE, SQuAD, XNLI, MARC, Zero/FSL Benchmarks, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2211.14865
  bibtex_key: [REDACTED]
  confidence: high
---

# bloom-empirical-eval

> Understanding BLOOM: An empirical study on diverse NLP tasks — Dakle et al. (2022) (arXiv:2211.14865, 2022)

## What this evaluates

Evaluates BLOOM model variants against BERT-style and GPT-style baselines across diverse NLP tasks including text classification, question answering, zero/few-shot learning, multilingual transfer, and text generation.

## Datasets

- **GLUE** — total ?; splits: validation (-1)
- **SQuAD** — total ?; splits: train (-1), validation (-1)
- **XNLI** — total ?; splits: validation (-1)
- **MARC** — total ?; splits: test (-1)
- **Zero/FSL Benchmarks** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted labels out of total samples. Used for GLUE, MARC, XNLI, and FSL tasks.
- `exact-match` — range: [0, 1]
  - Fraction of predictions where the generated answer exactly matches the ground truth span. Used for SQuAD.
- `F1` — range: [0, 1]
  - Harmonic mean of precision and recall over token overlap between predicted and gold answers. Used for SQuAD.
- `BLEU` — range: [0, 1]
  - Geometric mean of modified n-gram precisions with brevity penalty. Used for text generation evaluation.
- `Matthews correlation` — range: other
  - Correlation coefficient between observed and predicted binary classifications. Used for COLA.

## Input / output format

**Input**: Varies by task: fine-tuning inputs for GLUE/SQuAD/MARC/XNLI; prompt-based inputs with 0/1/4/8 examples for FSL tasks; context+question formatted as '[CLS]context[SEP]question[SEP]' for SQuAD; prompts for toxicity generation.

**Output**: Task-specific: class labels for classification, span indices for QA, generated text for TG/toxicity.

## Scoring recipe

```python
def compute_metrics(predictions, golds, task):
    if task == 'classification':
        return sum(p == g for p, g in zip(predictions, golds)) / len(golds)
    elif task == 'qa':
        em = sum(p == g for p, g in zip(predictions, golds)) / len(golds)
        # F1 computed via token overlap
        return em, f1_score(golds, predictions)
    elif task == 'generation':
        return bleu_score(golds, predictions)
    elif task == 'cola':
        return matthews_corrcoef(golds, predictions)
```

## Common pitfalls

- MNLI scores are averaged over matched and mismatched subtasks, not reported separately.
- BLOOM models initially predict a single class for COLA/SST2 due to prompt/tokenization issues, requiring padding strategy changes to fix.
- SQuAD uses '[CLS]' token not in BLOOM's vocabulary, affecting no-answer questions; replacing it with '<s>' improves performance.

## Evidence (verbatim from paper)

> For the MNLI task, we report the average of the matched and mismatched accuracy scores. ... The table shows that the bloom-560m model outperforms gpt2-medium on the SQUAD v.1.1 dataset. Although the metric scores are significantly lower compared to BERT (~73% exact match), the results show that when compared to gpt2-medium, a similar architecture style model, bloom-560m performs better.

## Citation

```bibtex
@misc{dakle2022understandingbloom,
  title={Understanding BLOOM: An empirical study on diverse NLP tasks},
  author={Dakle et al. (2022)},
  year={2022},
  note={arXiv:2211.14865}
}
```

- arXiv: 2211.14865