Flue Eval

---
name: flue-eval
description: Probes French sequence classification capabilities across sentiment analysis, paraphrase identification, and natural language inference. Use when the user wants to benchmark on CLS, PAWSX, XNLI, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2010.12321
  bibtex_key: moussa2020barthez
  confidence: high
---

# flue-eval

> BARThez: a Skilled Pretrained French Sequence-to-Sequence Model — Moussa Kamal Eddine et al. (2020) (arXiv:2010.12321, 2020)

## What this evaluates

Probes French sequence classification capabilities across sentiment analysis, paraphrase identification, and natural language inference.

## Datasets

- **CLS** — total 6000; splits: train (4800), test (2000)
- **PAWSX** — total 53378; splits: train (49401), dev (1992), test (1985)
- **XNLI** — total 400202; splits: train (392702), dev (2490), test (5010)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly classified instances over the total number of instances.

## Input / output format

**Input**: Single sentence or sentence pair (separated by a special token), fed to both encoder and decoder; last token representation used for classification.

**Output**: Class label (e.g., positive/negative, equivalent/not, entailment/contradiction/neutral)

## Scoring recipe

```python
correct = 0; total = 0
for doc in dataset:
    seq = format_pair(doc.s1, doc.s2)
    logits = model(seq)
    pred = argmax(logits)
    if pred == doc.label: correct += 1
    total += 1
accuracy = correct / total
```

## Common pitfalls

- BART-based models require feeding input to both encoder and decoder and using the last token's representation, unlike standard BERT classification.
- Results are averaged over 3 runs with different seeds; reporting single runs may misrepresent performance variance.
- CLS dataset is split into 3 product categories; evaluation should be averaged or reported per category.

## Evidence (verbatim from paper)

> We evaluate the different models on five discriminative tasks from the FLUE benchmark (Le et al., 2019), the French equivalent of GLUE (Wang et al., 2018). ... Table 10 reports the test set accuracies.

## Citation

```bibtex
@misc{moussa2020barthez,
  title={BARThez: a Skilled Pretrained French Sequence-to-Sequence Model},
  author={Moussa Kamal Eddine et al. (2020)},
  year={2020},
  note={arXiv:2010.12321}
}
```

- arXiv: 2010.12321