Frenchbench Eval

---
name: frenchbench-eval
description: Evaluates bilingual French-English language understanding, cultural knowledge, and generation capabilities of LLMs across classification and open-ended tasks. Probes the model's ability to perform few-shot reasoning, factual recall, and text generation in both languages. Use when the user wants to benchmark on FrenchBench, English Benchmarks, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.00786
  bibtex_key: faysse2024croissantllm
  confidence: high
---

# frenchbench-eval

> CroissantLLM: A Truly Bilingual French-English Language Model — Faysse et al. (2024) (arXiv:2402.00786, 2024)

## What this evaluates

Evaluates bilingual French-English language understanding, cultural knowledge, and generation capabilities of LLMs across classification and open-ended tasks. Probes the model's ability to perform few-shot reasoning, factual recall, and text generation in both languages.

## Datasets

- **FrenchBench** — total ?; splits: test (-1); repo https://github.com/CoderPat/croissant-llm-training
- **English Benchmarks** — total ?; splits: test (-1); repo https://github.com/CoderPat/croissant-llm-training

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted labels over total instances. Computed as the mean accuracy across all classification tasks in the benchmark.
- `ROUGE-1` — range: [0, 1]
  - Unigram overlap F1 score between the generated text and the reference text. Calculated as 2 * (precision * recall) / (precision + recall).

## Input / output format

**Input**: 5-shot prompt containing a task description, five example question-answer pairs, and the target query. For classification tasks, options are provided; for generation tasks, open-ended questions are given.

**Output**: Model generates a sequence of tokens. For classification, the predicted option label or text. For generation, free-form text response.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    acc_scores = []
    rouge_scores = []
    for pred, gold in zip(predictions, golds):
        if gold in ['A', 'B', 'C', 'D']:
            acc_scores.append(1.0 if pred.strip() == gold else 0.0)
        else:
            rouge_scores.append(rouge1_score(gold, pred))
    return {'accuracy': sum(acc_scores)/len(acc_scores) if acc_scores else 0,
            'rouge1': sum(rouge_scores)/len(rouge_scores) if rouge_scores else 0}
```

## Common pitfalls

- Evaluating base models in few-shot (5-shot) settings rather than zero-shot, which inflates performance on instruction-following and reasoning tasks.
- Using greedy decoding for translation and generation benchmarks instead of beam search or temperature sampling, which may underperform on open-ended generation.
- Assuming ROUGE-1 is sufficient for all generation tasks; the paper notes heuristic metrics may be insufficient and uses GPT-4 as a judge for some tasks.

## Evidence (verbatim from paper)

> We score results obtained in 5-shot settings with ROUGE-1. As heuristic-based metrics are often insufficient to capture the diversity of possible answers (Faysse et al., [2023]), we also score predictions using GPT4 as a judge, and confirm the ROUGE1 metric is well suited for this task given the closed and short nature of the answers.

## Citation

```bibtex
@misc{faysse2024croissantllm,
  title={CroissantLLM: A Truly Bilingual French-English Language Model},
  author={Faysse et al. (2024)},
  year={2024},
  note={arXiv:2402.00786}
}
```

- arXiv: 2402.00786