Cl Ifeval Eval

---
name: cl-ifeval-eval
description: Evaluates large language models' ability to follow complex, variable-driven instructions across multiple languages. It measures strict compliance with prompt constraints to reveal cross-lingual robustness disparities. The benchmark highlights how functional tasks expose performance gaps that static benchmarks often miss. Use when the user wants to benchmark on CL-IFEval, or asks about evaluating this task. Reports Strict Prompt Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.20793
  bibtex_key: ojewale2025multilingual
  confidence: high
---

# cl-ifeval-eval

> Multi-lingual Functional Evaluation for Large Language Models — Ojewale et al. (2025) (arXiv:2506.20793, 2025)

## What this evaluates

Evaluates large language models' ability to follow complex, variable-driven instructions across multiple languages. It measures strict compliance with prompt constraints to reveal cross-lingual robustness disparities. The benchmark highlights how functional tasks expose performance gaps that static benchmarks often miss.

## Datasets

- **CL-IFEval** — total ?; splits: test (-1)

## Metrics

- `Strict Prompt Accuracy` **(primary)** — range: percent
  - Percentage of instances where the model's output exactly matches all required constraints and formatting specified in the prompt.

## Input / output format

**Input**: Instruction prompts translated into English, French, Spanish, Hindi, Arabic, or Yoruba, containing dynamic variables and functional constraints.

**Output**: Model-generated text responses that must satisfy all specified constraints.

## Scoring recipe

```python
For each instance, parse the model output and check if it satisfies all constraints defined in the prompt.
Mark as correct (1) if all constraints are met, else incorrect (0).
Average across instances to get Strict Prompt Accuracy.
```

## Common pitfalls

- Strict accuracy requires exact compliance with all constraints; partial compliance is scored as 0.
- Performance varies significantly across languages, with low-resource languages like Yoruba showing drastically lower scores than high-resource ones.
- Model rankings on functional benchmarks differ from static benchmarks, so cross-benchmark comparisons can be misleading.

## Evidence (verbatim from paper)

> Figure 3: Cross-Lingual IFEval Strict Prompt Accuracy.

## Citation

```bibtex
@misc{ojewale2025multilingual,
  title={Multi-lingual Functional Evaluation for Large Language Models},
  author={Ojewale et al. (2025)},
  year={2025},
  note={arXiv:2506.20793}
}
```

- arXiv: 2506.20793