Cl Gsmsym Eval

---
name: cl-gsmsym-eval
description: Assesses mathematical reasoning and symbolic computation capabilities of LLMs across multiple languages. It uses dynamic, variable-driven templates to generate verifiable ground truths for each instance. The evaluation probes model resilience to linguistic variations and template-specific weaknesses. Use when the user wants to benchmark on CL-GSMSym, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.20793
  bibtex_key: ojewale2025multilingual
  confidence: high
---

# cl-gsmsym-eval

> Multi-lingual Functional Evaluation for Large Language Models — Ojewale et al. (2025) (arXiv:2506.20793, 2025)

## What this evaluates

Assesses mathematical reasoning and symbolic computation capabilities of LLMs across multiple languages. It uses dynamic, variable-driven templates to generate verifiable ground truths for each instance. The evaluation probes model resilience to linguistic variations and template-specific weaknesses.

## Datasets

- **CL-GSMSym** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly solved mathematical problems where the model's output matches the verifiable ground truth.

## Input / output format

**Input**: Mathematical reasoning prompts translated into English, French, Spanish, Hindi, Arabic, or Yoruba, featuring dynamic variables and symbolic constraints.

**Output**: Model-generated text responses containing the solution to the mathematical problem.

## Scoring recipe

```python
For each instance, extract the model's final answer and compare it against the verifiable ground truth.
Mark as correct (1) if they match exactly, else incorrect (0).
Average across instances to compute accuracy.
```

## Common pitfalls

- Performance on low-resource languages like Yoruba drops drastically compared to high-resource languages.
- Template-level robustness varies significantly; some templates (e.g., probabilistic inference) consistently yield lower accuracy across all models and languages.
- Static benchmark rankings do not correlate with functional benchmark rankings, leading to misleading model comparisons.

## Evidence (verbatim from paper)

> We define the language performance gap to be the difference of the model’s accuracy on its lowest performant language, and the highest performant language.

## Citation

```bibtex
@misc{ojewale2025multilingual,
  title={Multi-lingual Functional Evaluation for Large Language Models},
  author={Ojewale et al. (2025)},
  year={2025},
  note={arXiv:2506.20793}
}
```

- arXiv: 2506.20793