Balsam Eval

---
name: balsam-eval
description: Evaluates Arabic large language models across 14 diverse NLP categories, including creative writing, question answering, reading comprehension, logic, and machine translation. It probes the models' ability to handle complex Arabic morphology, long-form generation, and task-specific reasoning. Use when the user wants to benchmark on BALSAM, or asks about evaluating this task. Reports LLM as a judge.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.22603
  bibtex_key: almatham2025balsam
  confidence: high
---

# balsam-eval

> BALSAM: A Platform for Benchmarking Arabic Large Language Models — Al-Matham et al. (2025) (arXiv:2507.22603, 2025)

## What this evaluates

Evaluates Arabic large language models across 14 diverse NLP categories, including creative writing, question answering, reading comprehension, logic, and machine translation. It probes the models' ability to handle complex Arabic morphology, long-form generation, and task-specific reasoning.

## Datasets

- **BALSAM** — total 52000; splits: test (-1), dev (-1); repo https://github.com/ksaa-nlp/balsam-eval

## Metrics

- `LLM as a judge` **(primary)** — range: [0, 3]
  - An LLM (Gemini 2.5 Flash) first extracts a concise answer from the model's verbose output, then rates it on a 0–3 scale mirroring human evaluation guidelines.

## Input / output format

**Input**: Arabic language prompts/tasks spanning 14 categories (e.g., Creative Writing, QA, Translation, Logic, Program Execution).

**Output**: Model-generated text responses.

## Scoring recipe

```python
def score_llm_judge(predictions, gold):
    extracted = [llm_extract(p, prompt=EXTRACTION_PROMPT) for p in predictions]
    scores = [llm_rate(e, g, prompt=SCORING_PROMPT, scale=[0,1,2,3]) for e, g in zip(extracted, gold)]
    return sum(scores) / len(scores)
```

## Common pitfalls

- Traditional n-gram metrics (BLEU/ROUGE) heavily penalize verbose outputs and fail to handle Arabic's complex morphology, leading to artificially low scores for longer, correct answers.
- Model size is not a reliable predictor of Arabic capability; tokenization quality, Arabic training data volume, and Arabic-specific SFT are more critical factors.

## Evidence (verbatim from paper)

> Table [5] shows the correlation of human evaluation with ROUGE-LSum, BLEU, and BERTScore (with and without extraction of answers using an LLM) and LLM as a judge. We make the following observations: ... LLM as a judge was highly correlated with human judgments for all categories, with values ranging between 0.824 and 0.977.

## Citation

```bibtex
@misc{almatham2025balsam,
  title={BALSAM: A Platform for Benchmarking Arabic Large Language Models},
  author={Al-Matham et al. (2025)},
  year={2025},
  note={arXiv:2507.22603}
}
```

- arXiv: 2507.22603