Sahm Eval

---
name: sahm-eval
description: Evaluates Arabic language models on financial and Shari’ah-compliant reasoning across multiple task types, including multiple-choice questions, extractive summarization, and open-ended question answering. It probes the gap between general Arabic fluency and domain-specific procedural/financial reasoning. Use when the user wants to benchmark on SAHM, or asks about evaluating this task. Reports exact-match accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.19098
  bibtex_key: elbadry2026sahm
  confidence: high
---

# sahm-eval

> SAHM: A Benchmark for Arabic Financial and Shari'ah-Compliant Reasoning — Elbadry et al. (2026) (arXiv:2604.19098, 2026)

## What this evaluates

Evaluates Arabic language models on financial and Shari’ah-compliant reasoning across multiple task types, including multiple-choice questions, extractive summarization, and open-ended question answering. It probes the gap between general Arabic fluency and domain-specific procedural/financial reasoning.

## Datasets

- **SAHM** — total 14380; splits: train (-1), test (-1); repo https://github.com/rania-hossam/SAHM

## Metrics

- `exact-match accuracy` **(primary)** — range: [0, 1]
  - 1 if the normalized prediction matches the gold choice exactly, else 0. Free-form outputs (option text or letters) are normalized to a single canonical choice before comparison.
- `ROUGE-F1` — range: [0, 1]
  - Standard ROUGE-F1 score computed over ROUGE-1, ROUGE-2, and ROUGE-L n-gram overlaps between the model's verbatim output and the gold extractive reference.
- `LLM-as-a-judge score` — range: [0, 10]
  - Additive [0,10] score generated by Gemini-2.5-Flash evaluating alignment with the reference ruling, preservation of constraints/quantitative fidelity, correctness, Arabic clarity, and directness/grounding.

## Input / output format

**Input**: Arabic prompts with task-specific instructions (e.g., MCQ options, source text for summarization, or open-ended questions). For judge scoring, the original prompt, gold reference, and model answer are provided.

**Output**: For MCQs: normalized to a single choice (option letter or text). For summarization: verbatim sentences extracted from the source. For QA: free-form Arabic text.

## Scoring recipe

```python
def score_mcq(pred, gold):
    normalized_pred = normalize_to_single_choice(pred)
    return 1.0 if normalized_pred == gold else 0.0

def score_summarization(pred, gold):
    return rouge_f1(pred, gold) # ROUGE-1/2/L

def score_qa_judge(prompt, gold, pred):
    # Greedy decoding (temp=0) required
    score = judge_score(prompt, gold, pred, rubric="alignment, constraints, correctness, clarity, grounding")
    return score # [0, 10]
```

## Common pitfalls

- Arabic fluency does not guarantee financial reasoning; models often score >90% on recognition tasks but drop to <10% on open-ended generation.
- MCQ outputs must be normalized to a single choice (letter or text) before exact-match scoring; raw free-form outputs will fail.
- LLM-as-a-judge requires greedy decoding (temperature 0) and fixed max lengths to ensure reproducibility across models.

## Evidence (verbatim from paper)

> We evaluate Accounting Exams, Business Exams, Fatwa MCQ, and Financial Sentiment with exact-match accuracy, normalizing free-form outputs (e.g., option text/letters) to a single choice before scoring (Appendix [H](#A8 "Appendix H MCQ Answer Normalization and Scoring ‣ Sahm: A Benchmark for Arabic Financial and Shari’ah-Compliant Reasoning")). For extractive summarization, we report ROUGE-F1 (ROUGE-1/2/L) against gold extractive references (models are instructed to output verbatim sentences). For Fatwa QA, Shari’ah Standards QA, and Event-Cause QA, we use Gemini-2.5-Flash as an LLM-as-a-judge (blind to model identity): given the original Arabic prompt, gold reference, and model answer, it returns a JSON-validated, additive (sum-of-components) $[0,10]$ score under a shared rubric assessing alignment with the reference ruling/conclusion, preservation of key constraints or quantitative fidelity, correctness (doctrinal/factual or financial reasoning), Arabic clarity, and directness/grounding.

## Citation

```bibtex
@misc{elbadry2026sahm,
  title={SAHM: A Benchmark for Arabic Financial and Shari'ah-Compliant Reasoning},
  author={Elbadry et al. (2026)},
  year={2026},
  note={arXiv:2604.19098}
}
```

- arXiv: 2604.19098