Parsinlu Eval

---
name: parsinlu-eval
description: Evaluates Persian language understanding across six distinct NLU tasks, including reading comprehension, textual entailment, sentiment analysis, and machine translation. It measures how well pre-trained monolingual and multilingual models perform on native-speaker annotated Persian data compared to human baselines. Use when the user wants to benchmark on ParsiNLU, or asks about evaluating this task. Reports F1, Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2012.06154
  bibtex_key: khashabi2020parsinlu
  confidence: high
---

# parsinlu-eval

> ParsiNLU: A Suite of Language Understanding Challenges for Persian — Khashabi et al. (2020) (arXiv:2012.06154, 2020)

## What this evaluates

Evaluates Persian language understanding across six distinct NLU tasks, including reading comprehension, textual entailment, sentiment analysis, and machine translation. It measures how well pre-trained monolingual and multilingual models perform on native-speaker annotated Persian data compared to human baselines.

## Datasets

- **ParsiNLU** — total ?; splits: train (-1), dev (-1), eval (-1)

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Exact match F1 score computed between the predicted answer string and the gold answer string for reading comprehension tasks.
- `Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted labels or answers across classification tasks such as paraphrasing, entailment, multiple-choice QA, and aspect-specific sentiment.
- `Macro-F1` — range: [0, 1]
  - Macro-averaged F1 score computed across all classes, treating each class equally, used for document-level sentiment and aspect extraction.
- `Sacre-BLEU` — range: [0, 100]
  - BLEU score computed using the SacreBLEU toolkit with standard tokenization and sentence-level scoring for machine translation evaluation.

## Input / output format

**Input**: Varies by task: (1) Reading comprehension: context passage + question; (2) Classification tasks (entailment, paraphrasing, MCQA, sentiment): input text(s) formatted as questions or pairs; (3) Machine translation: source Persian sentence.

**Output**: Varies by task: (1) Reading comprehension: predicted answer string; (2) Classification tasks: predicted label or sentiment polarity; (3) Machine translation: translated Persian sentence.

## Scoring recipe

```python
if task == 'reading_comprehension':
    score = f1_score(gold_answer, pred_answer)
elif task in ['paraphrasing', 'entailment', 'mcqa', 'aspect_sentiment']:
    score = accuracy_score(gold_label, pred_label)
elif task in ['doc_sentiment', 'aspect_extraction']:
    score = macro_f1_score(gold_labels, pred_labels)
elif task == 'machine_translation':
    score = sacrebleu.compute([pred_translation], [gold_translation])
```

## Common pitfalls

- Human performance is estimated on small random subsets (100-150 instances) per task rather than the full evaluation set.
- English datasets cannot be directly used for sentiment analysis due to schema incompatibility, requiring task-specific Persian data.
- Model selection is strictly based on development set performance, not test set performance.

## Evidence (verbatim from paper)

> For reading-comprehension, we use  $F1$  between gold answer and the response string (Rajpurkar et al., 2016); for question paraphrasing, textual entailment, multiple-choice question-answering, and sentiment analysis, we use accuracy. For the first two sub-tasks of sentiment analysis (document-level sentiment, aspect extraction), we use macro- $F1$ . For the third sub-task (aspect-specific semi

ment) we use accuracy as our target evaluation metric (Angelidis and Lapata, 2018; Sun et al., 2019). For machine translation we use Sacre-BLEU (Post, 2018).

## Citation

```bibtex
@misc{khashabi2020parsinlu,
  title={ParsiNLU: A Suite of Language Understanding Challenges for Persian},
  author={Khashabi et al. (2020)},
  year={2020},
  note={arXiv:2012.06154}
}
```

- arXiv: 2012.06154