Persian Instruction Following Eval

---
name: persian-instruction-following-eval
description: Evaluates the instruction-following capability of Persian large language models across multiple NLP tasks, including paraphrasing, sentiment analysis, and textual entailment. Use when the user wants to benchmark on parsinlu queryparaphrasing, Digikala SentimentAnalysis, FarsTail, ParsinluEntailment, or asks about evaluating this task. Reports ROUGE-L F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.11186
  bibtex_key: [REDACTED]
  confidence: high
---

# persian-instruction-following-eval

> Empowering Persian LLMs for Instruction Following: A Novel Dataset and Training Approach — Mokhtarabadi et al. (2024) (arXiv:2407.11186, 2024)

## What this evaluates

Evaluates the instruction-following capability of Persian large language models across multiple NLP tasks, including paraphrasing, sentiment analysis, and textual entailment.

## Datasets

- **parsinlu queryparaphrasing** — total ?; splits: test (-1)
- **Digikala SentimentAnalysis** — total ?; splits: test (-1)
- **FarsTail** — total ?; splits: test (-1)
- **ParsinluEntailment** — total ?; splits: test (-1)

## Metrics

- `ROUGE-L F1` **(primary)** — range: percent
  - Computes the F1 score for the longest common subsequence (LCS) of words between the predicted and reference text. F1 is the harmonic mean of precision (LCS length / predicted length) and recall (LCS length / reference length).

## Input / output format

**Input**: Instruction prompt with task-specific context or query (exact format not specified in the provided section).

**Output**: Generated text response to the instruction (exact format not specified in the provided section).

## Scoring recipe

```python
def compute_rouge_l_f1(predictions, references):
    f1_scores = []
    for p, r in zip(predictions, references):
        lcs = longest_common_subsequence(p, r)
        prec = len(lcs) / len(p) if len(p) > 0 else 0
        rec = len(lcs) / len(r) if len(r) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores) * 100
```

## Common pitfalls

- ROUGE-L is a surface-form metric that may poorly correlate with semantic correctness, especially for entailment tasks.
- The evaluation table reports ROUGE-L F1, but some baseline models might report precision or recall separately, causing direct comparison issues if not standardized.
- Held-in versus held-out split definitions are not explicitly detailed in the provided text, making reproducibility of generalization claims difficult.

## Evidence (verbatim from paper)

> These models are evaluated using ROUGE-L F1 scores. As illustrated in Table[2], Co-CoLA performs comparably well to the Aya model, despite having fewer parameters and being trained on less instruction data and significantly outperforms all other models, indicating the effectiveness of Co-CoLA.

## Citation

```bibtex
@misc{mokhtarabadi2024farsinstruct,
  title={Empowering Persian LLMs for Instruction Following: A Novel Dataset and Training Approach},
  author={Mokhtarabadi et al. (2024)},
  year={2024},
  note={arXiv:2407.11186}
}
```

- arXiv: 2407.11186