Polish Nlp Eval

---
name: polish-nlp-eval
description: Evaluates Polish language understanding, summarization, and question answering capabilities of text-to-text models. It probes how well encoder-decoder and decoder-only architectures generalize from multilingual pre-training to monolingual Polish tasks using exact-match generation and ROUGE-based metrics. Use when the user wants to benchmark on KLEJ benchmark, Allegro Articles, Polish Summaries Corpus, or asks about evaluating this task. Reports exact-match accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2205.08808
  bibtex_key: chrabrowa2022evaluation
  confidence: high
---

# polish-nlp-eval

> Evaluation of Transfer Learning for Polish with a Text-to-Text Model — Chrabrowa et al. (2022) (arXiv:2205.08808, 2022)

## What this evaluates

Evaluates Polish language understanding, summarization, and question answering capabilities of text-to-text models. It probes how well encoder-decoder and decoder-only architectures generalize from multilingual pre-training to monolingual Polish tasks using exact-match generation and ROUGE-based metrics.

## Datasets

- **KLEJ benchmark** — total ?; splits: train (-1), val (-1), test (-1)
- **Allegro Articles** — total 33000; splits: train (-1), test (-1)
- **Polish Summaries Corpus** — total 569; splits: train (-1), test (-1)

## Metrics

- `exact-match accuracy` **(primary)** — range: [0, 1]
  - Percentage of test instances where the generated token sequence exactly matches the gold label. For specific KLEJ tasks, F1, Spearman correlation, or MAE-based scores are used instead.
- `ROUGE AVG` — range: [0, 1]
  - Arithmetic mean of the F-measure scores for ROUGE-1, ROUGE-2, and ROUGE-L. Calculated as (ROUGE-1 + ROUGE-2 + ROUGE-L) / 3.

## Input / output format

**Input**: Text-to-text prompts with descriptive prefixes (e.g., '<Prefix 1>: text 1 <Prefix 2>: text 2' for KLEJ), trimmed source texts (typically ≤1024 tokens for summarization), and question-context pairs for QA.

**Output**: Greedy token generation until EOS or max target length. For KLEJ, semantically significant text tokens matching the gold label. For summarization/QA, generated text sequences.

## Scoring recipe

```python
def score(predictions, golds, task_type):
    if task_type == 'classification':
        return sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    elif task_type == 'summarization':
        r1 = rouge_fmeasure(predictions, golds, n=1)
        r2 = rouge_fmeasure(predictions, golds, n=2)
        rl = rouge_fmeasure(predictions, golds, n='l')
        return (r1 + r2 + rl) / 3
```

## Common pitfalls

- Input truncation to ~1024 tokens severely limits context for summarization, meaning models never see 100% of source text during training.
- Exact-match scoring is strict; only precise token matches count as correct, ignoring semantic equivalence.
- ROUGE metrics are noted as imperfect for abstractive summaries, with human upper bounds sometimes matching model performance.

## Evidence (verbatim from paper)

> Targets were generated over the whole vocabulary, and only an exact match was treated as the correct answer. Results are shown in Table 4 which contains arithmetic mean of (f-measure) ROUGE-1, ROUGE-2 and ROUGE-L (Lin, 2004) for each model and task.

## Citation

```bibtex
@misc{chrabrowa2022evaluation,
  title={Evaluation of Transfer Learning for Polish with a Text-to-Text Model},
  author={Chrabrowa et al. (2022)},
  year={2022},
  note={arXiv:2205.08808}
}
```

- arXiv: 2205.08808