Czenecclassification Eval

---
name: czenecclassification-eval
description: Evaluates NLP models on four challenging text classification tasks using a large-scale Czech news dataset: identifying the news source, article category, inferred author gender, and publication day of the week. The benchmark tests a model's ability to capture deep textual dependencies and contextual cues beyond simple keyword matching. Use when the user wants to benchmark on CZE-NEC, or asks about evaluating this task. Reports F1 Macro.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2307.10666
  bibtex_key: kydlicek2023czenecczechnews
  confidence: high
---

# czenecclassification-eval

> A Dataset and Strong Baselines for Classification of Czech News Texts — Hynek Kydlíček, Jindřich Libovický (2023) (arXiv:2307.10666, 2023)

## What this evaluates

Evaluates NLP models on four challenging text classification tasks using a large-scale Czech news dataset: identifying the news source, article category, inferred author gender, and publication day of the week. The benchmark tests a model's ability to capture deep textual dependencies and contextual cues beyond simple keyword matching.

## Datasets

- **CZE-NEC** — total 1600000; splits: train (-1), test (-1), test_small (10000), test_human (-1); repo https://github.com/hynky1999/Czech-News-Classification-dataset

## Metrics

- `F1 Macro` **(primary)** — range: [0, 1]
  - Macro-averaged F1 score computed across all classes for each task. Calculated as the unweighted mean of the F1 scores for each individual class.

## Input / output format

**Input**: Raw Czech news article text (truncated to 510 tokens for Transformer models).

**Output**: Single class label per task (e.g., news source name, category, author gender, or day of the week).

## Scoring recipe

```python
def compute_macro_f1(predictions, gold_labels, classes):
    f1_scores = []
    for cls in classes:
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == cls and g == cls)
        fp = sum(1 for p, g in zip(predictions, gold_labels) if p == cls and g != cls)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != cls and g == cls)
        precision = tp / (tp + fp) if (tp + fp) > 0 else 0
        recall = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)
```

## Common pitfalls

- Models may achieve high accuracy by simply spotting keywords for specific classes rather than understanding deeper context.
- Text truncation to 510 tokens may discard relevant information for tasks like author gender or day of week.
- Human baseline performance is low due to inherent ambiguity, making human comparison scores misleading if not contextualized with inter-annotator agreement.

## Evidence (verbatim from paper)

> Table 3. Tasks F1 Macro scores for selected models on the Test set. We use — to denote that the model failed to converge for all tested learning rates.

## Citation

```bibtex
@misc{kydlicek2023czenecczechnews,
  title={A Dataset and Strong Baselines for Classification of Czech News Texts},
  author={Hynek Kydlíček, Jindřich Libovický (2023)},
  year={2023},
  note={arXiv:2307.10666}
}
```

- arXiv: 2307.10666