Germeval17 Eval

---
name: germeval17-eval
description: Evaluates German NLP models on aspect-based sentiment analysis (ABSA) tasks, including relevance classification, document-level polarity, aspect/sentiment classification, and opinion target extraction. Use when the user wants to benchmark on GermEval17, or asks about evaluating this task. Reports micro F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2102.12330
  bibtex_key: assenmacher2021reevaluatinggermeval17
  confidence: high
---

# germeval17-eval

> Re-Evaluating GermEval17 Using German Pre-Trained Language Models — Aßenmacher et al. (2021) (arXiv:2102.12330, 2021)

## What this evaluates

Evaluates German NLP models on aspect-based sentiment analysis (ABSA) tasks, including relevance classification, document-level polarity, aspect/sentiment classification, and opinion target extraction.

## Datasets

- **GermEval17** — total ?; splits: test_syn (-1), test_dia (-1); repo https://github.com/ac74/reevaluating-germeval2017

## Metrics

- `micro F1` **(primary)** — range: [0, 1]
  - Micro-averaged F1 score computed across all classes, treating each instance equally regardless of class frequency.
- `entity-level micro F1 (exact match)` — range: [0, 1]
  - Entity-level micro F1 score requiring exact token boundary matches between predicted and gold entities.
- `entity-level micro F1 (overlapping match)` — range: [0, 1]
  - Entity-level micro F1 score tolerating a deviation of +/- one token in entity boundaries.

## Input / output format

**Input**: German social media text documents. For token-level tasks, sequence positions are provided to map tokens to BIO tags.

**Output**: For classification tasks: predicted class labels (e.g., true/false, positive/neutral/negative, aspect categories, sentiments). For token-level tasks: BIO-tag sequences (B-, I-, O) for each token.

## Scoring recipe

```python
def compute_micro_f1(predictions, gold):
    tp = fp = fn = 0
    for p, g in zip(predictions, gold):
        if p == g: tp += 1
        elif p != g:
            fp += 1
            fn += 1
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
```

## Common pitfalls

- Multiple mentions of the same label in one document are not counted separately for Subtask C.
- Comparing results against models that used different data splits (e.g., Guhr et al. 2020) limits comparability.
- Overlapping match (D2) tolerates a deviation of +/- one token in entity boundaries, which significantly changes scoring compared to exact match.

## Evidence (verbatim from paper)

> Subtask A The Relevance Classification is a binary document classification task with classes true and false. Table 7 displays the micro F1 score obtained by each language model on each test set (best result per data set in bold).

## Citation

```bibtex
@misc{assenmacher2021reevaluatinggermeval17,
  title={Re-Evaluating GermEval17 Using German Pre-Trained Language Models},
  author={Aßenmacher et al. (2021)},
  year={2021},
  note={arXiv:2102.12330}
}
```

- arXiv: 2102.12330