Mevaker Conclusion Eval

---
name: mevaker-conclusion-eval
description: Evaluates a model's ability to perform binary classification for identifying conclusion sentences in Hebrew audit reports, and to rank sentence pairs by semantic similarity for hierarchical conclusion allocation. Use when the user wants to benchmark on MevakerConcSen, PS (Parallel Sentences), or asks about evaluating this task. Reports F1, Kendall Rank Correlation (KRC).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.09719
  bibtex_key: shalumov2024mevaker
  confidence: high
---

# mevaker-conclusion-eval

> Mevaker: Conclusion Extraction and Allocation Resources for the Hebrew Language — Shalumov et al. (2024) (arXiv:2403.09719, 2024)

## What this evaluates

Evaluates a model's ability to perform binary classification for identifying conclusion sentences in Hebrew audit reports, and to rank sentence pairs by semantic similarity for hierarchical conclusion allocation.

## Datasets

- **MevakerConcSen** — total ?; splits: test (-1)
- **PS (Parallel Sentences)** — total 999000; splits: dev (1000)

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Standard harmonic mean of precision and recall for binary sentence-level classification.
- `Kendall Rank Correlation (KRC)` **(primary)** — range: [-1, 1]
  - Kendall's rank correlation coefficient measuring the correspondence between predicted and ground truth similarity rankings.
- `Mean Absolute Error (MAE)` — range: [0, 1]
  - Mean absolute difference between predicted and ground truth similarity scores, normalized by the number of samples.

## Input / output format

**Input**: For extraction: a window of N sentences (context + target). For allocation: a pair of sentences.

**Output**: For extraction: binary label (conclusion vs. non-conclusion). For allocation: similarity score used to rank sentence pairs.

## Scoring recipe

```python
# F1 for extraction
tp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 1)
fp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 0)
fn = sum(1 for p, g in zip(preds, gold) if p == 0 and g == 1)
prec = tp / (tp + fp) if (tp + fp) > 0 else 0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

# KRC & MAE for allocation
krc = kendalltau(preds, gold).correlation
mae = mean(abs(np.array(preds) - np.array(gold))) / len(gold)
```

## Common pitfalls

- Sliding window training requires disabling shuffling to prevent train/test leakage from overlapping windows.
- Evaluation prioritizes relative ranking metrics (KRC) over absolute similarity scores to ensure fair comparison between cross-encoders and bi-encoders.
- Training splits are artificially balanced via duplication, but test splits retain the original severe class imbalance.

## Evidence (verbatim from paper)

> The evaluation was performed on two metrics - Kendall Rank Correlation (KRC) (Kendall, 1938) and Mean Absolute Error (MAE) normalized by number of samples.

## Citation

```bibtex
@misc{shalumov2024mevaker,
  title={Mevaker: Conclusion Extraction and Allocation Resources for the Hebrew Language},
  author={Shalumov et al. (2024)},
  year={2024},
  note={arXiv:2403.09719}
}
```

- arXiv: 2403.09719