Vad Prediction Eval

---
name: vad-prediction-eval
description: Evaluates a model's ability to predict continuous emotional dimensions (Valence, Arousal, Dominance) from text. Specifically probes the model's capacity to capture affective polarization signals in parliamentary discourse. Use when the user wants to benchmark on Knesset VAD Annotation, or asks about evaluating this task. Reports Pearson correlation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.05231
  bibtex_key: goldin2025unveiling
  confidence: medium
---

# vad-prediction-eval

> Unveiling Affective Polarization Trends in Parliamentary Proceedings — Goldin et al. (2025) (arXiv:2512.05231, 2025)

## What this evaluates

Evaluates a model's ability to predict continuous emotional dimensions (Valence, Arousal, Dominance) from text. Specifically probes the model's capacity to capture affective polarization signals in parliamentary discourse.

## Datasets

- **Knesset VAD Annotation** — total 120; splits: test (120); repo https://github.com/HaifaCLG/Polarization

## Metrics

- `Pearson correlation` **(primary)** — range: [-1, 1]
  - Computes the Pearson correlation coefficient between the model's predicted continuous VAD scores and the manually annotated gold scores for each dimension (Valence, Arousal, Dominance) separately.

## Input / output format

**Input**: Hebrew sentence text.

**Output**: Three continuous floating-point scores in the [0, 1] range representing Valence, Arousal, and Dominance.

## Scoring recipe

```python
def compute_vad_metrics(predictions, gold):
    # predictions and gold are arrays of shape (N, 3) for V, A, D
    metrics = {}
    for i, dim in enumerate(['Valence', 'Arousal', 'Dominance']):
        metrics[dim] = pearsonr(predictions[:, i], gold[:, i]).statistic
    return metrics
```

## Common pitfalls

- VAD scores are derived from human rankings converted to continuous values using a specific scoring script; direct continuous annotation or different normalization will yield different results.
- The evaluation focuses on committee protocols only, excluding plenary sessions to avoid topic and stylistic confounds.
- Hebrew text must be undotted (non-diacriticized) to match the lexicon and model training format.

## Evidence (verbatim from paper)

> We evaluated inter-annotator agreement by computing the mean pair-wise Pearson correlation between individual annotator scores, after applying the scoring script. Despite the rather subjective nature of this task, we obtained reasonably good inter-annotator agreement scores: 0.904, 0.662 and 0.675 for V, A and D, respectively.

## Citation

```bibtex
@misc{goldin2025unveiling,
  title={Unveiling Affective Polarization Trends in Parliamentary Proceedings},
  author={Goldin et al. (2025)},
  year={2025},
  note={arXiv:2512.05231}
}
```

- arXiv: 2512.05231