Headlinecause Eval

---
name: headlinecause-eval
description: Evaluates a model's ability to detect implicit causal relationships between two news headlines without relying on explicit causal linking words. It probes commonsense reasoning and world knowledge to distinguish between causal, refutational, same-event, and unrelated headline pairs. Use when the user wants to benchmark on HeadlineCause, or asks about evaluating this task. Reports causality ROC AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2108.12626
  bibtex_key: gusev2021headlinecause
  confidence: high
---

# headlinecause-eval

> HeadlineCause: A Dataset of News Headlines for Detecting Causalities — Ilya Gusev, Alexey Tikhonov (2021) (arXiv:2108.12626, 2021)

## What this evaluates

Evaluates a model's ability to detect implicit causal relationships between two news headlines without relying on explicit causal linking words. It probes commonsense reasoning and world knowledge to distinguish between causal, refutational, same-event, and unrelated headline pairs.

## Datasets

- **HeadlineCause** — total ?; splits: test (967); repo https://github.com/IlyaGusev/HeadlineCause

## Metrics

- `causality ROC AUC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve for the binary causal vs non-causal classification task. Calculated by combining Left-right and Right-left causality classes into a single positive class to allow threshold variation.
- `F-score` — range: [0, 1]
  - Per-class F1-score for the multiclass Full task, computed across seven relationship categories including causality, refutation, and non-causal relations.
- `Total multiclass accuracy` — range: [0, 1]
  - Percentage of correctly classified headline pairs across all seven relationship categories in the Full task.

## Input / output format

**Input**: A pair of news headline strings (one cause, one effect, or vice versa) in either English or Russian.

**Output**: A categorical label indicating the relationship type: binary (Causal vs Non-causal) for the Simple task, or multiclass (No relationship, Same event, Other relationship, Left-right causality, Right-left causality, Left-right refutation, Right-left refutation) for the Full task.

## Scoring recipe

```python
def score(predictions, gold, task='simple'):
    if task == 'simple':
        y_true = [1 if g in ['Left-right causality', 'Right-left causality'] else 0 for g in gold]
        y_pred = [1 if p in ['Left-right causality', 'Right-left causality'] else 0 for p in predictions]
        return roc_auc_score(y_true, y_pred)
    else:
        acc = accuracy_score(gold, predictions)
        f1s = f1_score(gold, predictions, average=None, zero_division=0)
        return acc, f1s
```

## Common pitfalls

- The dataset specifically targets implicit causality; models relying on explicit causal connectors (e.g., 'because', 'therefore') will artificially inflate performance.
- Refutation classes are extremely imbalanced in the test set (e.g., only 5-8 samples per language), making per-class F1 scores for refutations statistically unreliable.
- The Simple task merges Left-right and Right-left causality into a single positive class, so evaluators must not report separate F1 scores for directionality when evaluating the primary metric.

## Evidence (verbatim from paper)

> For this task, we consider causality ROC AUC on two classes as a main metric. To calculate it, we unite Left-right and Right-left classes to be able to vary a classifier threshold.

## Citation

```bibtex
@misc{gusev2021headlinecause,
  title={HeadlineCause: A Dataset of News Headlines for Detecting Causalities},
  author={Ilya Gusev, Alexey Tikhonov (2021)},
  year={2021},
  note={arXiv:2108.12626}
}
```

- arXiv: 2108.12626