Eci Eval

---
name: eci-eval
description: This evaluation protocol assesses the capability of NLP models to identify causal relationships between event pairs in text. It probes both sentence-level and document-level reasoning, measuring how well models can distinguish true causal links from mere correlations or coincidental co-occurrences. Use when the user wants to benchmark on CTB, ESL, MAVEN-ERE, MECI, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.10371
  bibtex_key: cheng2024survey
  confidence: high
---

# eci-eval

> A Survey of Event Causality Identification: Taxonomy, Challenges, Assessment, and Prospects — Qing Cheng et al. (2024) (arXiv:2411.10371, 2024)

## What this evaluates

This evaluation protocol assesses the capability of NLP models to identify causal relationships between event pairs in text. It probes both sentence-level and document-level reasoning, measuring how well models can distinguish true causal links from mere correlations or coincidental co-occurrences.

## Datasets

- **CTB** — total ?; splits: test (-1)
- **ESL** — total ?; splits: dev (-1), train (-1)
- **MAVEN-ERE** — total ?; splits: test (-1), val (-1)
- **MECI** — total ?; splits: test (-1)

## Metrics

- `F1-score` **(primary)** — range: percent
  - F1 = 2 * (Precision * Recall) / (Precision + Recall). Precision is the proportion of predicted causal pairs that are correct, and Recall is the proportion of actual causal pairs that are correctly identified.
- `Precision` — range: percent
  - Precision = TP / (TP + FP). Measures the proportion of predicted causal relationships that are actually correct.
- `Recall` — range: percent
  - Recall = TP / (TP + FN). Measures the proportion of actual causal relationships that are correctly identified by the model.

## Input / output format

**Input**: For sentence-level tasks: [Sentence] [SEP] Is there a causal relationship between [Event1] and [Event2]? For document-level tasks: [Document] [SEP] Is there a causal relationship between [Event1] and [Event2]?

**Output**: Binary classification output (Yes/No or direct generation) indicating the presence or absence of a causal relationship between the specified event pair.

## Scoring recipe

```python
tp = sum(1 for pred, gold in zip(predictions, golds) if pred == 1 and gold == 1)
fp = sum(1 for pred, gold in zip(predictions, golds) if pred == 1 and gold == 0)
fn = sum(1 for pred, gold in zip(predictions, golds) if pred == 0 and gold == 1)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
return {'precision': precision, 'recall': recall, 'f1': f1}
```

## Common pitfalls

- Directional accuracy is explicitly ignored; only the existence of a causal link is evaluated.
- LLMs often suffer from 'causal hallucination,' producing high recall but low precision due to over-predicting causal links.
- Dataset splits and cross-validation schemes vary significantly across benchmarks (e.g., 10-fold vs 5-fold vs fixed dev/test), complicating direct performance comparisons.
- Some baseline models use distinct data processing methods or different splits than reported, requiring careful reproduction or direct citation to maintain fairness.

## Evidence (verbatim from paper)

> We evaluated the models using Precision (P), Recall (R), and F1-score (F1) as metrics, distinguishing between intra-sentence causality, inter-sentence causality, and overall performance. Given that most existing methods evaluate only the presence of causalities without considering directional accuracy, we focused our comparison similarly on causality existence alone.

## Citation

```bibtex
@misc{cheng2024survey,
  title={A Survey of Event Causality Identification: Taxonomy, Challenges, Assessment, and Prospects},
  author={Qing Cheng et al. (2024)},
  year={2024},
  note={arXiv:2411.10371}
}
```

- arXiv: 2411.10371