E Care Eval

---
name: e-care-eval
description: Evaluates a model's ability to perform commonsense causal reasoning by predicting the reasonableness of causal facts, and to generate conceptually grounded natural language explanations for those causal relationships. Use when the user wants to benchmark on e-CARE, or asks about evaluating this task. Reports Accuracy (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2205.05849
  bibtex_key: du2022ecare
  confidence: high
---

# e-care-eval

> e-CARE: a New Dataset for Exploring Explainable Causal Reasoning — Li Du et al. (2022) (arXiv:2205.05849, 2022)

## What this evaluates

Evaluates a model's ability to perform commonsense causal reasoning by predicting the reasonableness of causal facts, and to generate conceptually grounded natural language explanations for those causal relationships.

## Datasets

- **e-CARE** — total 21000; splits: test (-1); repo https://github.com/Waste-Wood/e-CARE

## Metrics

- `Accuracy (%)` **(primary)** — range: percent
  - Standard classification accuracy calculated as the proportion of correctly predicted causal fact reasonableness labels out of the total test instances.
- `CEQ score` — range: [0, 1]
  - A novel metric proposed by the authors to evaluate explanation quality by assessing how well explanations promote understanding of causal mechanisms, rather than relying on textual similarity.
- `Human Evaluation (%)` — range: percent
  - Percentage of model-generated explanations labeled as valid by three human annotators who verify if the explanation correctly explains the corresponding causal fact.
- `AVG-BLEU` — range: [0, 1]
  - Average n-gram precision (n=4) between generated explanations and human-written reference explanations, averaged across all test instances.
- `ROUGE-1` — range: [0, 1]
  - Recall of unigram overlap between generated explanations and reference explanations.
- `ΔAccu. (%) after Adv. Attack` — range: percent
  - Change in causal reasoning accuracy after applying a gradient-based FGM adversarial perturbation to the input word embeddings.

## Input / output format

**Input**: Causal reasoning: A candidate causal fact composed of a premise and a corresponding hypothesis. Explanation generation: Concatenation of the cause and effect.

**Output**: Causal reasoning: A score or label measuring the reasonableness of the candidate causal fact. Explanation generation: A natural language conceptual explanation.

## Scoring recipe

```python
def evaluate_causal_reasoning(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels) * 100

def evaluate_explanations(generated_exps, gold_exps, human_labels):
    bleu = compute_avg_bleu4(generated_exps, gold_exps)
    rouge1 = compute_rouge1(generated_exps, gold_exps)
    valid_count = sum(1 for exp in generated_exps if human_labels[exp] == 'valid')
    human_acc = valid_count / len(generated_exps) * 100
    return bleu, rouge1, human_acc
```

## Common pitfalls

- Models may exploit superficial lexical cues rather than genuine causal understanding, leading to inflated performance on smaller benchmarks.
- Standard text-similarity metrics (AVG-BLEU, ROUGE-1) show near-zero correlation with human judgments of explanation quality, making them unreliable for this task.

## Evidence (verbatim from paper)

> We cast the causal reasoning task as a prediction problem: The input of the model is a candidate causal fact composed of a premise and one of the corresponding candidate hypotheses. The output is a score measuring the reasonableness of the candidate causal fact. We automatically evaluate the quality of generated explanations using average-BLEU (n = 4) (Papineni et al., 2002), ROUGE-1 (Lin, 2004), Perplexity (Horgan, 1995), together with our proposed CEQ score.

## Citation

```bibtex
@misc{du2022ecare,
  title={e-CARE: a New Dataset for Exploring Explainable Causal Reasoning},
  author={Li Du et al. (2022)},
  year={2022},
  note={arXiv:2205.05849}
}
```

- arXiv: 2205.05849