Countereval Eval

---
name: countereval-eval
description: Evaluates the human-centric quality of counterfactual explanations across eight explanatory virtues. It probes how well explanations convey desired outcomes, remain feasible, consistent, complete, trustworthy, understandable, fair, and appropriately complex. Use when the user wants to benchmark on CounterEval, or asks about evaluating this task. Reports Overall Satisfaction, Feasibility, Consistency, Completeness, Trust, Understandability, Fairness, Complexity.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.21131
  bibtex_key: [REDACTED]
  confidence: high
---

# countereval-eval

> Towards Unifying Evaluation of Counterfactual Explanations: Leveraging Large Language Models for Human-Centric Assessments — Domnich et al. (2024) (arXiv:2410.21131, 2024)

## What this evaluates

Evaluates the human-centric quality of counterfactual explanations across eight explanatory virtues. It probes how well explanations convey desired outcomes, remain feasible, consistent, complete, trustworthy, understandable, fair, and appropriately complex.

## Datasets

- **CounterEval** — total 30; splits: full (30); HF `anitera/CounterEval`; repo https://github.com/anitera/CounterEval

## Metrics

- `Overall Satisfaction, Feasibility, Consistency, Completeness, Trust, Understandability, Fairness, Complexity` **(primary)** — range: 1-6 (or -2 to 2 for Complexity)
  - Ordinal rating assigned by human evaluators on a 6-point scale (1-6), except Complexity on a 5-point scale (-2 to 2, where 0 is ideal). LLM performance is measured by exact-match accuracy in predicting these human-assigned scores.

## Input / output format

**Input**: Textual counterfactual explanation scenarios describing feature changes to achieve a different outcome, accompanied by contextual information and specific evaluation questions.

**Output**: Ordinal score from 1 to 6 for each of the 8 metrics (or -2 to 2 for Complexity), representing the evaluator's agreement or assessment of the explanation's quality on that dimension.

## Scoring recipe

```python
def compute_prediction_accuracy(predictions, gold):
    # predictions and gold are lists of human-assigned scores per instance
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(predictions)
```

## Common pitfalls

- Complexity uses a different scale (-2 to 2) where 0 is ideal, unlike the other 7 metrics which use a 1-6 scale.
- Human ratings are collected via survey and averaged per question; LLMs must predict these ordinal scores rather than continuous values.
- The dataset is small (30 instances) and designed to cover extreme variations across metrics, so results may not generalize to typical real-world explanations.

## Evidence (verbatim from paper)

> All metrics were rated on scales detailed in Table[1], with a 6-point ordinal scale from 1 (lowest) to 6 (highest) except for Complexity, rated on a 5-point scale from -2 (too simple) to 2 (too complex), where 0 corresponded to desired complexity.

## Citation

```bibtex
@misc{domnich2024countereval,
  title={Towards Unifying Evaluation of Counterfactual Explanations: Leveraging Large Language Models for Human-Centric Assessments},
  author={Domnich et al. (2024)},
  year={2024},
  note={arXiv:2410.21131}
}
```

- arXiv: 2410.21131