Cfkgr Eval

---
name: cfkgr-eval
description: Evaluates a model's ability to perform counterfactual reasoning on knowledge graphs by determining whether a target triple remains plausible after a hypothetical scenario is introduced, while also assessing knowledge retention for unaffected facts. Use when the user wants to benchmark on CFKGR-CoDEx-S, CFKGR-CoDEx-M, CFKGR-CoDEx-L, CFKGR-CoDEx-M*, or asks about evaluating this task. Reports Overall F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.06936
  bibtex_key: zellinger2024counterfactual
  confidence: high
---

# cfkgr-eval

> Counterfactual Reasoning with Knowledge Graph Embeddings — Zellinger et al. (2024) (arXiv:2403.06936, 2024)

## What this evaluates

Evaluates a model's ability to perform counterfactual reasoning on knowledge graphs by determining whether a target triple remains plausible after a hypothetical scenario is introduced, while also assessing knowledge retention for unaffected facts.

## Datasets

- **CFKGR-CoDEx-S** — total ?; splits: test (-1); repo https://github.com/LenaZellinger/counterfactual_KGR
- **CFKGR-CoDEx-M** — total ?; splits: test (-1); repo https://github.com/LenaZellinger/counterfactual_KGR
- **CFKGR-CoDEx-L** — total ?; splits: test (-1); repo https://github.com/LenaZellinger/counterfactual_KGR
- **CFKGR-CoDEx-M*** — total 1159; splits: test (1159); repo https://github.com/LenaZellinger/counterfactual_KGR

## Metrics

- `Overall F1-score` **(primary)** — range: [0, 100]
  - Harmonic mean of precision and recall across all test instances. The paper reports this metric as a percentage (0-100).
- `F1 (E)` — range: [0, 100]
  - F1 score computed against expected (rule-mined) labels for changed and unchanged facts.
- `F1 (H)` — range: [0, 100]
  - F1 score computed against human-annotated labels for changed and unchanged facts.
- `Rule-wise accuracy` — range: [0, 100]
  - Average accuracy per logical inference rule used to generate test triples, filtered to rules covering at least five test instances.

## Input / output format

**Input**: A knowledge graph context containing a hypothetical scenario (e.g., a modified fact or added edge) and a target triple to evaluate. For LLMs, formatted as a natural language prompt asking if the triple is still plausible given the scenario.

**Output**: Binary classification label (plausible/true or false/untrue) or a continuous score used with relation-specific decision thresholds.

## Scoring recipe

```python
def compute_f1(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return f1 * 100  # Paper reports as percentage
```

## Common pitfalls

- Models may ignore the hypothetical scenario and simply predict based on the original knowledge graph facts (factual reasoning instead of counterfactual).
- ChatGPT often misinterprets the task by attempting to infer the target triple from the context rather than evaluating whether it still holds given the hypothetical change.
- Threshold tuning for CoDEx-L lacks hard negative triples, leading to suboptimal decision boundaries and lower reported performance.

## Evidence (verbatim from paper)

> First, we evaluate pre-trained KGEs, COULDD, and ChatGPT on our CFKGR datasets with expected labels to assess whether the methods can apply inference rules found by a rule mining system in hypothetical scenarios. In our second set of experiments, we evaluate on human-labeled data to check whether the methods also capture human reasoning... COULDD achieves the best results in terms of overall F1-score on all datasets.

## Citation

```bibtex
@misc{zellinger2024counterfactual,
  title={Counterfactual Reasoning with Knowledge Graph Embeddings},
  author={Zellinger et al. (2024)},
  year={2024},
  note={arXiv:2403.06936}
}
```

- arXiv: 2403.06936