Fairness Explanation Eval

---
name: fairness-explanation-eval
description: This benchmark evaluates the faithfulness and utility of counterfactual explanations in recommendation systems. It measures how effectively generated explanations identify fairness-disparaging attributes by iteratively erasing them and observing the resulting impact on recommendation accuracy and item exposure inequality. Use when the user wants to benchmark on Yelp, Douban Movie, Last-FM, or asks about evaluating this task. Reports NDCG@K.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2307.04386
  bibtex_key: wang2023counterfactual
  confidence: high
---

# fairness-explanation-eval

> Counterfactual Explanation for Fairness in Recommendation — Xiangmeng Wang et al. (2023) (arXiv:2307.04386, 2023)

## What this evaluates

This benchmark evaluates the faithfulness and utility of counterfactual explanations in recommendation systems. It measures how effectively generated explanations identify fairness-disparaging attributes by iteratively erasing them and observing the resulting impact on recommendation accuracy and item exposure inequality.

## Datasets

- **Yelp** — total 198397; splits: train (-1), val (-1), test (-1)
- **Douban Movie** — total 1068278; splits: train (-1), val (-1), test (-1)
- **Last-FM** — total 92834; splits: train (-1), val (-1), test (-1)

## Metrics

- `NDCG@K` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at cutoff K. Measures the quality of the ranked recommendation list by weighting relevance scores by their logarithmic position.
- `Hit Ratio (HR)@K` — range: [0, 1]
  - Binary metric indicating whether at least one relevant item appears in the top-K recommendation list.
- `Head-tailed Rate (HT)@K` — range: [0, 1]
  - Ratio of head-tailed (popular) item count to the list length K. Lower values indicate reduced popularity bias.
- `Gini@K` — range: [0, 1]
  - Gini coefficient measuring inequality between head-tailed and long-tailed items within the top-K recommendation list. Lower values indicate fairer exposure distribution.

## Input / output format

**Input**: Historical user-item interaction logs, user attributes, and item attributes from heterogeneous information networks, processed into fused user and item embeddings.

**Output**: Top-K ranked item lists and a set of attribute-level counterfactual explanations (specific user/item attributes to remove for fairness diagnostics).

## Scoring recipe

```python
def evaluate_erasure(model, explanations, K, E, ground_truth):
    # explanations: list of attribute sets per user
    # E: number of top attributes to erase from explanation
    erased_attrs = explanations[:E]
    # Remove erased attributes from attribute space
    new_emb = fuse_embeddings(model.user_emb, model.item_emb, exclude=erased_attrs)
    # Generate Top-K recommendations
    recs = generate_topk(new_emb, K)
    # Compute metrics
    ndcg = calc_ndcg(recs, ground_truth, K)
    hr = calc_hr(recs, ground_truth, K)
    ht = calc_head_tailed_rate(recs, K)
    gini = calc_gini_coefficient(recs, K)
    return ndcg, hr, ht, gini
```

## Common pitfalls

- HT@K and Gini@K are inverse fairness metrics: lower values indicate better fairness, opposite to NDCG@K and HR@K.
- The evaluation requires a chronological split and a 10-core filter; skipping these preprocessing steps invalidates dataset density and temporal fairness baselines.
- E (erasure length) and K (ranking cutoff) are distinct hyperparameters; results must be reported for both E∈{5,10,20} and K∈{5,20,40}.

## Evidence (verbatim from paper)

> Given the recommendation results at each evaluation point, we use Normalized Discounted Cumulative Gain (NDCG)@K and Hit Ratio (HR)@K to measure the recommendation performance. As this work focuses on item exposure fairness in recommendations, we use two wildly-adopted item-side evaluation metrics, i.e., Head-tailed Rate (HT)@K and Gini@K, for fairness evaluation.

## Citation

```bibtex
@misc{wang2023counterfactual,
  title={Counterfactual Explanation for Fairness in Recommendation},
  author={Xiangmeng Wang et al. (2023)},
  year={2023},
  note={arXiv:2307.04386}
}
```

- arXiv: 2307.04386