Xrl Bench Eval

---
name: xrl-bench-eval
description: Evaluates the fidelity and stability of state-explaining methods in Reinforcement Learning across tabular and image-based environments. It measures how accurately explanations identify critical states and how consistently they perform under perturbations. Use when the user wants to benchmark on XRL-Bench Environments (DunkCityDynasty-v1, LunarLander-v2, CartPole-v0, FlappyBird-v0, Breakout-v0, Pong-v0), or asks about evaluating this task. Reports AIM.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.12685
  bibtex_key: xiong2024xrlbench
  confidence: high
---

# xrl-bench-eval

> XRL-Bench: A Benchmark for Evaluating and Comparing Explainable Reinforcement Learning Techniques — Xiong et al. (2024) (arXiv:2402.12685, 2024)

## What this evaluates

Evaluates the fidelity and stability of state-explaining methods in Reinforcement Learning across tabular and image-based environments. It measures how accurately explanations identify critical states and how consistently they perform under perturbations.

## Datasets

- **XRL-Bench Environments (DunkCityDynasty-v1, LunarLander-v2, CartPole-v0, FlappyBird-v0, Breakout-v0, Pong-v0)** — total ?; splits: evaluation (-1); repo https://github.com/fuxiAIlab/xrl-bench

## Metrics

- `AIM` **(primary)** — range: [0, 1] | other
  - Area Under the Importance curve; computed by zero-padding the top-K most important states and measuring the drop in policy performance. Lower values indicate higher fidelity.
- `AUM` — range: [0, 1] | other
  - Area Under the Masking curve; computed by zero-padding the bottom-K least important states. Higher values indicate higher fidelity.
- `PGI` — range: [0, 1] | other
  - Prediction Gain for Important states; measures the difference in target action probability when important states are perturbed. Higher is better.
- `PGU` — range: [0, 1] | other
  - Prediction Gain for Unimportant states; measures the difference in target action probability when unimportant states are perturbed. Lower is better.
- `RIS` — range: [0, 1] | other
  - Residual Importance Score; measures the stability of explanation scores across multiple runs. Lower indicates higher stability.

## Input / output format

**Input**: State observations (tabular vectors or image frames) from RL environments, passed to a trained policy network to obtain action probabilities, alongside explainer-generated state importance scores.

**Output**: Scalar fidelity and stability metric values (AIM, AUM, PGI, PGU, RIS) aggregated over K-sized state subsets and environments.

## Scoring recipe

```python
def compute_fidelity(importance_scores, state, policy, K_range, mask_type='top'):
    indices = np.argsort(importance_scores)
    if mask_type == 'top': indices = indices[-K_range:]
    else: indices = indices[:K_range]
    auc_values = []
    for K in K_range:
        masked_state = state.copy()
        masked_state[indices[:K]] = 0
        pred_diff = abs(policy(state) - policy(masked_state))
        auc_values.append(pred_diff)
    return np.trapz(auc_values)

# For PGI/PGU, compute prediction difference on target action only.
# For RIS, compute variance of importance scores across runs.
```

## Common pitfalls

- Top-K state selection can use absolute or original importance values; the paper reports the better of the two, which can mask method weaknesses.
- For image-based environments, computing AUC over all K values is computationally prohibitive; the benchmark approximates it using K mod 10.
- Perturbation-based explainers (SARFA, PS, LIME) lack a solid theoretical framework, leading to high variance and unstable fidelity scores compared to gradient-based methods.

## Evidence (verbatim from paper)

> The values of the four fidelity evaluation metrics were calculated based on the Area Under the Curve (AUC) over all values of K. For AIM and AUM, zero-padding was employed to mask the most and least important states.

## Citation

```bibtex
@misc{xiong2024xrlbench,
  title={XRL-Bench: A Benchmark for Evaluating and Comparing Explainable Reinforcement Learning Techniques},
  author={Xiong et al. (2024)},
  year={2024},
  note={arXiv:2402.12685}
}
```

- arXiv: 2402.12685