Protoscore Eval

---
name: protoscore-eval
description: This benchmark evaluates the quality of prototype-based explainable AI (XAI) methods for time series and image data. It probes how well generated prototypes capture model behavior and data structure across nine interpretability properties, including correctness, consistency, continuity, and latent space cohesion. Use when the user wants to benchmark on ECG200, or asks about evaluating this task. Reports Total.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.08361
  bibtex_key: monke2025protoscore
  confidence: high
---

# protoscore-eval

> From Confusion to Clarity: ProtoScore - A Framework for Evaluating Prototype-Based XAI — Monke et al. (2025) (arXiv:2511.08361, 2025)

## What this evaluates

This benchmark evaluates the quality of prototype-based explainable AI (XAI) methods for time series and image data. It probes how well generated prototypes capture model behavior and data structure across nine interpretability properties, including correctness, consistency, continuity, and latent space cohesion.

## Datasets

- **ECG200** — total ?; splits: test (-1)

## Metrics

- `Total` **(primary)** — range: [0, 1]
  - Arithmetic mean of the nine individual prototype quality properties (Correctness, Consistency, Continuity, Contrastivity, Covariate Complexity, Compactness, Confidence, Input Completeness, Cohesion of Latent Space), each scored on a [0, 1] scale where 0 indicates poor quality and 1 indicates highest quality.

## Input / output format

**Input**: Trained prototype-based model, its generated prototypes, and the target dataset instances (time series or images).

**Output**: A dictionary containing scores for nine interpretability properties and a final Total score, each ranging from 0 to 1.

## Scoring recipe

```python
def compute_protoscore(prototypes, dataset):
    scores = {}
    for prop in ["CR", "CS", "CN", "CT", "CC", "CP", "CF", "IC", "CLS"]:
        scores[prop] = evaluate_property(prop, prototypes, dataset) # Returns 0.0-1.0
    scores["Total"] = sum(scores.values()) / len(scores)
    return scores
```

## Common pitfalls

- High validation loss does not necessarily indicate poor prototype quality; predictive performance and explanation quality are often decoupled.
- Dimensionality reduction techniques like UMAP distort true distances, so visual latent space plots should not be used for quantitative metric comparisons.
- Trade-offs exist between properties (e.g., high Contrastivity reduces Continuity), so optimizing a single metric may degrade others.

## Evidence (verbatim from paper)

> The total score averages the individual scores with equal weighting to summarize the prototype quality. Alongside, the mean squared error (MSE) validation loss for each model is given, which provides context for the model performance. The methods are sorted by descending validation loss. Note that a score of 0 indicates poor quality of the assessed prototypes, and a score of 1 represents the highest quality possible.

## Citation

```bibtex
@misc{monke2025protoscore,
  title={From Confusion to Clarity: ProtoScore - A Framework for Evaluating Prototype-Based XAI},
  author={Monke et al. (2025)},
  year={2025},
  note={arXiv:2511.08361}
}
```

- arXiv: 2511.08361