Psiloqa Eval

---
name: psiloqa-eval
description: Evaluates the ability of models to detect span-level hallucinations in multilingual question-answering contexts. It probes cross-lingual generalization and token-level inconsistency detection between generated answers and ground truth. Use when the user wants to benchmark on PsiloQA, or asks about evaluating this task. Reports IoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.04849
  bibtex_key: rykov2025psiloqa
  confidence: high
---

# psiloqa-eval

> When Models Lie, We Learn: Multilingual Span-Level Hallucination Detection with PsiloQA — Rykov et al. (2025) (arXiv:2510.04849, 2025)

## What this evaluates

Evaluates the ability of models to detect span-level hallucinations in multilingual question-answering contexts. It probes cross-lingual generalization and token-level inconsistency detection between generated answers and ground truth.

## Datasets

- **PsiloQA** — total ?; splits: train (-1), test (-1); repo https://github.com/s-nlp/psiloqa

## Metrics

- `IoU` **(primary)** — range: percent
  - Intersection over Union at the character level. Calculated as the size of the intersection divided by the size of the union between the set of binarized character-level gold annotations and the set of characters predicted as hallucinated.
- `AP` — range: percent
  - Average Precision for ranking-based character-level evaluation. Computed as the area under the precision-recall curve: sum of precision at each cut-off multiplied by the change in recall between consecutive items.

## Input / output format

**Input**: Context-question-answer triples processed for token-level classification.

**Output**: Binary token-level labels indicating whether each character is hallucinated, or continuous confidence scores for ranking.

## Scoring recipe

```python
def compute_iou(pred_chars, gold_chars):
    intersection = len(pred_chars & gold_chars)
    union = len(pred_chars | gold_chars)
    return intersection / union if union > 0 else 0.0

def compute_ap(scores, gold_mask):
    ranks = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)
    tp, fp = 0, 0
    precisions, recalls = [], []
    for i in ranks:
        if gold_mask[i]: tp += 1
        else: fp += 1
        precisions.append(tp / (tp + fp))
        recalls.append(tp / sum(gold_mask))
    return sum(p * (recalls[i] - recalls[i-1]) for i, p in enumerate(precisions))
```

## Common pitfalls

- IoU requires language-specific threshold calibration on the validation set rather than a global threshold.
- AP measures ranking quality at the character level, not span-level accuracy; confusing the two granularities leads to misinterpretation.
- LLM-based baselines like FActScore must be adapted from sentence-level to token-level by computing scores based on unsupported claim frequency.

## Evidence (verbatim from paper)

> Due to the complex nature of hallucination detection, we employ a dual-level evaluation approach combining span-level and character-level assessment. As with Mu-SHROOM, we selected the intersection over union (IoU) metric to evaluate span-level hallucination detection. Additionally, we use average precision (AP) for ranking-based evaluation on character-level.

## Citation

```bibtex
@misc{rykov2025psiloqa,
  title={When Models Lie, We Learn: Multilingual Span-Level Hallucination Detection with PsiloQA},
  author={Rykov et al. (2025)},
  year={2025},
  note={arXiv:2510.04849}
}
```

- arXiv: 2510.04849