Ipqa Eval

---
name: ipqa-eval
description: This benchmark evaluates a model's ability to identify core user intents in personalized question answering. It probes whether systems can infer prioritized motivations from a user's historical Q&A interactions and a target question's narrative, rather than relying on explicit user statements. Use when the user wants to benchmark on IPQA, or asks about evaluating this task. Reports IPQA-Eval F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.23536
  bibtex_key: kim2025ipqa
  confidence: high
---

# ipqa-eval

> IPQA: A Benchmark for Core Intent Identification in Personalized Question Answering — Jieyong Kim et al. (2025) (arXiv:2510.23536, 2025)

## What this evaluates

This benchmark evaluates a model's ability to identify core user intents in personalized question answering. It probes whether systems can infer prioritized motivations from a user's historical Q&A interactions and a target question's narrative, rather than relying on explicit user statements.

## Datasets

- **IPQA** — total 7730; splits: train (4655), val (1530), test (1545)

## Metrics

- `IPQA-Eval F1` **(primary)** — range: [0, 1]
  - Precision, recall, and F1 are computed based on LLM-based alignment between predicted core intents and ground truth intents. Each predicted intent is matched to the best-aligned ground truth intent. A ground truth intent is counted as correctly identified only once. Precision = |correctly identified GT intents| / |predicted intents|, Recall = |correctly identified GT intents| / |GT intents|, F1 = 2 * (Precision * Recall) / (Precision + Recall).

## Input / output format

**Input**: A target question q and a chronological user profile P_u consisting of historical pairs of questions and source information (narrative + selected answer).

**Output**: A set of predicted core intents, where each intent comprises an intent name, a description, and a reference text excerpt.

## Scoring recipe

```python
def ipqa_eval(predicted_intents, ground_truth_intents):
    matched_gt = set()
    for pred in predicted_intents:
        best_match = llm_find_best_alignment(pred, ground_truth_intents)
        if best_match is not None:
            matched_gt.add(best_match)
    tp = len(matched_gt)
    precision = tp / len(predicted_intents) if predicted_intents else 0
    recall = tp / len(ground_truth_intents) if ground_truth_intents else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return precision, recall, f1
```

## Common pitfalls

- One-to-many comparison reduces computational cost but may cause multiple predicted intents to map to the same ground truth intent, which only counts once for recall.
- LLM-based matching introduces potential bias or inconsistency compared to exact string matching, requiring careful prompt design for the evaluator.
- Intents are multi-component (name, description, reference), so evaluators must align on semantic meaning rather than exact text.

## Evidence (verbatim from paper)

> Based on these alignments, we compute precision, recall, and F1 to measure intent identification performance.

## Citation

```bibtex
@misc{kim2025ipqa,
  title={IPQA: A Benchmark for Core Intent Identification in Personalized Question Answering},
  author={Jieyong Kim et al. (2025)},
  year={2025},
  note={arXiv:2510.23536}
}
```

- arXiv: 2510.23536