Trec Ikat 2023 Eval

---
name: trec-ikat-2023-eval
description: This benchmark evaluates a system's ability to perform personalized conversational search by retrieving relevant passages and generating fluent, grounded responses. It specifically probes how well an agent can adapt its output to user-specific context encoded in a Personal Text Knowledge Base (PTKB) while maintaining provenance traceability. Use when the user wants to benchmark on TREC iKAT 2023, ClueWeb22-B Subset, or asks about evaluating this task. Reports groundedness.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.01330
  bibtex_key: aliannejadi2024trecikat
  confidence: high
---

# trec-ikat-2023-eval

> TREC iKAT 2023: The Interactive Knowledge Assistance Track Overview — Aliannejadi et al. (2024) (arXiv:2401.01330, 2024)

## What this evaluates

This benchmark evaluates a system's ability to perform personalized conversational search by retrieving relevant passages and generating fluent, grounded responses. It specifically probes how well an agent can adapt its output to user-specific context encoded in a Personal Text Knowledge Base (PTKB) while maintaining provenance traceability.

## Datasets

- **TREC iKAT 2023** — total 36; splits: train (11), test (25)
- **ClueWeb22-B Subset** — total 116838987; splits: collection (116838987)

## Metrics

- `naturalness` — range: [0, 4]
  - 0-4 scale assessing if the response sounds human-like and fluent, ranging from 0 (No) to 4 (Yes).
- `groundedness` **(primary)** — range: [0, 4]
  - 0-4 scale assessing if the response appropriately references and connects to the provenance passages, ranging from 0 (No) to 4 (Yes).

## Input / output format

**Input**: User utterance, conversation history, Personal Text Knowledge Base (PTKB) statements, and a candidate passage collection.

**Output**: Ranked list of PTKB statements, ranked list of retrieved passages, and a generated response text.

## Scoring recipe

```python
def score(predictions, gold):
    response = predictions['response']
    passages = gold['provenance_passages']
    # GPT-4 evaluates based on track-specified rubrics
    nat_score = call_gpt4(f"Naturalness: Does the response sound human-like?\n{response}")
    grd_score = call_gpt4(f"Groundedness: Does the response reference the passages?\n{response}\nPassages: {passages}")
    return {'naturalness': nat_score, 'groundedness': grd_score}
```

## Common pitfalls

- Evaluating on generic or clarification turns that were explicitly pruned from the assessment pool.
- Assuming traditional IR metrics (e.g., nDCG) are the primary evaluation method, whereas response quality relies on GPT-4 automated scoring.
- Failing to provide provenance passages when requesting groundedness scores, as GPT-4 requires them to assess factual alignment.

## Evidence (verbatim from paper)

> Each response was then evaluated from two perspectives: groundedness and naturalness. The criteria and the definitions we provided to GPT-4 for the assessment of each were as follows:

Naturalness: Does the response sound human-like?

• 0. No - The response does not sound like something a human would say given the conversation.

• 1. Somewhat - Parts of the response can be generated by human, but it is overall not fluent.

• 2. Slightly natural - The response is almost human-like. The response is well-formed but is not natural.

• 3. Yes (but not completely) - The response is almost human-like. The response is well-formed and natural in most parts but has some parts that are not natural.

• 3. Yes - The response is perfectly human-like and fluent.

Groundedness: Does the response appropriately reference or connect to the information provided in the provenance passages?

## Citation

```bibtex
@misc{aliannejadi2024trecikat,
  title={TREC iKAT 2023: The Interactive Knowledge Assistance Track Overview},
  author={Aliannejadi et al. (2024)},
  year={2024},
  note={arXiv:2401.01330}
}
```

- arXiv: 2401.01330