Qrecc Attribution Fluency Eval

---
name: qrecc-attribution-fluency-eval
description: Evaluates the tradeoff between response fluency and factual attribution in retrieval-augmented conversational LLMs. It measures how well models generate coherent, context-aware responses while correctly grounding answers in provided evidence or dialog history. Use when the user wants to benchmark on QReCC, or asks about evaluating this task. Reports Auto-AIS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2302.05578
  bibtex_key: aksitov2023characterizing
  confidence: medium
---

# qrecc-attribution-fluency-eval

> Characterizing Attribution and Fluency Tradeoffs for Retrieval-Augmented Large Language Models — Aksitov et al. (2023) (arXiv:2302.05578, 2023)

## What this evaluates

Evaluates the tradeoff between response fluency and factual attribution in retrieval-augmented conversational LLMs. It measures how well models generate coherent, context-aware responses while correctly grounding answers in provided evidence or dialog history.

## Datasets

- **QReCC** — total ?; splits: test (-1)

## Metrics

- `Auto-AIS` **(primary)** — range: [0, 1]
  - Automated scoring metric for attribution accuracy. Exact formula not provided in the experimental setup; likely measures alignment between generated response and provided evidence/facts.
- `Auto-SSA` — range: [0, 1]
  - Automated scoring metric for response fluency and coherence. Exact formula not provided in the experimental setup; validated against human ratings in a pilot study.

## Input / output format

**Input**: Conversational turns formatted as 'turn_index parent_index speaker_id utterance [eot]', optionally prefixed with 'Instructions:' and 'Fact:' sections containing retrieved, golden, absent, or non-evidence text.

**Output**: Model-generated text completion, parsed by stopping at the next '[eot]' token.

## Scoring recipe

```python
def evaluate(predictions, gold, metric_type):
    if metric_type == 'Auto-AIS':
        return compute_attribution_alignment(predictions, gold_evidence)
    elif metric_type == 'Auto-SSA':
        return compute_fluency_coherence(predictions, dialog_history)
    # Note: Exact implementation details for Auto-AIS and Auto-SSA are defined in the full paper, not the setup section.
```

## Common pitfalls

- Stopping generation at the wrong [eot] token can truncate valid responses or include prompt artifacts.
- Using non-evidence or absent facts without adjusting the prompt structure may artificially deflate attribution scores.
- Temperature 0.0 vs 0.7 significantly impacts fluency scores, making cross-setting comparisons sensitive to sampling parameters.

## Evidence (verbatim from paper)

> To better understand the impact of various components, we further adjust structure of the prompts along several dimensions: “instructions” could be either present or absent, similarly, dialog history could be also present or absent, and the provided evidence could be: golden, retrieved, absent, non-evidence... We arrive into a “full grid” of experiments by running the prompting setups described above in 6 settings each: 3 different model sizes (8B, 62B and 540B) and 2 sampling temperatures (0.0 and 0.7). As was mentioned in the previous section, conducting the full grid of experiments with human raters is unrealistic. Instead, after confirming that the auto-metrics are well-aligned with the human ratings from the pilot, we apply Auto-AIS and Auto-SSA to the full grid of experiments for further analysis.

## Citation

```bibtex
@misc{aksitov2023characterizing,
  title={Characterizing Attribution and Fluency Tradeoffs for Retrieval-Augmented Large Language Models},
  author={Aksitov et al. (2023)},
  year={2023},
  note={arXiv:2302.05578}
}
```

- arXiv: 2302.05578