Context Conflict Merge Eval

---
name: context-conflict-merge-eval
description: Evaluates how language models merge conflicting generated and retrieved contexts in open-domain QA. It probes whether models exhibit a systematic bias toward generated contexts over retrieved ones when only one context contains the correct answer. Use when the user wants to benchmark on NQ-CC, TQA-CC, or asks about evaluating this task. Reports DiffGR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.11911
  bibtex_key: tan2024blinded
  confidence: high
---

# context-conflict-merge-eval

> Blinded by Generated Contexts: How Language Models Merge Generated and Retrieved Contexts When Knowledge Conflicts? — Hexiang Tan et al. (2024) (arXiv:2401.11911, 2024)

## What this evaluates

Evaluates how language models merge conflicting generated and retrieved contexts in open-domain QA. It probes whether models exhibit a systematic bias toward generated contexts over retrieved ones when only one context contains the correct answer.

## Datasets

- **NQ-CC** — total ?; splits: test (-1), dev (-1); repo https://github.com/Tan-Hexiang/RetrieveOrGenerated
- **TQA-CC** — total ?; splits: test (-1), dev (-1); repo https://github.com/Tan-Hexiang/RetrieveOrGenerated

## Metrics

- `DiffGR` **(primary)** — range: [-1, 1]
  - DiffGR = (ρ_gen - ρ_ret) / (ρ_gen + ρ_ret), where ρ_gen = avg(exact_match(model_answer, generated_only_answer)) and ρ_ret = avg(exact_match(model_answer, retrieved_only_answer)). Ranges from [-1, 1].

## Input / output format

**Input**: Question q, retrieved context d_1^γ, and generated context d_1^ϱ provided together as input to the LLM.

**Output**: A single generated answer string a_φ.

## Scoring recipe

```python
def compute_diffgr(predictions, gold_gen, gold_ret):
    em_gen = [1 if p == g else 0 for p, g in zip(predictions, gold_gen)]
    em_ret = [1 if p == g else 0 for p, g in zip(predictions, gold_ret)]
    rho_gen = sum(em_gen) / len(em_gen)
    rho_ret = sum(em_ret) / len(em_ret)
    if rho_gen + rho_ret == 0:
        return 0.0
    return (rho_gen - rho_ret) / (rho_gen + rho_ret)
```

## Common pitfalls

- Length discrepancy between generated and retrieved contexts can bias merging; the authors enforce a strict length constraint (<3% difference) to isolate semantic/conflict effects.
- Parametric knowledge may still influence answers despite traceability filtering; the paper notes complete elimination is challenging but shows negligible impact on conclusions.
- The metric assumes the model selects one context; instances where the model answers correctly from neither context are considered negligible but can skew ρ values if frequent.

## Evidence (verbatim from paper)

> To facilitate a simple and efficient experiment, we define a synthesized metric as follows: DiffGR = (ρ_gen - ρ_ret) / (ρ_gen + ρ_ret). The metric DiffGR, ranging from [-1,1], quantifies the extent of LLMs’ tendency to rely on generated contexts over retrieved contexts.

## Citation

```bibtex
@misc{tan2024blinded,
  title={Blinded by Generated Contexts: How Language Models Merge Generated and Retrieved Contexts When Knowledge Conflicts?},
  author={Hexiang Tan et al. (2024)},
  year={2024},
  note={arXiv:2401.11911}
}
```

- arXiv: 2401.11911