Decap Bbq Eval

---
name: decap-bbq-eval
description: Evaluates zero-shot question answering robustness against social biases, specifically testing how models adapt to ambiguous versus unambiguous contexts without relying on internal stereotypical knowledge. Use when the user wants to benchmark on BBQ, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.19426
  bibtex_key: bae2025decap
  confidence: medium
---

# decap-bbq-eval

> DeCAP: Context-Adaptive Prompt Generation for Debiasing Zero-shot Question Answering in Large Language Models — Bae et al. (2025) (arXiv:2503.19426, 2025)

## What this evaluates

Evaluates zero-shot question answering robustness against social biases, specifically testing how models adapt to ambiguous versus unambiguous contexts without relying on internal stereotypical knowledge.

## Datasets

- **BBQ** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly classified question types (ambiguous vs. unambiguous) on the BBQ dataset.

## Input / output format

**Input**: Context and question provided to the LLM, optionally augmented with a prefix instruction and few-shot neutral answer guidance demonstrations.

**Output**: A generated answer with reasoning, or a classification label (ambiguous/unambiguous) based on context similarity.

## Scoring recipe

```python
# Ambiguity detection scoring
answer_with_reason = llm.generate(context, question, prompt="answer with reason")
rouge_score = compute_rouge(context, answer_with_reason)
is_ambiguous = rouge_score < threshold
accuracy = mean(is_ambiguous == ground_truth_label)
```

## Common pitfalls

- Relying on LLM internal knowledge for ambiguity detection causes significant performance gaps between ambiguous and unambiguous questions.
- Fixed or self-reliant debiasing prompts degrade performance specifically in ambiguous contexts where context information is insufficient.

## Evidence (verbatim from paper)

> Table[2]: Experimental results for the question ambiguity detector on the BBQ dataset. The table shows the accuracy (%) in correctly classifying each question type. ... To quantify this similarity, in the second step, we calculate the ROUGE score*(Lin, [2004])* between the context and the answer combined with the question. If the score is lower than our threshold, the question is classified as ambiguous; if the score is higher, the question is classified as unambiguous.

## Citation

```bibtex
@misc{bae2025decap,
  title={DeCAP: Context-Adaptive Prompt Generation for Debiasing Zero-shot Question Answering in Large Language Models},
  author={Bae et al. (2025)},
  year={2025},
  note={arXiv:2503.19426}
}
```

- arXiv: 2503.19426