Cfq Eval

---
name: cfq-eval
description: This benchmark evaluates compositional generalization in semantic parsing by measuring how well models translate anonymized natural language questions into executable SPARQL queries. It specifically probes the ability to generalize to unseen combinations of logical rules (compounds) while maintaining familiarity with individual rules (atoms), using Maximum Compound Divergence splits to ensure fair yet challenging evaluation. Use when the user wants to benchmark on CFQ, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2012.08266
  bibtex_key: tsarkov2020cfq
  confidence: medium
---

# cfq-eval

> *-CFQ: Analyzing the Scalability of Machine Learning on a Compositional Task — Tsarkov et al. (arXiv:2012.08266, 2020)

## What this evaluates

This benchmark evaluates compositional generalization in semantic parsing by measuring how well models translate anonymized natural language questions into executable SPARQL queries. It specifically probes the ability to generalize to unseen combinations of logical rules (compounds) while maintaining familiarity with individual rules (atoms), using Maximum Compound Divergence splits to ensure fair yet challenging evaluation.

## Datasets

- **CFQ** — total ?; splits: MCD (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard classification accuracy: the proportion of correctly predicted SPARQL queries out of the total number of test instances.

## Input / output format

**Input**: Anonymized natural language question

**Output**: A SPARQL query string

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred.strip() == gold.strip():
        correct += 1
return correct / len(gold_labels)
```

## Common pitfalls

- Evaluating on splits with high atom divergence rather than Maximum Compound Divergence (MCD) splits fails to isolate compositional generalization.
- Models may exploit entity memorization if anonymization is not strictly enforced, leading to inflated accuracy on unseen compositions.

## Evidence (verbatim from paper)

> The authors release a number of MCD splits for CFQ, and show that there is a strong negative correlation between the accuracy of three standard sequence-to-sequence architectures and the compound divergence.

## Citation

```bibtex
@misc{tsarkov2020cfq,
  title={*-CFQ: Analyzing the Scalability of Machine Learning on a Compositional Task},
  author={Tsarkov et al.},
  year={2020},
  note={arXiv:2012.08266}
}
```

- arXiv: 2012.08266