Cogs Eval

---
name: cogs-eval
description: Probes compositional generalization in semantic parsing by evaluating whether models can correctly map out-of-distribution natural language sentences to their corresponding lambda calculus semantic representations. It specifically tests structural generalizations like argument role reversal, depth generalization, and voice transformation, as well as lexical generalizations. Use when the user wants to benchmark on COGS, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2010.05465
  bibtex_key: kim2020cogs
  confidence: high
---

# cogs-eval

> COGS: A Compositional Generalization Challenge Based on Semantic Interpretation — Kim et al. (2020) (arXiv:2010.05465, 2020)

## What this evaluates

Probes compositional generalization in semantic parsing by evaluating whether models can correctly map out-of-distribution natural language sentences to their corresponding lambda calculus semantic representations. It specifically tests structural generalizations like argument role reversal, depth generalization, and voice transformation, as well as lexical generalizations.

## Datasets

- **COGS** — total ?; splits: train (-1), dev (-1), test (-1), gen (-1); repo https://github.com/najoungkim/COGS

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Exact-match accuracy: an output sequence is considered correct only if it exactly matches the gold sequence.

## Input / output format

**Input**: Natural language sentence

**Output**: Lambda calculus semantic representation with index-based labels

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = sum(1 for pred, gold in zip(predictions, golds) if pred == gold)
    return correct / len(golds)
```

## Common pitfalls

- Models may produce structurally valid but token-mismatched outputs (e.g., wrong indices or extra tokens), which are strictly counted as incorrect under exact-match scoring.
- Performance exhibits high variance across random seeds; evaluating a single run is unreliable and the protocol mandates averaging over five runs.
- Depth generalization failures often stem from early stopping or excessive output length rather than pure compositional inability, complicating error attribution.

## Evidence (verbatim from paper)

> All architectures performed well on the development and test sets (Table 2), with little variability across runs (Figure 2a, green dots). By contrast, generalization accuracy was low across the board, and was characterized by much higher variance (blue dots). An output sequence is considered correct only if it exactly matches the gold sequence.

## Citation

```bibtex
@misc{kim2020cogs,
  title={COGS: A Compositional Generalization Challenge Based on Semantic Interpretation},
  author={Kim et al. (2020)},
  year={2020},
  note={arXiv:2010.05465}
}
```

- arXiv: 2010.05465