Groundcocoa Eval

---
name: groundcocoa-eval
description: Evaluates compositional and conditional reasoning in LLMs by requiring them to match complex, logically constrained user preferences to specific flight booking options. It probes the model's ability to handle interdependent requirements and atypical constraints without external reasoning engines. Use when the user wants to benchmark on GroundCocoa, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.04237
  bibtex_key: kohli2024groundcocoa
  confidence: high
---

# groundcocoa-eval

> GroundCocoa: A Benchmark for Evaluating Compositional & Conditional Reasoning in Language Models — Kohli et al. (2024) (arXiv:2404.04237, 2024)

## What this evaluates

Evaluates compositional and conditional reasoning in LLMs by requiring them to match complex, logically constrained user preferences to specific flight booking options. It probes the model's ability to handle interdependent requirements and atypical constraints without external reasoning engines.

## Datasets

- **GroundCocoa** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly matched flight options to user requirements. Calculated as (correct predictions / total samples) * 100.

## Input / output format

**Input**: A user requirement encoded as logically constrained, interdependent preferences, followed by 5 flight options.

**Output**: Selection of the flight option(s) that satisfy the user requirement.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
accuracy = (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Context length limits can degrade CoT performance; CoT-partial (2 options) often outperforms CoT-full (5 options) due to confusion between requirements and options.
- Least-to-Most prompting scales turns linearly with complexity, leading to high inference costs and potential error accumulation.
- Models show strong bias toward typical user needs, causing significant accuracy drops when atypical/unconventional constraints are introduced.

## Evidence (verbatim from paper)

> To measure performance on GroundCocoa, we test several models of different sizes including both open-source and closed-source LLMs - LLAMA 2-chat (Touvron et al., [2023]) / LLAMA 3-Instruct (Dubey et al., [2024]), Mixtral 8x7B - Instruct (Jiang et al., [2024]) / Mistral 7B Instruct (Jiang et al., [2023]), Gemini Pro (Team et al., [2023]), and GPT-4 Turbo. Results from our experiments are shown in Table[3]. We have 3 different evaluation setups for the our models - direct prompting, chain-of-thought (CoT) (Wei et al., [2022]) prompting, and least-to-most (L2M) prompting (Zhou et al. ([2023])). Table 3: Accuracy (%) on GroundCocoa.

## Citation

```bibtex
@misc{kohli2024groundcocoa,
  title={GroundCocoa: A Benchmark for Evaluating Compositional & Conditional Reasoning in Language Models},
  author={Kohli et al. (2024)},
  year={2024},
  note={arXiv:2404.04237}
}
```

- arXiv: 2404.04237