Gem Cot Mixed Task Eval

---
name: gem-cot-mixed-task-eval
description: Evaluates the ability of LLMs to perform zero-shot and few-shot reasoning across a heterogeneous mix of unseen and known task types without manual task-specific prompting. It probes dynamic demonstration routing, clustering-based generalization, and streaming adaptation in mixed-task scenarios. Use when the user wants to benchmark on AQUA-RAT, MultiArith, AddSub, GSM8K, SingleEq, SVAMP, Last Letter Concatenation, Coin Flip, StrategyQA, CSQA, BIG-Bench Hard (BBH), or asks about evaluating this task. Reports Accuracy (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.06692
  bibtex_key: zou2023gemcot
  confidence: high
---

# gem-cot-mixed-task-eval

> Generalizable Chain-of-Thought Prompting in Mixed-task Scenarios with Large Language Models — Anni Zou et al. (2023) (arXiv:2310.06692, 2023)

## What this evaluates

Evaluates the ability of LLMs to perform zero-shot and few-shot reasoning across a heterogeneous mix of unseen and known task types without manual task-specific prompting. It probes dynamic demonstration routing, clustering-based generalization, and streaming adaptation in mixed-task scenarios.

## Datasets

- **AQUA-RAT** — total ?; splits: test (-1)
- **MultiArith** — total ?; splits: test (-1)
- **AddSub** — total ?; splits: test (-1)
- **GSM8K** — total ?; splits: test (-1)
- **SingleEq** — total ?; splits: test (-1)
- **SVAMP** — total ?; splits: test (-1)
- **Last Letter Concatenation** — total ?; splits: test (-1)
- **Coin Flip** — total ?; splits: test (-1)
- **StrategyQA** — total ?; splits: test (-1)
- **CSQA** — total ?; splits: test (-1)
- **BIG-Bench Hard (BBH)** — total 2000; splits: test (2000)

## Metrics

- `Accuracy (%)` **(primary)** — range: percent
  - Percentage of correctly predicted answers out of the total number of test instances. Calculated as (number of exact matches between model output and gold answer) / (total predictions) * 100.

## Input / output format

**Input**: A single natural language question or problem statement.

**Output**: A chain-of-thought reasoning trace followed by a final answer.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if normalize(p) == normalize(g))
    return (correct / len(golds)) * 100
```

## Common pitfalls

- The BBH evaluation uses a shuffled, randomly sampled subset of 2000 questions to simulate mixed-task scenarios, rather than the official full test set.
- The method dynamically updates its demonstration pool via density-based clustering during inference, so static few-shot baselines cannot be directly compared without accounting for this continuous learning aspect.

## Evidence (verbatim from paper)

> Table 3: Accuracy (%) on ten reasoning datasets. The backbone model is GPT-3.5-Turbo. Results in bold and underline are the best and second-best performances, respectively. ... For the BBH (Suzgun et al., 2022) tasks, we shuffle all the data and randomly sample 2000 questions to imitate the realistic mixed-task scenarios.

## Citation

```bibtex
@misc{zou2023gemcot,
  title={Generalizable Chain-of-Thought Prompting in Mixed-task Scenarios with Large Language Models},
  author={Anni Zou et al. (2023)},
  year={2023},
  note={arXiv:2310.06692}
}
```

- arXiv: 2310.06692