Iao Prompting Eval

---
name: iao-prompting-eval
description: Evaluates LLMs' ability to perform structured reasoning and knowledge application across arithmetic, logical, commonsense, and symbolic tasks using a template-based prompting framework. Use when the user wants to benchmark on GSM8K, AQuA, Date Understanding, Object Tracking, StrategyQA, CommonsenseQA, Last Letter, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.03080
  bibtex_key: diallo2025iaoprompting
  confidence: high
---

# iao-prompting-eval

> IAO Prompting: Making Knowledge Flow Explicit in LLMs through Structured Reasoning Templates — Diallo et al. (2025) (arXiv:2502.03080, 2025)

## What this evaluates

Evaluates LLMs' ability to perform structured reasoning and knowledge application across arithmetic, logical, commonsense, and symbolic tasks using a template-based prompting framework.

## Datasets

- **GSM8K** — total ?; splits: test (-1)
- **AQuA** — total ?; splits: test (-1)
- **Date Understanding** — total ?; splits: test (-1)
- **Object Tracking** — total ?; splits: test (-1)
- **StrategyQA** — total ?; splits: test (-1)
- **CommonsenseQA** — total ?; splits: test (-1)
- **Last Letter** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions, calculated as (number of correct predictions / total number of instances) * 100.

## Input / output format

**Input**: Natural language question or problem statement, optionally preceded by a structured reasoning template (Input-Action-Output steps) in a zero-shot setting.

**Output**: A single extracted answer or final solution, obtained via a dedicated answer extraction prompt in a single API call.

## Scoring recipe

```python
def calculate_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Evaluations are strictly zero-shot; any few-shot demonstrations violate the protocol.
- All baselines and IAO must use the identical answer extraction prompt in a single stage to ensure fair comparison.
- Model scale significantly impacts template adherence; results vary notably between PALM-2 and GPT-4.

## Evidence (verbatim from paper)

> Table 1: Evaluation results for PALM-2. Bold denotes best result. All methods use the same answer extraction prompt in a single stage for fair comparison. All methods are evaluated under the zero-shot setting.

## Citation

```bibtex
@misc{diallo2025iaoprompting,
  title={IAO Prompting: Making Knowledge Flow Explicit in LLMs through Structured Reasoning Templates},
  author={Diallo et al. (2025)},
  year={2025},
  note={arXiv:2502.03080}
}
```

- arXiv: 2502.03080