Econlogicqa Eval

---
name: econlogicqa-eval
description: This benchmark evaluates large language models' ability to perform economic sequential reasoning by logically ordering interconnected business and supply chain events. It probes multi-event causality and temporal reasoning beyond simple chronological sorting, requiring models to understand complex economic narratives. Use when the user wants to benchmark on EconLogicQA, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.07938
  bibtex_key: quan2024econlogicqa
  confidence: high
---

# econlogicqa-eval

> EconLogicQA: A Question-Answering Benchmark for Evaluating Large Language Models in Economic Sequential Reasoning — Quan et al. (2024) (arXiv:2405.07938, 2024)

## What this evaluates

This benchmark evaluates large language models' ability to perform economic sequential reasoning by logically ordering interconnected business and supply chain events. It probes multi-event causality and temporal reasoning beyond simple chronological sorting, requiring models to understand complex economic narratives.

## Datasets

- **EconLogicQA** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Exact match accuracy between the model's predicted event sequence and the ground-truth sequence. Calculated as the number of correctly sequenced instances divided by the total number of instances.

## Input / output format

**Input**: A prompt containing 1 or 5 few-shot examples of economic event sequencing tasks, followed by the target question requiring the model to order a set of events.

**Output**: A sequentially ordered list of events. The model's raw text output is parsed using a regular expression to extract the final answer.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return (correct / len(golds)) * 100
```

## Common pitfalls

- 0-shot evaluation yields unsatisfactory results; the protocol explicitly requires 1-shot or 5-shot settings.
- Increasing the number of shots beyond 5 or applying Chain-of-Thought prompting in 1-shot settings can degrade performance or provide no benefit.
- Permuting the order of few-shot examples in the prompt significantly impacts models like GPT-4-Turbo, reducing accuracy in 1-shot scenarios.

## Evidence (verbatim from paper)

> Accuracy is the primary metric used, offering a direct measure of each model’s understanding of the concepts within the EconLogicQA dataset. ... Then, we extract the answer from the response generated by LLMs using regular expression. Finally, we verify the answer through exact matching and use accuracy as the evaluation metric.

## Citation

```bibtex
@misc{quan2024econlogicqa,
  title={EconLogicQA: A Question-Answering Benchmark for Evaluating Large Language Models in Economic Sequential Reasoning},
  author={Quan et al. (2024)},
  year={2024},
  note={arXiv:2405.07938}
}
```

- arXiv: 2405.07938