Portbench Eval

---
name: portbench-eval
description: Evaluates large language models' ability to perform quantitative reasoning and structured decision-making in financial portfolio optimization. It probes whether models can correctly apply convex optimization principles under varying constraints and multi-criteria objectives. Use when the user wants to benchmark on PortBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.09301
  bibtex_key: cho2026portbench
  confidence: high
---

# portbench-eval

> Constructing a Portfolio Optimization Benchmark Framework for Evaluating Large Language Models — Cho et al. (2026) (arXiv:2603.09301, 2026)

## What this evaluates

Evaluates large language models' ability to perform quantitative reasoning and structured decision-making in financial portfolio optimization. It probes whether models can correctly apply convex optimization principles under varying constraints and multi-criteria objectives.

## Datasets

- **PortBench** — total 9500; splits: test (9500); repo https://github.com/noahardyx/PortBench

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard classification accuracy: the proportion of correctly selected optimal portfolio choices out of the total number of questions.

## Input / output format

**Input**: A multiple-choice question describing a portfolio optimization scenario, including asset return/risk statistics, a specific investment objective (e.g., minimize variance, maximize Sharpe ratio), constraint settings (e.g., weight bounds, cardinality limits), and four candidate portfolio allocations (one optimal, three distractors).

**Output**: The model must output the label of the single portfolio choice that optimally satisfies the given objective and constraints.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().upper() == gold.strip().upper():
        correct += 1
return correct / len(gold_labels)
```

## Common pitfalls

- Models may exploit superficial heuristics (e.g., picking highest return) rather than solving the underlying convex optimization, especially when distractors are highly similar.
- Performance on complex objectives like Sharpe ratio maximization drops below 10% because models struggle to integrate multidimensional risk-return trade-offs and constraint handling simultaneously.
- Evaluators should not assume high accuracy on unconstrained problems generalizes to constrained settings, as restrictive bounds significantly widen performance gaps between models.

## Evidence (verbatim from paper)

> Figure 4 compares the accuracy of LLMs across the five investment objectives. The results show that GPT attains the highest accuracy for risk-based objectives such as minimizing variance and MDD. This outcome suggests that GPT demonstrates a solid structural understanding of risk-related quantitative concepts and is capable of interpreting and applying mathematically defined objectives.

## Citation

```bibtex
@misc{cho2026portbench,
  title={Constructing a Portfolio Optimization Benchmark Framework for Evaluating Large Language Models},
  author={Cho et al. (2026)},
  year={2026},
  note={arXiv:2603.09301}
}
```

- arXiv: 2603.09301