Bwor Eval

---
name: bwor-eval
description: Evaluates LLMs' ability to automate operations research problem solving through mathematical modeling, code generation, and solver-based optimization. It probes whether reasoning agents can correctly translate natural language OR problems into executable models and compute optimal solutions. Use when the user wants to benchmark on BWOR, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.10009
  bibtex_key: zhang2025orllmagent
  confidence: medium
---

# bwor-eval

> OR-LLM-Agent: Automating Modeling and Solving of Operations Research Optimization Problems with Reasoning LLM — Zhang et al. (2025) (arXiv:2503.10009, 2025)

## What this evaluates

Evaluates LLMs' ability to automate operations research problem solving through mathematical modeling, code generation, and solver-based optimization. It probes whether reasoning agents can correctly translate natural language OR problems into executable models and compute optimal solutions.

## Datasets

- **BWOR** — total 82; splits: test (82); repo https://github.com/bwz96sco/or_llm_agent

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Percentage of problems where the model's final solution matches the ground-truth optimal solution.

## Input / output format

**Input**: LaTeX-formatted natural language problem descriptions, with tabular data included where applicable.

**Output**: Mathematical model, solver-based code, and optimal solution.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, ground_truth):
    if pred.solution == gold.solution:
        correct += 1
return correct / len(predictions)
```

## Common pitfalls

- Problems are manually translated from Chinese, so models may struggle with translation artifacts or domain-specific terminology.
- Ground-truth answers combine textbook solutions and domain expert computations, requiring careful verification of the scoring baseline.
- The task requires a multi-stage pipeline (modeling, coding, debugging), so evaluating only the final output may obscure where agent failures occur.

## Evidence (verbatim from paper)

> We release BWOR, an OR benchmark dataset consisting of 82 problems collected from standard OR textbooks. Ground-truth answers are partially sourced from the textbooks and partially computed by domain experts. Experimental results show that DeepSeek-R1-based OR-LLM-Agent outperforms GPT-o3, Gemini 2.5 Pro, and ORLM by at least 7% in accuracy across OR benchmarks...

## Citation

```bibtex
@misc{zhang2025orllmagent,
  title={OR-LLM-Agent: Automating Modeling and Solving of Operations Research Optimization Problems with Reasoning LLM},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2503.10009}
}
```

- arXiv: 2503.10009