Hftbench Eval

---
name: hftbench-eval
description: Evaluates an LLM agent's ability to execute profitable high-frequency trading decisions under strict latency constraints, balancing response speed with financial accuracy. The benchmark measures how well the model recognizes market patterns and executes trades within a fixed time window without degrading portfolio performance. Use when the user wants to benchmark on HFTBench, or asks about evaluating this task. Reports Daily Yield (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.19481
  bibtex_key: kang2025winfast
  confidence: high
---

# hftbench-eval

> Win Fast or Lose Slow: Balancing Speed and Accuracy in Latency-Sensitive Decisions of LLMs — Kang et al. (2025) (arXiv:2505.19481, 2025)

## What this evaluates

Evaluates an LLM agent's ability to execute profitable high-frequency trading decisions under strict latency constraints, balancing response speed with financial accuracy. The benchmark measures how well the model recognizes market patterns and executes trades within a fixed time window without degrading portfolio performance.

## Datasets

- **HFTBench** — total ?; splits: test (-1); repo https://github.com/HaoKang-Timmy/LatencySensitiveBench

## Metrics

- `Daily Yield (%)` **(primary)** — range: percent
  - Percentage change in portfolio value over a 1-minute evaluation window, starting with $10,000 initial cash and a profit threshold of 2%. Calculated as (final_cash - 10000) / 10000 * 100.
- `Latency (ms)` — range: ms
  - End-to-end inference time in milliseconds required to generate a trading decision.

## Input / output format

**Input**: Real-time stock market data (e.g., Nvidia and Amazon prices) and current portfolio state.

**Output**: Trading action (e.g., buy/sell/hold) and corresponding quantity.

## Scoring recipe

```python
cash = 10000
for each 1-min window:
    obs = get_stock_data()
    action = model(obs)
    cash = execute_trade(cash, action)
daily_yield = (cash - 10000) / 10000 * 100
latency = measure_inference_time_ms()
```

## Common pitfalls

- Assuming lower latency always improves yield; the paper shows that faster but inaccurate decisions on smaller models actually increase losses.
- Ignoring the fixed profit threshold (2%) and time window (1 min) when calculating yield, which standardizes the evaluation period across models.

## Evidence (verbatim from paper)

> Table[1] demonstrates that FPX , by dynamically trading off latency and quality through adaptive model size and bitwidth selection, achieves the highest daily yield on HFTBench and the best overall reward across both benchmarks.

## Citation

```bibtex
@misc{kang2025winfast,
  title={Win Fast or Lose Slow: Balancing Speed and Accuracy in Latency-Sensitive Decisions of LLMs},
  author={Kang et al. (2025)},
  year={2025},
  note={arXiv:2505.19481}
}
```

- arXiv: 2505.19481