Ucfe Eval

---
name: ucfe-eval
description: Evaluates large language models' ability to handle dynamic, multi-turn financial dialogues across diverse user personas and task types, measuring their adaptability to shifting user needs and financial expertise. Use when the user wants to benchmark on UCFE, or asks about evaluating this task. Reports Elo score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.14059
  bibtex_key: yang2024ucfe
  confidence: medium
---

# ucfe-eval

> UCFE: A User-Centric Financial Expertise Benchmark for Large Language Models — Yang et al. (2024) (arXiv:2410.14059, 2024)

## What this evaluates

Evaluates large language models' ability to handle dynamic, multi-turn financial dialogues across diverse user personas and task types, measuring their adaptability to shifting user needs and financial expertise.

## Datasets

- **UCFE** — total 330; splits: test (330); repo https://github.com/TobyYang7/UCFE-Benchmark

## Metrics

- `Elo score` **(primary)** — range: other
  - Pairwise comparison rating system where models earn points based on win/loss/draw outcomes against other models in head-to-head dialogues.
- `Pearson correlation` — range: [-1, 1]
  - Measures linear correlation between model Elo scores and human expert preference ratings to validate alignment.

## Input / output format

**Input**: Multi-turn financial dialogues categorized by 4 user types (analysts, professionals, regulators, public) and 17 task types, provided in zero-shot or few-shot prompt formats.

**Output**: Model-generated responses per dialogue turn, evaluated via pairwise comparison and LLM-as-Judge scoring.

## Scoring recipe

```python
# 1. Generate model responses for each multi-turn dialogue
predictions = model.generate(dialogue_history)
# 2. Pairwise comparison via LLM-as-Judge or human experts
win_rate = count_wins(predictions) / total_comparisons
# 3. Compute Elo rating from win rates
elo = 400 * log10(win_rate / (1 - win_rate))
# 4. Validate against human preferences
pearson_r = np.corrcoef(elo_scores, human_scores)[0, 1]
```

## Common pitfalls

- Zero-shot task text limitations can artificially depress model performance scores.
- Dynamic task evolution requires models to adapt mid-dialogue, making static evaluation metrics insufficient.
- LLM-as-Judge scoring may inherit biases from the judge model's training data.

## Evidence (verbatim from paper)

> Table D-1 presents the Elo scores across all tasks. The results show that LLMs trained on financial text data demonstrate outstanding performance across various tasks. However, due to limitations in the zero-shot task texts, the models' performance in those tasks may be affected.

## Citation

```bibtex
@misc{yang2024ucfe,
  title={UCFE: A User-Centric Financial Expertise Benchmark for Large Language Models},
  author={Yang et al. (2024)},
  year={2024},
  note={arXiv:2410.14059}
}
```

- arXiv: 2410.14059