Finteam Financial Eval

---
name: finteam-financial-eval
description: Evaluates a multi-agent financial system's ability to answer real-world investor inquiries across macroeconomic, industry, and company analysis scenarios. Probes accuracy, thoroughness, clarity, and professional financial reasoning through automated LLM-judging and human preference testing. Use when the user wants to benchmark on NGA Grand Era Investor Inquiries, or asks about evaluating this task. Reports Overall Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.10448
  bibtex_key: wu2025finteam
  confidence: high
---

# finteam-financial-eval

> FinTeam: A Multi-Agent Collaborative Intelligence System for Comprehensive Financial Scenarios — Wu et al. (2025) (arXiv:2507.10448, 2025)

## What this evaluates

Evaluates a multi-agent financial system's ability to answer real-world investor inquiries across macroeconomic, industry, and company analysis scenarios. Probes accuracy, thoroughness, clarity, and professional financial reasoning through automated LLM-judging and human preference testing.

## Datasets

- **NGA Grand Era Investor Inquiries** — total 150; splits: test (150); repo https://github.com/FudanDISC/DISC-FinLLM

## Metrics

- `Overall Score` **(primary)** — range: [1, 5]
  - Average of 1-to-5 ratings assigned by GPT-4o across four dimensions: Accuracy, Thoroughness, Clarity, and Professionalism.
- `Acceptance Rate` — range: percent
  - Percentage of test cases where the model's response is selected as the best among all compared models by human evaluators.

## Input / output format

**Input**: Natural language investor inquiries regarding macroeconomic trends, industry sub-sectors, or publicly listed companies.

**Output**: Natural language financial analysis/response addressing the inquiry.

## Scoring recipe

```python
# GPT-4o Automated Scoring
gpt_scores = []
for response in responses:
    score = gpt4o_judge.evaluate(response, criteria=["Accuracy", "Thoroughness", "Clarity", "Professionalism"])
    gpt_scores.append(score.overall)  # 1-5 scale
overall_score = mean(gpt_scores)

# Human Acceptance Rate
human_picks = 0
for case in test_cases:
    best_model = human_evaluator.select_best([model_outputs[case]])
    if best_model == target_model:
        human_picks += 1
acceptance_rate = (human_picks / len(test_cases)) * 100
```

## Common pitfalls

- GPT-4o is used as an automated judge, which may introduce LLM-judge bias or over-reward verbose but shallow responses.
- Human evaluators are finance undergraduates rather than professional analysts, potentially affecting the reliability of the Acceptance Rate.
- Acceptance Rate is calculated over a multi-way comparison (5 models) rather than pairwise, making direct comparison with pairwise benchmarks difficult.

## Evidence (verbatim from paper)

> For evaluation, we employ GPT-4o to score the outputs from our agent system and the other models, ensuring objectivity and accuracy. The evaluation is conducted across four dimensions: (1) Accuracy: The model addresses key points directly, avoiding irrelevant details. (2) Thoroughness: The model provides a detailed, in-depth answer. (3) Clarity: The response is clear, concise, and logical. (4) Professionalism: The model uses appropriate financial perfessional terms. We compare model outputs across multiple dimensions, with GPT-4o rating each response from 1 to 5 per category and assigning an overall score. ... The Acceptance Rate indicates how often a model's output is chosen as the top answer.

## Citation

```bibtex
@misc{wu2025finteam,
  title={FinTeam: A Multi-Agent Collaborative Intelligence System for Comprehensive Financial Scenarios},
  author={Wu et al. (2025)},
  year={2025},
  note={arXiv:2507.10448}
}
```

- arXiv: 2507.10448