Treeeval Eval

---
name: treeeval-eval
description: TreeEval probes an LLM's ability to handle complex, adaptive reasoning through dynamically generated hierarchical questions. It evaluates how well a model's relative performance ranking aligns with established leaderboards like AlpacaEval2.0, while testing the framework's efficiency in distinguishing fine-grained capability differences without relying on static datasets. Use when the user wants to benchmark on TreeEval (Dynamic/Benchmark-Free), or asks about evaluating this task. Reports Spearman correlation ($
ho$).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.13125
  bibtex_key: li2024treeeval
  confidence: high
---

# treeeval-eval

> TreeEval: Benchmark-Free Evaluation of Large Language Models through Tree Planning — Li et al. (2024) (arXiv:2402.13125, 2024)

## What this evaluates

TreeEval probes an LLM's ability to handle complex, adaptive reasoning through dynamically generated hierarchical questions. It evaluates how well a model's relative performance ranking aligns with established leaderboards like AlpacaEval2.0, while testing the framework's efficiency in distinguishing fine-grained capability differences without relying on static datasets.

## Datasets

- **TreeEval (Dynamic/Benchmark-Free)** — total ?; splits: (unstated); repo https://github.com/Ashura5/TreeEval

## Metrics

- `Spearman correlation ($
ho$)` **(primary)** — range: other
  - Measures the rank correlation between the model rankings produced by TreeEval and the gold rankings from AlpacaEval2.0. Values range from -1 to 1, where 1 indicates perfect agreement in ranking order.
- `Kendall correlation ($	au$)` — range: other
  - Measures the ordinal association between TreeEval rankings and AlpacaEval2.0 rankings by comparing concordant and discordant pairs. Values range from -1 to 1.

## Input / output format

**Input**: A dynamically generated question or prompt produced by the examiner LLM (GPT-4) based on a hierarchical tree structure. Questions are tailored to specific topics and adapt in difficulty based on pairwise comparison progress.

**Output**: A natural language response to the generated question. The response is later compared pairwise against a reference model's response by the examiner LLM.

## Scoring recipe

```python
def compute_spearman_correlation(treeeval_scores, gold_scores):
    # treeeval_scores: list of aggregated TreeEval scores per model
    # gold_scores: list of AlpacaEval2.0 scores per model
    from scipy.stats import spearmanr
    rho, _ = spearmanr(treeeval_scores, gold_scores)
    return rho
```

## Common pitfalls

- TreeEval is benchmark-free and generates questions dynamically; results depend on the examiner's temperature (set to 1) and random seed, so exact reproducibility requires fixing these.
- Scores are relative pairwise comparisons against a fixed reference model (Mistral-7B-Instruct-v0.2), not absolute capability measures.
- The number of questions (#Q) varies per model pair; the reported average (#Q) is for efficiency comparison, not a fixed evaluation budget.

## Evidence (verbatim from paper)

> We report the correlation of rankings obtained through different methods with those from AlpacaEval2.0, using $	au$ for the Kendall correlation coefficient and $
ho$ for the Spearman correlation coefficient.

## Citation

```bibtex
@misc{li2024treeeval,
  title={TreeEval: Benchmark-Free Evaluation of Large Language Models through Tree Planning},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2402.13125}
}
```

- arXiv: 2402.13125