Hume Eval

---
name: hume-eval
description: Evaluates text embedding models against human baselines across 16 MTEB datasets, probing semantic similarity, classification, clustering, and reranking capabilities. It specifically measures cross-lingual performance and identifies task ambiguities where model scores may reflect label pattern reproduction rather than genuine understanding. Use when the user wants to benchmark on MTEB (16 datasets, 26 task-language pairs), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.10062
  bibtex_key: elassadi2025hume
  confidence: high
---

# hume-eval

> HUME: Measuring the Human-Model Performance Gap in Text Embedding Tasks — El Assadi et al. (2025) (arXiv:2510.10062, 2025)

## What this evaluates

Evaluates text embedding models against human baselines across 16 MTEB datasets, probing semantic similarity, classification, clustering, and reranking capabilities. It specifically measures cross-lingual performance and identifies task ambiguities where model scores may reflect label pattern reproduction rather than genuine understanding.

## Datasets

- **MTEB (16 datasets, 26 task-language pairs)** — total ?; splits: test (-1); repo https://github.com/embeddings-benchmark/mteb

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly classified instances out of the total number of instances.
- `Spearman correlation` — range: [-1, 1]
  - Rank correlation coefficient between predicted and gold similarity scores.
- `V-measure` — range: [0, 1]
  - Clustering metric measuring homogeneity and completeness of cluster assignments.
- `MAP` — range: [0, 1]
  - Mean Average Precision for evaluating document reranking quality.

## Input / output format

**Input**: Text pairs (query, document) for STS and reranking tasks; single text instances for classification and clustering tasks.

**Output**: Model-generated embeddings or predicted labels/rankings/similarity scores per task.

## Scoring recipe

```python
def compute_metrics(predictions, gold, task_type):
    if task_type == 'classification':
        return accuracy_score(gold, predictions) * 100
    elif task_type == 'clustering':
        return v_measure_score(gold, predictions) * 100
    elif task_type == 'reranking':
        return map_score(gold, predictions) * 100
    elif task_type == 'sts':
        return spearmanr(gold, predictions)[0] * 100
    return mean([compute_metrics(p, g, t) for p, g, t in zip(preds, golds, tasks)])
```

## Common pitfalls

- Task ambiguity in emotion classification and academic clustering leads to low human agreement, making 'superhuman' model scores potentially reflect pattern matching rather than true understanding.
- Cross-lingual performance heavily depends on training data dominance; English tasks favor models while non-English tasks (Arabic, Russian) favor humans due to cultural/linguistic knowledge.
- Aggregating heterogeneous metrics (accuracy, V-measure, MAP, Spearman) into a single score requires careful normalization, which may obscure task-specific strengths and weaknesses.

## Evidence (verbatim from paper)

> Human annotators rank 4th overall with a score of 0.776, trailing 3 large models but outperforming 10 others. This positioning reveals that humans neither represent a performance ceiling nor a lower bound, but rather occupy a middle ground that varies significantly by task category and language. Classification: Human performance averages 70.3 across classification tasks, with substantial variation reflecting task-specific challenges. Clustering presents the greatest challenge for humans, averaging 67.4 V-measure with extreme variation. Reranking: Humans achieve strong performance in reranking (87.2 average MAP), demonstrating intuitive document relevance understanding. STS: Human performance averages 83.2 Spearman correlation, with notable dataset-specific variation.

## Citation

```bibtex
@misc{elassadi2025hume,
  title={HUME: Measuring the Human-Model Performance Gap in Text Embedding Tasks},
  author={El Assadi et al. (2025)},
  year={2025},
  note={arXiv:2510.10062}
}
```

- arXiv: 2510.10062