Embeddinggemma Eval

---
name: embeddinggemma-eval
description: Evaluates the quality of text embeddings across diverse tasks including retrieval, classification, clustering, and semantic similarity. It probes multilingual, cross-lingual, and code understanding capabilities, measuring how well dense vector representations capture semantic relationships for downstream applications. Use when the user wants to benchmark on MTEB (Massive Text Embedding Benchmark), XOR-Retrieve, XTREME-UP, or asks about evaluating this task. Reports MTEB Task Mean.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.20354
  bibtex_key: [REDACTED]
  confidence: high
---

# embeddinggemma-eval

> EmbeddingGemma: Powerful and Lightweight Text Representations — Henrique Schechter Vera et al. (2025) (arXiv:2509.20354, 2025)

## What this evaluates

Evaluates the quality of text embeddings across diverse tasks including retrieval, classification, clustering, and semantic similarity. It probes multilingual, cross-lingual, and code understanding capabilities, measuring how well dense vector representations capture semantic relationships for downstream applications.

## Datasets

- **MTEB (Massive Text Embedding Benchmark)** — total 162; splits: test (-1)
- **XOR-Retrieve** — total ?; splits: test (-1)
- **XTREME-UP** — total ?; splits: test (-1)

## Metrics

- `MTEB Task Mean` **(primary)** — range: [0, 1]
  - Average of normalized scores across all individual MTEB tasks. Scores are typically normalized to [0, 1] per task before averaging.
- `MTEB Type Mean` — range: [0, 1]
  - Average of normalized scores grouped by task type (e.g., Retrieval, Classification, Clustering, STS).
- `Borda rank` — range: rank
  - Ranking metric computed by assigning points based on task-level performance across all models, then summing to produce an aggregate rank.
- `Recall@5kt` — range: [0, 1]
  - Fraction of queries where the correct passage is retrieved within the top 5,000 candidates.
- `MRR@10` — range: [0, 1]
  - Mean Reciprocal Rank at 10, averaging the inverse rank of the first correct passage across queries.

## Input / output format

**Input**: Text pairs (query, passage) or single documents, formatted with task-specific prompt instructions from the model card. Context length is typically 512 tokens, extended to 1024 or 2048 for long-context tasks.

**Output**: Dense embedding vectors, defaulting to 768 dimensions but also evaluated at 512, 256, and 128 dimensions.

## Scoring recipe

```python
def compute_mteb_scores(predictions, gold, task_type):
    if task_type in ['retrieval', 'reranking']:
        return compute_ranking_metrics(predictions, gold, k=10)
    elif task_type in ['classification', 'multilabel']:
        return compute_accuracy(predictions, gold)
    elif task_type == 'clustering':
        return compute_clustering_score(predictions, gold)
    elif task_type == 'sts':
        return pearson_spearson_correlation(predictions, gold)
    return normalized_score

def aggregate_scores(task_scores):
    task_mean = mean(task_scores)
    type_mean = mean(group_by_type(task_scores))
    return task_mean, type_mean
```

## Common pitfalls

- Models trained on more than 25% of the MTEB data are excluded to mitigate potential overfitting; failing to check training data composition can lead to invalid comparisons.
- Prompt instructions must strictly follow the model card; using default or mismatched prompts significantly degrades performance.
- Context length varies by task (512 vs 1024/2048); using a fixed context length across all tasks may underperform on long-context benchmarks.
- Embedding dimensionality heavily impacts scores; comparing 768d results directly with 128d results without noting the trade-off is misleading.

## Evidence (verbatim from paper)

> We also show results on XOR-Retrieve and XTREME-UP, reporting Recall@5kt and MRR@10 respectively. EmbeddingGemma achieves the #1 rank and highest overall performance on the MTEB multilingual, English, and code leaderboards across models under 500M parameters, with a significant lead over all previous top performing models on each of the metrics summarizing aggregate performance across tasks (Task Mean, Task Type Mean, and Borda rank).

## Citation

```bibtex
@misc{vera2025embeddinggemma,
  title={EmbeddingGemma: Powerful and Lightweight Text Representations},
  author={Henrique Schechter Vera et al. (2025)},
  year={2025},
  note={arXiv:2509.20354}
}
```

- arXiv: 2509.20354