Gemini Embedding Eval

---
name: gemini-embedding-eval
description: Evaluates the quality of multilingual and code-aware text embeddings across diverse tasks including retrieval, classification, clustering, and cross-lingual matching. It probes the model's ability to generate generalizable representations that perform well across 250+ languages, English, and programming languages. Use when the user wants to benchmark on MMTEB, XTREME-UP, XOR-Retrieve, or asks about evaluating this task. Reports Task Mean.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.07891
  bibtex_key: lee2025geminiembedding
  confidence: high
---

# gemini-embedding-eval

> Gemini Embedding: Generalizable Embeddings from Gemini — Lee et al. (2025) (arXiv:2503.07891, 2025)

## What this evaluates

Evaluates the quality of multilingual and code-aware text embeddings across diverse tasks including retrieval, classification, clustering, and cross-lingual matching. It probes the model's ability to generate generalizable representations that perform well across 250+ languages, English, and programming languages.

## Datasets

- **MMTEB** — total 164; splits: test (-1)
- **XTREME-UP** — total ?; splits: test (-1)
- **XOR-Retrieve** — total ?; splits: test (-1)

## Metrics

- `Task Mean` **(primary)** — range: [0, 1] or percent
  - Equal weighting of all individual task scores across the benchmark.
- `Task Type Mean` — range: [0, 1] or percent
  - Equal weighting of all task type scores (e.g., Retrieval, Classification, STS).
- `Borda rank` — range: integer
  - Official leaderboard ranking metric based on aggregate task performance.
- `MRR@10` — range: [0, 1]
  - Mean Reciprocal Rank at cutoff 10, used for cross-lingual retrieval tasks.

## Input / output format

**Input**: Text queries and passages/documents in various languages (including English, 250+ languages, and code), optionally with task-specific prompts.

**Output**: Dense vector embeddings for each text instance, used to compute cosine similarity for ranking/retrieval tasks.

## Scoring recipe

```python
def compute_metrics(embeddings, queries, passages, gold_labels, task_type):
    scores = cosine_similarity(queries, passages)
    if task_type in ['retrieval', 'reranking']:
        ranks = argsort(scores, descending=True)
        return compute_mrr(ranks, gold_labels, k=10)
    elif task_type == 'classification':
        return accuracy(scores, gold_labels)
    # ... other task-specific scorers (STS, clustering, etc.)
    return task_score

task_scores = [compute_metrics(e, q, p, g, t) for t, e, q, p, g in tasks]
task_mean = mean(task_scores)
task_type_mean = mean(mean(task_scores_by_type))
borda_rank = rank_by_score(task_mean)
```

## Common pitfalls

- Confusing Task Mean (equal weighting of all tasks) with Task Type Mean (equal weighting of task types), which can yield different model rankings.
- MTEB(Code) leaderboard results exclude 4 specific tasks (CodeFeedbackMT, CodeFeedbackST, StackOverflowQA, SyntheticText2SQL); reporting on all 12 without noting the exclusion misrepresents standing.
- Cross-lingual retrieval benchmarks (XTREME-UP, XOR-Retrieve) evaluate zero-shot retrieval without translation, which heavily penalizes models lacking shared multilingual alignment.

## Evidence (verbatim from paper)

> Gemini Embedding establishes a new state-of-the-art in performance, achieving the highest overall performance on the MTEB(Multilingual) leaderboard (March 10th, 2025) with a substantial performance lead over all previous top performing models on each of the overall metrics summarizing aggregate performance across tasks: Task Mean (equal weighting of all tasks): 68.32, Task Type Mean (equal weighting of all task types): 59.64, and Borda rank #1 (official leaderboard ranking metric).

## Citation

```bibtex
@misc{lee2025geminiembedding,
  title={Gemini Embedding: Generalizable Embeddings from Gemini},
  author={Lee et al. (2025)},
  year={2025},
  note={arXiv:2503.07891}
}
```

- arXiv: 2503.07891