Patenteb Eval

---
name: patenteb-eval
description: Evaluates patent text embedding models across 15 diverse tasks including symmetric/asymmetric retrieval, classification, paraphrase detection, and clustering. It specifically probes domain-specific challenges like cross-domain retrieval, fragment-to-document matching, and temporal citation dynamics. Use when the user wants to benchmark on PatenTEB, or asks about evaluating this task. Reports NDCG@10, Macro-F1, Pearson r, V-measure.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.22264
  bibtex_key: [REDACTED]
  confidence: high
---

# patenteb-eval

> PatenTEB: A Comprehensive Benchmark and Model Family for Patent Text Embedding — Ayaou et al. (2025) (arXiv:2510.22264, 2025)

## What this evaluates

Evaluates patent text embedding models across 15 diverse tasks including symmetric/asymmetric retrieval, classification, paraphrase detection, and clustering. It specifically probes domain-specific challenges like cross-domain retrieval, fragment-to-document matching, and temporal citation dynamics.

## Datasets

- **PatenTEB** — total 2057286; splits: train (1556751), val (181215), test (319320); repo https://github.com/iliass-y/patenteb

## Metrics

- `NDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. Measures ranking quality by summing relevance scores discounted by position, normalized by the ideal DCG.
- `Macro-F1` **(primary)** — range: [0, 1]
  - Macro-averaged F1 score. Computes F1 for each class independently and takes the unweighted mean, treating all classes equally regardless of support.
- `Pearson r` **(primary)** — range: [-1, 1]
  - Pearson correlation coefficient. Measures the linear correlation between predicted similarity scores and ground-truth labels.
- `V-measure` **(primary)** — range: [0, 1]
  - V-measure clustering metric. Harmonic mean of homogeneity and completeness, evaluating how well clusters match ground-truth groups.

## Input / output format

**Input**: Varies by task: (1) Retrieval: query text and candidate patent documents/fragments; (2) Classification: patent text or citation pairs; (3) Paraphrase: pairs of patent segments; (4) Clustering: sets of patent families.

**Output**: Varies: (1) Retrieval: ranked list of candidates; (2) Classification: predicted class label; (3) Paraphrase: similarity score or binary label; (4) Clustering: cluster assignments.

## Scoring recipe

```python
def evaluate(predictions, gold, task_type):
    if task_type == 'retrieval':
        return ndcg_at_k(predictions, gold, k=10)
    elif task_type == 'classification':
        return macro_f1(predictions, gold)
    elif task_type == 'paraphrase':
        return pearson_correlation(predictions, gold)
    elif task_type == 'clustering':
        return v_measure_score(predictions, gold)
```

## Common pitfalls

- Hard negative mining strictly excludes citation-connected candidates and enforces domain-specific selection to prevent models from relying on superficial domain cues.
- Asymmetric retrieval tasks apply deterministic fragment removal to targets to prevent trivial lexical matching between query and document.
- Clustering tasks are evaluation-only with no training data, requiring unsupervised embedding quality.
- Splits are stratified by IPC3 code to ensure domain balance and prevent patent family leakage across train/val/test.

## Evidence (verbatim from paper)

> Table 2. Summary of dataset statistics by task family. NDCG@10 is used for retrieval tasks, Pearson correlation for paraphrase, Macro-F1 for classification (with 20% few-shot training), and V-measure for clustering.

## Citation

```bibtex
@misc{ayaou2025patenteb,
  title={PatenTEB: A Comprehensive Benchmark and Model Family for Patent Text Embedding},
  author={Ayaou et al. (2025)},
  year={2025},
  note={arXiv:2510.22264}
}
```

- arXiv: 2510.22264