Dapfam Eval

---
name: dapfam-eval
description: Evaluates cross-domain patent retrieval systems by measuring how well they rank relevant patent documents or passages when queries and targets share or lack overlapping IPC3 classifications. Use when the user wants to benchmark on DAPFAM, or asks about evaluating this task. Reports NDCG@100.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.22141
  bibtex_key: ayaou2025dapfam
  confidence: high
---

# dapfam-eval

> DAPFAM: A Domain-Aware Family-level Dataset to benchmark cross domain patent retrieval — Ayaou et al. (2025) (arXiv:2506.22141, 2025)

## What this evaluates

Evaluates cross-domain patent retrieval systems by measuring how well they rank relevant patent documents or passages when queries and targets share or lack overlapping IPC3 classifications.

## Datasets

- **DAPFAM** — total ?; splits: ALL (-1), IN (-1), OUT (-1)

## Metrics

- `NDCG@100` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 100, averaged across queries within evaluation subsets.
- `Recall@100` — range: [0, 1]
  - Recall at rank 100, measuring the proportion of relevant documents found in the top 100 results, averaged across queries.

## Input / output format

**Input**: Patent query (Title, Abstract, Claims) and a corpus of patent documents or passages indexed for retrieval.

**Output**: Ranked list of up to 100 patent documents or passages per query.

## Scoring recipe

```python
def compute_ndcg_at_k(relevance_scores, k=100):
    dcg = sum(r / math.log2(i + 2) for i, r in enumerate(relevance_scores[:k]))
    ideal = sorted(relevance_scores, reverse=True)[:k]
    idcg = sum(r / math.log2(i + 2) for i, r in enumerate(ideal))
    return dcg / idcg if idcg > 0 else 0.0

def compute_recall_at_k(relevance_scores, k=100):
    relevant_count = sum(1 for r in relevance_scores if r > 0)
    return sum(1 for r in relevance_scores[:k] if r > 0) / max(relevant_count, 1)
```

## Common pitfalls

- Performance drops drastically on OUT-domain queries due to vocabulary divergence, so reporting only ALL/IN masks cross-domain failure.
- Passage length and aggregation strategy (avg_top3 vs maxP) must be tuned per backend and domain; fixed settings hurt performance.
- Execution time excludes preprocessing/indexing, so efficiency claims only reflect query-time scoring.

## Evidence (verbatim from paper)

> All results report NDCG@100 and Recall@100 averaged across queries within evaluation subsets.

## Citation

```bibtex
@misc{ayaou2025dapfam,
  title={DAPFAM: A Domain-Aware Family-level Dataset to benchmark cross domain patent retrieval},
  author={Ayaou et al. (2025)},
  year={2025},
  note={arXiv:2506.22141}
}
```

- arXiv: 2506.22141