Fanns Benchmark Eval

---
name: fanns-benchmark-eval
description: Evaluates the accuracy and efficiency of filtered approximate nearest neighbor search (FANNS) algorithms on high-dimensional transformer-based embeddings. It measures how well different indexing methods maintain recall under various real-world attribute filtering constraints while scaling to millions of vectors. Use when the user wants to benchmark on arxiv-for-fanns-medium, arxiv-for-fanns-large, or asks about evaluating this task. Reports recall@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.21989
  bibtex_key: iff2025benchmarking
  confidence: high
---

# fanns-benchmark-eval

> Benchmarking Filtered Approximate Nearest Neighbor Search Algorithms on Transformer-based Embedding Vectors — Iff et al. (2025) (arXiv:2507.21989, 2025)

## What this evaluates

Evaluates the accuracy and efficiency of filtered approximate nearest neighbor search (FANNS) algorithms on high-dimensional transformer-based embeddings. It measures how well different indexing methods maintain recall under various real-world attribute filtering constraints while scaling to millions of vectors.

## Datasets

- **arxiv-for-fanns-medium** — total 100000; splits: test (100000); repo https://github.com/spcl/fanns-benchmark
- **arxiv-for-fanns-large** — total 2700000; splits: test (2700000); repo https://github.com/spcl/fanns-benchmark

## Metrics

- `recall@10` **(primary)** — range: [0, 1]
  - Intersection over k of predicted and ground truth nearest neighbor sets: |knn_alg ∩ knn_gt| / k.
- `QPS` — range: other
  - Queries processed per second during single-threaded query execution.
- `index construction time` — range: other
  - Wall-clock time to build the index using all available hardware threads.
- `peak memory usage` — range: other
  - Maximum RAM consumed during index construction or query execution.
- `index size` — range: other
  - Disk/memory footprint of the built index.

## Input / output format

**Input**: 4096-dimensional normalized transformer embedding vectors paired with categorical/numerical attribute filters (equality, range, or equality-with-missing-value-is-satisfied). Queries specify a target vector and a filter condition.

**Output**: A ranked list of k=10 candidate item identifiers (indices) returned by the FANNS algorithm.

## Scoring recipe

```python
def compute_recall_at_10(pred_ids, gt_ids, k=10):
    pred_set = set(pred_ids[:k])
    gt_set = set(gt_ids[:k])
    return len(pred_set & gt_set) / k
```

## Common pitfalls

- Parameter tuning is highly sensitive to dataset scale and filter type; greedy search may converge to local optima, especially for methods with long construction times.
- ACORN's baseline implementation checks filters on all visited vertices rather than just traversal nodes, inflating overhead; the paper reports both upper and lower bound curves to address this inconsistency.
- Transformer embeddings are 4096-dimensional, causing int32 overflows in memory allocation for several methods when scaling to 2.7M vectors, requiring code modifications.

## Evidence (verbatim from paper)

> We focus on the well-established recall vs. queries per second (QPS) plots, which illustrate the trade-off between accuracy and query throughput achieved by each method. We define recall@k as: $$ \operatorname {r e c a l l} @ k = \frac {\left| \mathrm {k n n} _ {\mathrm {a l g}} \cap \mathrm {k n n} _ {\mathrm {g t}} \right|}{k} $$ where $\mathrm{knn}_{\mathrm{alg}}$ denotes the set of $k$ nearest neighbors returned by the algorithm, and $\mathrm{knn}_{\mathrm{gt}}$ is the ground truth.

## Citation

```bibtex
@misc{iff2025benchmarking,
  title={Benchmarking Filtered Approximate Nearest Neighbor Search Algorithms on Transformer-based Embedding Vectors},
  author={Iff et al. (2025)},
  year={2025},
  note={arXiv:2507.21989}
}
```

- arXiv: 2507.21989