Matchbench Eval

---
name: matchbench-eval
description: Evaluates the matching ability, correspondence sufficiency, and computational efficiency of local feature matchers across short- and wide-baseline image pairs. It measures how well matchers recover camera pose and how many correct correspondences they produce, enabling fair comparison for real-time applications like SLAM. Use when the user wants to benchmark on SfM/SLAM datasets (sequences 01-08), or asks about evaluating this task. Reports AUC (SP curve).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1808.02267
  bibtex_key: [REDACTED]
  confidence: high
---

# matchbench-eval

> MatchBench: An Evaluation of Feature Matchers — Bian et al. (2018) (arXiv:1808.02267, 2018)

## What this evaluates

Evaluates the matching ability, correspondence sufficiency, and computational efficiency of local feature matchers across short- and wide-baseline image pairs. It measures how well matchers recover camera pose and how many correct correspondences they produce, enabling fair comparison for real-time applications like SLAM.

## Datasets

- **SfM/SLAM datasets (sequences 01-08)** — total ?; splits: test (-1)

## Metrics

- `AUC (SP curve)` **(primary)** — range: [0, 1]
  - Area under the Success Ratio (SP) curve, which plots the fraction of correctly estimated camera poses against varying pose error thresholds.
- `AP (correspondence sufficiency)` — range: other
  - Average number of correspondences computed over image pairs that yield a correct pose estimate.
- `Time consumption` — range: other
  - Wall-clock time in milliseconds for feature detection, matching, and correspondence selection.

## Input / output format

**Input**: Pairs of images (short-baseline or wide-baseline) from SfM/SLAM sequences.

**Output**: Set of matched keypoint correspondences, estimated relative camera pose (essential matrix), and execution time logs.

## Scoring recipe

```python
# For each image pair:
# 1. Extract features & match (FLANN+Euclidean for real-valued, BF+Hamming for binary)
# 2. Estimate pose (OpenCV 5-point RANSAC for sparse matchers, RepMatch estimator for rich matchers)
# 3. Compute pose error threshold vs success ratio -> plot SP curve -> compute AUC
# 4. Count matches for correctly matched pairs -> average -> AP
# 5. Record detection, matching, selection times in ms
```

## Common pitfalls

- Using the default OpenCV pose estimator for rich matchers (CODE, RepMatch, GMS) instead of the RepMatch built-in estimator, which fails on dense correspondences.
- Not overriding ORB's default nfeatures=500 to 100,000, artificially capping feature detection.
- Evaluating slow matchers (KVLD, GAIM, CODE, RepMatch) on short-baseline tasks where real-time efficiency is required.
- Applying Euclidean distance to binary descriptors or Hamming distance to real-valued features.

## Evidence (verbatim from paper)

> The AUC score of matchers is shown in Tab.[2](#S5.T2 "Table 2 ‣ 5.2 Evaluation results and analyses ‣ 5 Experiments ‣ MatchBench: An Evaluation of Feature Matchers"), and the time consumption of matchers is shown in Tab.[3](#S5.T3 "Table 3 ‣ 5.2 Evaluation results and analyses ‣ 5 Experiments ‣ MatchBench: An Evaluation of Feature Matchers"). These results enable us to analyze the matching ability, correspondence sufficiency, as well as efficiency of different matchers.

## Citation

```bibtex
@misc{bian2018matchbench,
  title={MatchBench: An Evaluation of Feature Matchers},
  author={Bian et al. (2018)},
  year={2018},
  note={arXiv:1808.02267}
}
```

- arXiv: 1808.02267