Evaluation

---
name: evaluation
description: "Evaluate FlagEmbedding embedders and optional rerankers on local
  retrieval data or supported benchmarks."
disable-model-invocation: true
metadata:
  disco-role: operating
license: MIT
---

# FlagEmbedding Evaluation

Use this sub-skill when a task needs FlagEmbedding evaluation commands, custom retrieval JSONL preparation, metric/output selection, or benchmark routing for MTEB, BEIR, MSMARCO, MIRACL, MLDR, MKQA, AIR-Bench, or BRIGHT.

Do not use this sub-skill for low-level embedder or reranker API details; route those to sibling `inference`. Do not use it for training or fine-tuning data preparation; route those to sibling `fine-tuning`. Broad package install, backend, or import troubleshooting belongs to the root `troubleshooting` reference.

## First Decision

1. For local retrieval data with `corpus.jsonl`, `<split>_queries.jsonl`, and `<split>_qrels.jsonl`, prefer `python -m FlagEmbedding.evaluation.custom`.
2. For local data stored in multiple subdirectories, run the custom module once per subdirectory with its own `--dataset_dir`, `--output_dir`, `--corpus_embd_save_dir`, and `--eval_output_path`. The custom loader reports no dataset names, so do not use `--dataset_names` for this case.
3. For official benchmark names or languages, choose the benchmark module from `references/cli-reference.md` and confirm network, model-cache, and compute approval before running.
4. If the user wants only schema validation or a smoke fixture, create a tiny dataset with `scripts/create_tiny_retrieval_dataset.py` before building a model command.

## Operating Procedure

1. Confirm dependencies for the selected path. All dense retrieval evaluation paths import `faiss` and `pytrec_eval`; the focused CPU dependency set is `faiss-cpu` plus `pytrec_eval`.
2. Check the data layout against `references/data-formats.md`. For custom evaluation, use the generic IR schema with corpus `id`, optional `title`, `text`; query `id`, `text`; qrels `qid`, `docid`, `relevance`.
3. Build the command from `references/cli-reference.md`. Always make output paths explicit: `--output_dir`, `--eval_output_path`, and, when reusing corpus embeddings, `--corpus_embd_save_dir`.
4. Decide `--ignore_identical_ids` deliberately. Use it when query ids and document ids share an id namespace and self-hits are invalid; leave it false for datasets where identical ids are meaningful or when unsure.
5. If adding a reranker, set `--reranker_name_or_path`, choose `--reranker_model_class` for custom checkpoints, and keep `--rerank_top_k` no larger than `--search_top_k`.
6. After execution, inspect both raw search result JSON under `--output_dir` and the aggregate file at `--eval_output_path`.

## Verification Hooks

- Safe help check: `python -m FlagEmbedding.evaluation.custom --help`.
- Loader fact check: `CustomEvalDataLoader.available_dataset_names()` returns `[]`; `available_splits()` returns `["test"]`.
- Fixture check: `python scripts/create_tiny_retrieval_dataset.py --help` and a run that writes `corpus.jsonl`, `test_queries.jsonl`, and `test_qrels.jsonl`.
- Official benchmark shell flows are reference candidates only. Treat them as network and compute dependent, not safe defaults.

## References

- `references/cli-reference.md`: module commands, shared arguments, and benchmark-specific notes.
- `references/data-formats.md`: local JSONL schema and output structure.
- `references/troubleshooting.md`: dependency, layout, cache, metrics, reranker, and identical-id issues.
- `scripts/create_tiny_retrieval_dataset.py`: tiny self-contained custom retrieval fixture generator.