Tasks And Benchmarks

---
name: tasks-and-benchmarks
description: "Discover, filter, inspect, and reason about MTEB tasks and
  predefined benchmarks before evaluation."
disable-model-invocation: true
metadata:
  disco-role: operating
license: Apache 2.0
---

# MTEB Tasks and Benchmarks

Use this sub-skill when the user needs to choose MTEB tasks or benchmarks, inspect task metadata, filter by language/script/domain/task type/category/modality, understand beta/private/superseded defaults, or compare benchmark task lists.

## Route First

- For running selected tasks or benchmarks with a model, use `../evaluation-workflows/SKILL.md` after selection is complete.
- For shell workflows around `mteb available-tasks` or `mteb available-benchmarks`, use `../cli-and-automation/SKILL.md`.
- For implementing or registering new tasks, use `../contributing-to-mteb/SKILL.md`.

## Core Workflows

- Select named tasks with `mteb.get_tasks(tasks=[...])` or one task with `mteb.get_task(name, languages=..., script=..., eval_splits=..., hf_subsets=...)`; named-task selection ignores broad `domains`, `task_types`, and `categories` filters, so apply `mteb.filter_tasks(...)` to a returned list if both name and metadata filtering are required.
- Discover tasks with `mteb.get_tasks(languages=..., script=..., domains=..., task_types=..., categories=..., modalities=..., eval_splits=...)`; filters are combined as an AND operation, with OR semantics inside each list unless an exclusive flag is documented.
- Select benchmarks with `mteb.get_benchmark("MTEB(eng, v2)")` or enumerate them with `mteb.get_benchmarks(names=..., display_on_leaderboard=...)`; a `Benchmark` is iterable and can be passed as `tasks=` to evaluation.
- Filter benchmark tasks with `mteb.filter_tasks(benchmark, task_types=[...], languages=[...], modalities=[...])`, then pass the filtered list to evaluation.
- Inspect selections safely with `scripts/inspect_task_selection.py`; it imports MTEB and prints task/benchmark summaries without downloading datasets or running models.

## References

- `references/task-benchmark-reference.md` - API patterns for task and benchmark discovery, selection, and benchmark filtering.
- `references/metadata-and-filters.md` - metadata fields, filter semantics, language/script distinctions, and default exclusion behavior.
- `references/troubleshooting.md` - diagnosis for import issues, optional dependencies, private/beta/superseded surprises, task-name typos, and CLI/API misuse.

## Quick Validation

- Confirm package import with `python -c "import mteb; print(mteb.__version__)"`.
- Preview a task filter with `python scripts/inspect_task_selection.py --task-types Retrieval --languages eng --format markdown --limit 20`.
- Preview a benchmark with `python scripts/inspect_task_selection.py --benchmark "MTEB(eng, v2)" --task-types Retrieval --format json --limit 10`.
- If selection looks correct, hand the task objects, task names, or benchmark to `evaluation-workflows` for actual `mteb.evaluate(...)` usage.