Find Eval

---
name: find-eval
description: Evaluates automated interpretability methods on their ability to describe black-box functions across numeric, string, and semantic domains. It probes both static language model capabilities and interactive agent reasoning, including handling complexities like composition, noise, bias, and approximation. Use when the user wants to benchmark on FIND, or asks about evaluating this task. Reports adequately described rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2309.03886
  bibtex_key: schwettmann2023find
  confidence: medium
---

# find-eval

> FIND: A Function Description Benchmark for Evaluating Interpretability Methods — Schwettmann et al. (2023) (arXiv:2309.03886, 2023)

## What this evaluates

Evaluates automated interpretability methods on their ability to describe black-box functions across numeric, string, and semantic domains. It probes both static language model capabilities and interactive agent reasoning, including handling complexities like composition, noise, bias, and approximation.

## Datasets

- **FIND** — total 2000; splits: test (-1); repo https://github.com/multimodal-interpretability/FIND

## Metrics

- `adequately described rate` **(primary)** — range: percent
  - Percentage of functions for which the generated description adequately captures the function's behavior compared to human-labeled concepts. Calculated as (adequately described functions / total functions) * 100.

## Input / output format

**Input**: Black-box functions (numeric, string, or semantic) provided as executable code or API calls. For initialization experiments, 10 exemplar inputs per function are provided (2 associated with the function, 8 distractors).

**Output**: A textual description of the function's behavior, or an interactive sequence of hypotheses, experiments, and refinements from an Automated Interpretability Agent (AIA).

## Scoring recipe

```python
def score(predictions, gold):
    adequate_count = 0
    for pred, gold_concepts in zip(predictions, gold):
        if matches_human_concepts(pred, gold_concepts):
            adequate_count += 1
    return (adequate_count / len(predictions)) * 100
```

## Common pitfalls

- Interpreters may hallucinate function outputs instead of actually querying the black-box function.
- Providing initialization exemplars with many distractor inputs can severely handicap the interpreter's search strategy.
- Agents may exhibit high-level scientific reasoning but still fail to capture local function details or polysemantic behaviors.

## Evidence (verbatim from paper)

> FIND introduces a procedural benchmark of over 2000 black-box function interpretation problems across numeric, string, and semantic domains, designed to evaluate automated interpretability methods. It assesses both static (LM-based) and interactive (Automated Interpretability Agent, AIA) approaches, revealing that while AIAs exhibit scientific reasoning—proposing experiments, updating hypotheses, and refining descriptions—they still fail to capture local function details, with 48% of functions inadequately described.

## Citation

```bibtex
@misc{schwettmann2023find,
  title={FIND: A Function Description Benchmark for Evaluating Interpretability Methods},
  author={Schwettmann et al. (2023)},
  year={2023},
  note={arXiv:2309.03886}
}
```

- arXiv: 2309.03886