Ev Discovery

---
name: ev-discovery
description: >
  Skippable Phase 0 of the Gaia evidence verification pipeline. Use only on declared need for higher-quality Stage-2 evidence: a promotion candidate, or a skill that gaia-meta-sweep flagged as under-evidenced. Given a skill id, source repo/context, and needed evidence types (benchmark-result, arxiv, peer-review, social-signal), it runs Firecrawl searches, scrapes top hits to markdown, and appends discovered sources into Phase 1 inputs. Missing benchmark evidence is recorded as benchmark-source candidates first, not direct scoring rows. Trigger phrases: "discover evidence", "find new evidence", "Phase 0", "run ev-discovery", "search for benchmarks/papers", "under-evidenced", "promotion candidate needs stronger proof". This is the ONLY ev-* skill that searches the web for NEW evidence; other ev-* skills aggregate or verify known sources. Requires Firecrawl to be usable (`firecrawl --status` authenticated OR FIRECRAWL_API_KEY set); skips gracefully when neither holds.
---

# Evidence Discovery (ev-discovery)

Phase 0 is optional. It searches for new, higher-quality Stage-2 evidence only when a maintainer declares a need.

## Type-First Evidence Lake Contract (#1148)

The evidence lake is **type-first**. Discovery feeds Phase 1, which materializes `evidence/by-type/<canonical-evidence-type>.md`. Legacy `evidence/tier_*.md` files may still exist as coexistence artifacts, but they are **not** the semantic routing key.

## Discovery Scope

Prioritize new evidence rows for:

- `benchmark-result`
- `arxiv`
- `peer-review`
- richer `social-signal`

Discovery may also identify better `repo-own` or `github-stars-own` sources, but it must not assign authoritative rank, tier, grade, or Trust Magnitude.

## Benchmark candidate handoff

When discovery finds missing benchmark evidence, create a benchmark-source candidate record for Phase 2B review instead of appending a scoring `benchmark-result` row from a vendor page. The candidate should name the target skill, source URL, proposed `benchmarkId`, proposed generic applicability (`appliesToGenericSkillRefs`) when known, `status: candidate`, missing reproducibility fields, and notes. No discovered vendor page may become a scoring row until a human promotes the source in `registry/benchmark-sources.json` and approves `/gaia-ingest-batch`. A promoted `registered` benchmark source is still generic-applicable, non-scoring catalog metadata only until a named `benchmark-result` row satisfies reproducibility fields and approved provenance. Firecrawl-style vendor benchmark claims are candidate-only or registered-only until reproducibility fields, allowed provenance, and human approval exist.

## Handoff to Phase 1

Append discovered raw rows to the appropriate source inputs with a canonical or normalizable evidence type. Phase 1 is responsible for normalizing aliases and materializing `evidence/by-type/<type>.md`. Do not hand-write generated by-type partitions as if they were source rows.

If one discovered peer-review source legitimately covers multiple named skills, hand Phase 1 a scratch multi-target packet manifest for `evidence/scripts/peer_review_source_packets.py` rather than cloning the same review into multiple intake mutations. The helper is `peer-review`-only, allows the same URL once per reviewed skill, rejects strength/scoring fields (`trustNumber`, `grade`, `class`, `tier`, `level`, `stars`, `rank`), and should write only temporary by-type output.

## Preflight (#1788)

Firecrawl usability is **not** gated on the `FIRECRAWL_API_KEY` env var alone — the
CLI can be authenticated via stored browser/CLI credentials independent of it.
Check both before deciding to skip Phase 0:

```bash
firecrawl --status                 # Authenticated? credits remaining?
echo "${FIRECRAWL_API_KEY:+set}"   # non-empty output if the env var is set
```

Proceed with discovery if **either** check succeeds. Skip Phase 0 gracefully
only when `firecrawl --status` reports unauthenticated **and** the env var is
unset/empty.

## Commands

Quote every URL and query — unquoted `?`/`&`/glob characters expand under zsh,
this repo's shell, and silently turn a single request into a file-glob error.

```bash
mkdir -p .firecrawl
firecrawl search "<query>" -o .firecrawl/result.json --json
firecrawl scrape '<url>' --json -o .firecrawl/scrape.json
gh api -X GET 'repos/<owner>/<repo>/git/trees/main' -f recursive=1
```

The `gh api` form above (path + `-f recursive=1` instead of a literal
`?recursive=1` in the URL) avoids the same zsh glob-expansion problem for
GitHub tree listings.

## Retry rule

Retry is a **status-code rule**, not a named-host list — a source is not
"unresolved" just because it wasn't on someone's retry shortlist:

- On any `401`, `402`, `403`, `429`, or SSL/TLS failure from a direct fetch,
  retry **once** via `firecrawl scrape` before marking the source dead.
- Do not maintain or consult a list of specific domains to retry — apply the
  rule to every failing URL, every time.

## Guardrails

- Requires Firecrawl usability per the Preflight above (`firecrawl --status`
  OR `FIRECRAWL_API_KEY`); skip gracefully only when both fail.
- Do not run as a broad web crawl without declared need.
- Do not mutate registry files.
- Do not append direct scoring `benchmark-result` rows from discovered vendor pages; route them through benchmark-source candidates and Phase 2B first.
- Do not treat generic-applicable registered benchmark sources as Trust Magnitude evidence; they are non-scoring until a named row satisfies reproducibility and approved provenance.
- Do not commit generated collector/by-type/report artifacts without a human gate.