Update References

---
name: update-references
description: >-
  Find, download, read, and summarize research papers related to the work in the
  current directory. Searches arXiv, OpenAlex, Semantic Scholar, and Google
  Scholar, with a hierarchical mode (hubs, surveys, citation snowball).
  Maintains a CSL-YAML
  reference database (references.yaml) compatible with pandoc's --bibliography
  flag, so the same file serves both the skill and document builds. Triggers:
  update references, refresh the bibliography, find papers, related work,
  literature review, what's been published on, recent papers, cite sources for,
  background reading, summarize this paper, search scholar.
---

# Update references (research paper search)

This skill turns "find me what's been written about this" into a small,
repeatable pipeline: read what the user is working on, search arXiv (and, for
off-arXiv work, Semantic Scholar or Google Scholar) for related papers,
download the relevant ones, read
them, write a summary per paper, and keep a Citation Style Language (CSL) YAML
database so work is never
repeated and versions stay current.

The standing interests are LLMs, AI agents, finite state machines, and
declarative agent patterns — the territory of spindle
(github.com/petar-djukic/spindle), a Go state-machine engine for agentic loops.
Lean toward those topics, but always sharpen the search using whatever the
current working directory is actually about.

## Where things live

This skill is context-aware. It runs from the directory the user is working in,
and it stores its outputs there — not in a fixed global location. Resolve paths
like this:

- **Database:** `references.yaml` under the current working directory, unless
  an existing one is already nearby (look for a `references.yaml` at or above
  the working directory and reuse it). Pass the chosen path as `--db` to every script call so
  search, fetch, and record all agree. If only a legacy `arxiv/papers.yaml`
  exists, the scripts read it transparently and convert on next write.
- **PDFs:** `<db-dir>/pdfs/` (transient — they exist to be read; they don't need
  to be committed). The script resolves this **relative to the database**, not
  the current directory, so PDFs land next to the db even when run from
  elsewhere.
- **Papers (markdown):** `<db-dir>/papers/` — `fetch` converts each PDF to
  markdown automatically (best-effort via `pymupdf4llm`), preserving headings,
  tables, and math. Papers are readable in any editor or tool.
- **Summaries:** `<db-dir>/summaries/`, one file per paper.

The PDF, markdown, and summary of a paper share one human-friendly stem:
`<Family>-<Year>-<title-slug>-<source>-<id>` (for example
`Lee-2026-meta-harness-end-to-end-optimization-arxiv-2603.28052`). The
`<source>-<id>` tail is `arxiv-<id>vN`, `doi-<doi>`, or `scholar-<key>`, so any
file traces back to its source at a glance. This differs from the pandoc
citation key, which stays short — just `<family>-<year>` (see the database
format). `repair` renames files created under the older naming scheme.
- **Manual-download list:** `<db-dir>/downloads-needed.md` — regenerated by
  `scholar.py pending`, a checklist of papers that could not be downloaded
  (paywalled or behind a login) with a landing URL for each. Derived from the
  database, so it is disposable, not hand-maintained. An entry carrying
  `download: deferred` is kept off the checklist (it lists compactly at the
  bottom instead) — set it on metadata-only entries the current work does not
  cite, and remove it when a citation pass starts citing the entry.

If the user names a directory or an existing database, use that instead. Because
every output path is derived from `--db`, passing an absolute `--db` keeps all
artifacts together regardless of the working directory.

## The database format

The database is CSL (Citation Style Language) YAML — by default a bare list with
no root key. Each entry has
standard CSL fields that pandoc understands (`id`, `type`, `title`, `author`,
`container-title`, `URL`, `issued`) plus skill-internal fields (`status`,
`version`, `pdf_path`, `arxiv_id`, etc.) that pandoc ignores. This means the
file is directly usable as `pandoc --bibliography references.yaml` with no
conversion step.

A `references:` root key is also accepted, because that is the form pandoc's own
documentation shows and what an existing pandoc bibliography usually looks like;
so is the skill's earlier `papers:` key. Whichever shape a file arrives in is
preserved when the skill writes it back, so pointing the skill at a bibliography
you already maintain does not reformat it. Any other shape is refused rather
than read as an empty database, and no command will write an empty database over
a file that had entries.

Every write goes out in `yq`'s normal form: the skill runs `yq -i '.'` over the
file after serializing it. PyYAML and yq indent nested sequences differently,
and on a bibliography of a few hundred entries that disagreement turns any
hand-edit made with yq into a diff the length of the file. One tool decides the
format, and it is the one the operator edits with. Without `yq` on PATH the
database is still written and still valid, in PyYAML's style, and the skill says
so on stderr — the next yq edit will reformat it whole.

An entry looks like:

```yaml
- id: lee-2026
  type: article
  title: "Meta-Harness: End-to-End Optimization of Model Harnesses"
  author:
    - family: Lee
      given: Yoonho
  container-title: arXiv preprint arXiv:2603.28052
  URL: https://arxiv.org/abs/2603.28052
  issued:
    year: 2026
  arxiv_id: "2603.28052"
  version: 1
  status: downloaded
  pdf_path: pdfs/Lee-2026-meta-harness-end-to-end-optimization-arxiv-2603.28052v1.pdf
  md_path: papers/Lee-2026-meta-harness-end-to-end-optimization-arxiv-2603.28052v1.md
```

The `id` field is a pandoc citation key (used as `@lee-2026` in markdown),
generated from the first author's family name and the year — just
`<family>-<year>`, with a letter suffix on collision (`lee-2026a`). It is
deliberately short; the richer provenance lives in the file names. The
`arxiv_id` field is the base arXiv identifier used for deduplication and
version tracking.

Entries fetched through OpenAlex additionally carry two marker blocks that
later steps use (pandoc ignores them):

```yaml
  openalex_id: W2158864412
  ranking:
    cited_by: 5671            # citations TO the paper
    references_count: 52      # citations IN the paper
    fwci: 102.6               # field-weighted citation impact (1.0 = field average)
    citation_percentile: 100.0
    venue: IEEE Transactions on Software Engineering
    institutions: [Massachusetts Institute of Technology]
    institution_types: [education]
    retrieved: 2026-07-15     # counts age; they are dated
  discovery:
    method: survey-references # seed-search | survey-references | forward-citations | author-drill
    via: "Wang et al. 2024 autonomous-agents survey"
```

`ranking` answers "how important, by whose measure, as of when"; `discovery`
answers "how did this paper enter the corpus". Both persist so the reading
step can prioritize and the summaries can cite them.

## Running the scripts

The Python scripts run in a pixi-managed environment that ships beside the
skill: `pixi.toml` and `pixi.lock` sit at the root of the agent directory
(the parent of this skill's `skills/` folder). Provision it once per machine —
the agent does this on repo open — with the preflight:

```bash
<agent-dir>/scripts/ensure-env.sh
```

Then run every script through pixi. For brevity the commands below write
`$RUN` for the wrapper:

```bash
RUN="pixi run --manifest-path <skill>/../../pixi.toml python"
```

where `<skill>/../../` resolves to the agent directory holding `pixi.toml`.
This replaces the older `pip install --user` path — no packages are installed
into the ambient interpreter.

## The workflow

### 0. Reconcile the database

Run this first, every time the skill is invoked. The goal is a complete
database in the correct naming convention before anything else happens:

```bash
$RUN <skill>/scripts/arxiv.py --db <db-path> reconcile
```

`reconcile` (the same command as `repair`) walks the whole database and the
`pdfs/` directory and makes disk and db agree. **The corpus is additive
(GH-234): reconcile never renames, moves, or deletes an existing file by
default.** Its default duties:

- an entry missing its markdown is converted;
- pdf/markdown/summary files on the older naming scheme are *reported* as
  rename candidates; the migration itself runs only under `--rename`
  (paths updated in the db; citation ids left intact), and even then a
  file living outside the skill's own subdirectory for its field — a
  curated `fulltext/` tree, say — is never touched: the skill did not put
  it there and does not own its layout. Every executed rename prints its
  old -> new pair;
- a PDF in `pdfs/` that no entry references is imported — its metadata is
  recovered from an arXiv id in the filename (fetched from arXiv), or, failing
  that, from the PDF's embedded title/author (recorded `status: needs-review`
  with year `nd`). Anything unrecoverable is listed in
  `<db-dir>/unregistered-pdfs.md` with a ready-to-run `ingest` command.

It is idempotent and prints counts (`converted`, `renamed`,
`rename_candidates`, `curated_skipped`, `collisions`, `imported`,
`needs_review`, `unregistered`). In the additive default an orphan whose
derived name would land on a DIFFERENT live entry's parked markdown is
refused with the reason (the GH-31 hazard, revived by opt-in renames and
re-guarded); the same paper reattaching its lost PDF still rewrites its
own markdown. A non-zero `collisions` means two
entries resolved to the same filename — usually a duplicated citation id — so
the rename was refused and those files were left on their old names rather than
one overwriting the other. Fix the duplicate and re-run.

The same count covers a refused import: an orphan that identifies as a paper
the database already holds is a duplicate copy, not a new paper, so it is left
in `pdfs/` under its own name instead of being renamed over the copy already
there. Delete it, or correct the entry it duplicates. To register an
unidentified PDF by hand, give `ingest` the metadata and it creates the entry:

```bash
$RUN <skill>/scripts/scholar.py --db <db-path> ingest --file pdfs/<file>.pdf \
  --title "Exact Title" --authors "Given Family" --year 2024
```

After reconciling, refresh the Obsidian tags and source links on every summary
(safe to run each time — it only changes what is stale):

```bash
$RUN <skill>/scripts/keywords.py --db <db-path> tag
```

### 1. Understand the current work first

Before searching, read what's in the working directory — a draft paper, notes,
an outline, existing summaries. The search is only as good as its query, and the
query should come from the actual problem the user is working on, not just the
standing topic list. If the directory is empty or the intent is unclear, ask the
user what angle they care about (one question, then proceed).

### 2-3. Search, dedupe, fetch

Per-backend mechanics — source order, dedupe rules, and how each fetch path
behaves — are in [search-and-fetch.md](./references/search-and-fetch.md).
In short: search the hierarchical backends, dedupe against `references.yaml`
before fetching anything, fetch the PDF where a direct link exists, and record
a `metadata-only` entry where it does not.

### 4. Read and summarize

When several papers await reading, use the `ranking` markers to order them:
pivotal first (high `fwci`/`cited_by`), and read a survey before the papers
discovered through it (`discovery.via` tells you which those are).

Read the markdown conversion that `fetch` produced — its path is in the
`md_path` field of the fetch output and the db entry (under
`<db-dir>/papers/`). If the markdown file is missing (e.g. an older fetch
before markdown conversion was available), run `repair` first to regenerate
it. Then write a summary file following
`references/summary-template.md` exactly. The summary's job is to let the user
decide, in two minutes, whether to cite the paper — so the "Relevance to the
current work" section carries the weight. Tie findings back to the current draft
and, where it fits, to spindle's state-machine / declarative-agent view. Prefer
the paper's own numbers over adjectives. An honest "low relevance" beats a
stretch.

Name the summary with the same stem as the paper's PDF and markdown so its
source is obvious:
`<db-dir>/summaries/<Family>-<Year>-<title-slug>-<source>-<id>.md` (the stem is
the basename of the entry's `md_path`, with the `summaries/` directory). If you
are unsure of the exact stem, run `repair` afterward — it renames any summary
to match.

### 5. Record it

Close the loop so the database reflects reality:

```bash
$RUN <skill>/scripts/arxiv.py --db <db-path> record --id 2310.12345 \
  --summary-file summaries/Lee-2026-meta-harness-end-to-end-optimization-arxiv-2603.28052v1.md \
  --topics llm agents fsm declarative-agents \
  --relevance "One line on why it matters to this work."
```

This flips the entry to `status: summarized`. The database is now the
single source of truth: re-running a search later will mark these papers
`known` and skip them.

Then apply Obsidian tags and source links to the new summaries:

```bash
$RUN <skill>/scripts/keywords.py --db <db-path> tag
```

This walks the db and, for every summary, writes a normalized `tags:` block
into the frontmatter — merging the paper's declared keywords (or a frequency
fallback when it declares none) with the entry's `topics`, plus a `paper` root
tag — and adds a `**Source:**` line linking the local PDF, the converted full
text, and the source URL. It also drops a one-time PDF back-link at the top of
each converted paper. Tags a human added are preserved; `--dry-run` previews
without writing; a second run is a no-op. Because it walks the whole db, it
also tags any older summary that predates this step, so a corpus converges.

### 6. Report back

Summarize what was found: how many candidates, how many were new vs. already
known, which were summarized, and a one-line takeaway per paper with a link to
its summary file. Point out the two or three most relevant to the current work.
If any papers came back `metadata-only`, run `scholar.py pending` and hand the
user `downloads-needed.md` so they can fetch those PDFs and return them via
`ingest`.

### Repair / reconcile

`repair` and `reconcile` are the same command — the database reconcile run at
step 0. Run it any time to bring disk and db back into agreement:

```bash
$RUN <skill>/scripts/arxiv.py --db <db-path> reconcile
```

It walks every entry, re-converts any missing `md_path` from the PDF on disk,
migrates legacy `text_path` entries, and renames any pdf/markdown/summary still
on the older naming scheme to the human-friendly stem (updating `pdf_path`,
`md_path`, `summary_file`; citation `id` values left unchanged). It then scans
`pdfs/` for files no entry references and imports them (arXiv-id recovery, then
embedded-metadata `needs-review`, then the `unregistered-pdfs.md` list). It is
idempotent — correctly-named, tracked files are left alone — and no step ever
moves a file onto one that already exists: a rename or an import that would
replace another paper is refused, counted as a `collision`, and explained on
stderr. Prints a JSON summary (`checked`, `converted`, `renamed`, `collisions`,
`imported`, `needs_review`, `unregistered`, `skipped`).

## Dependencies

The pixi environment (see "Running the scripts") supplies PyYAML (required),
`pymupdf4llm` (PDF-to-markdown conversion preserving headings, tables, and
math), and `pypdf` (the plain-text fallback). It is provisioned by
`ensure-env.sh`; no `pip install` is needed. If the conversion libraries were
somehow absent, fetch still downloads the PDF and just skips the conversion.
`yq` (the Go implementation, `brew install yq`) normalizes the database on
write; it is optional, and its absence costs formatting, not correctness.
Everything else is Python stdlib plus the arXiv public API — no key needed. Be
a good citizen: the script already retries with backoff; don't hammer the API
with huge `--max` values in a tight loop.

OpenAlex needs no key — set `OPENALEX_MAILTO` (any email) for the polite
pool's higher rate limits. Google Scholar search requires a SerpAPI key (same key as the idea-factory
job-search skill). Semantic Scholar needs no key — the public Graph API is
open, though shared and rate-limited; an optional `SEMANTIC_SCHOLAR_API_KEY`
(or `--api-key`) raises the limit.