Pdf To Md

---
name: pdf-to-md
description: >-
  Convert PDFs and office documents to clean Markdown, with structured bundles
  for scientific papers. Use when extracting article structure, preparing a
  manuscript for analysis, or creating CSAG input.
license: CC0-1.0
metadata:
  version: "1.0.0"
---

# pdf-to-md

Turn a PDF into Markdown. The right path depends on the document type and whether
external document submission has been approved:

- **Scientific paper** → produce the canonical `paper-to-md` bundle (Markdown +
  `section_audit.json` + `article.json`) so it can feed `csag-extraction`.
  Use **LiteParse v2** locally unless the user explicitly approves the remote OCR API.
- **Any other PDF** (reports, slides, letters, forms) → just convert to Markdown
  with **LiteParse v2** for a fast, local, no-key result. Stop there.

**LiteParse must be v2** ([run-llama/liteparse](https://github.com/run-llama/liteparse),
the Rust rewrite with the `LiteParse` Python API and `lit` CLI). LiteParse v1 is a
different, unsupported API. `liteparse_to_md.py` pins `liteparse>=2,<3` and refuses
to run on anything else, so `uv run` always provisions the right per-platform v2
binary inside the wheel — nothing to vendor or compile, and no API key. OCR is on by
default (bundled Tesseract).

**LiteParse output is a draft, not the deliverable.** LiteParse is a *mechanical*
parser: it has no native Markdown, infers headings from font size/weight, and
introduces artifacts (split words, broken hyphenation, dropped author blocks, merged
columns). Whenever LiteParse is the engine, the LLM running this skill is responsible
for shaping that draft into the right form — see "Shape the LiteParse output" below.
The OCR API engine needs far less shaping.

## Instructions

### Step 0 — Classify the document and pick a path

| Document | Remote upload approved? | Path |
|----------|-------------------------|------|
| Scientific paper / manuscript | yes, and an OCR key is configured | Mode A, OCR API with `--allow-remote` |
| Scientific paper / manuscript | no | Mode A, LiteParse v2 locally |
| Anything else | no remote upload needed | Mode B, LiteParse v2 locally |

Check for a key without printing it:

```bash
if [ -n "${OCR_API_KEY:-}${NELLI_API_KEY:-}" ]; then
  echo "OCR key configured"
else
  echo "No OCR key configured"
fi
```

Having a key is not approval to upload a confidential document. Use the remote
engine only after the user authorizes external submission. LiteParse v2 OCRs
locally when remote upload is not approved.

Resolve the installed skill once per shell:

```bash
PDF_TO_MD_SKILL="${PDF_TO_MD_SKILL:-$HOME/.agents/skills/pdf-to-md}"
```

### Mode A — Scientific paper (full bundle)

Produces, beside the input, for stem `<stem>`:
`<stem>.md`, `<stem>.section_audit.json`, `<stem>.article.json`
(and optionally `<stem>.ocr.json`, `<stem>.job.json`, `figure_review/`).

1. **Convert to Markdown** with the first engine that fits.

   OCR API (only after remote upload is approved):

   ```bash
   uv run "$PDF_TO_MD_SKILL/scripts/ocr_api_job.py" \
     /path/to/input.pdf --output-dir /path/to/output-dir \
     --base-url https://api.newlineages.com/ocr --allow-remote
   ```

   Without `--base-url`, the helper uses the local OCR host at
   `http://127.0.0.1:8002/ocr`. A non-local URL is rejected unless
   `--allow-remote` is present.

   LiteParse v2 fallback (no key required):

   ```bash
   uv run "$PDF_TO_MD_SKILL/scripts/liteparse_to_md.py" \
     /path/to/input.pdf --output-dir /path/to/output-dir
   ```

   If you used the LiteParse engine, **shape `<stem>.md` before continuing** —
   see "Shape the LiteParse output" below. The downstream steps only work as well
   as the Markdown they read.

2. **Build the section audit:**

   ```bash
   uv run "$PDF_TO_MD_SKILL/scripts/build_section_audit.py" /path/to/output-dir/<stem>.md
   ```

3. **Populate the first-pass article JSON** (also writes the audit):

   ```bash
   uv run "$PDF_TO_MD_SKILL/scripts/populate_article_json.py" /path/to/output-dir/<stem>.md
   ```

   This is a *first pass*. Review and complete fields the heuristics miss
   (authors with superscripts, methods, references, figure interpretation)
   against the Markdown and the article schema.

4. **Render figure pages** when figure/table captions are present, then fill
   `figure_interpretation` from captions plus the rendered pages:

   ```bash
   uv run "$PDF_TO_MD_SKILL/scripts/render_pdf_pages_to_png.py" \
     /path/to/input.pdf --output-dir /path/to/output-dir/figure_review
   ```

5. **Validate** against the schema and the section audit:

   ```bash
   uv run "$PDF_TO_MD_SKILL/scripts/validate_article_json.py" \
     /path/to/output-dir/<stem>.article.json \
     --scientific-paper \
     --section-audit /path/to/output-dir/<stem>.section_audit.json
   ```

   Resolve every reported error before stopping. A missing field that is
   genuinely absent from the source is fixed by confirming absence, not by
   inventing content.

You may also start Mode A from a Markdown file you already trust — skip step 1
and run steps 2–5 on that `.md`.

### Mode B — Any other PDF (fast Markdown)

One step, fully local, no key:

```bash
uv run "$PDF_TO_MD_SKILL/scripts/liteparse_to_md.py" \
  /path/to/input.pdf --output-dir /path/to/output-dir
```

Useful flags: `--no-ocr` (faster on text-based PDFs), `--ocr-server-url URL`
(higher-accuracy OCR server), `--target-pages "1-5,10"`, `--max-pages N`, and
`--password-env NAME` (read a protected document password without exposing it
in the process list). The converter detects the title and section headings from font
size and weight, filters page furniture (watermarks, running headers, repeated
footers), and reflows text into paragraphs — then **shape the result** (next section).

### Shape the LiteParse output (required when LiteParse is the engine)

LiteParse v2 gives a fast first draft. Because it is mechanical, you (the LLM
running this skill) must read `<stem>.md` against the rendered pages and bring it
into the right shape before treating the conversion as done. Do not hand back raw
script output. Fix what the heuristics cannot:

- **Title** — confirm `# ` is the real title, not a journal banner, DOI line, or
  "Downloaded from…" watermark; set it correctly if wrong or missing.
- **Headings** — promote section headings the font heuristic missed (`## Abstract`,
  `## Introduction`, `## Methods`, `## Results`, `## Discussion`, `## References`,
  etc.) and demote false positives; keep reading order.
- **Broken words** — rejoin words split mid-token (e.g. "Berke ley" → "Berkeley")
  and fix hyphenation that did not rejoin across line breaks.
- **Front matter** — reconstruct the author list and affiliations, which LiteParse
  often drops or scrambles around superscripts and email addresses.
- **Captions & tables** — keep one figure/table caption per block; rebuild simple
  tables that collapsed into runs of text.
- **Residual furniture** — delete any leftover running headers, page numbers, or
  license boilerplate the filter missed.
- **References** — ensure each reference is its own entry, not one merged blob.

For **Mode A**, after this Markdown cleanup run `populate_article_json.py`, then
complete every `article.json` field the first-pass heuristics leave empty
(`authors`, `affiliations`, `methods`, `references`, `figure_interpretation`) from
the shaped Markdown and rendered pages, so validation passes for the right reasons —
never by inventing content. For **Mode B**, the shaped Markdown is the deliverable.

## Quick Reference

| Task | Command |
|------|---------|
| Is there an OCR key? | Test `[ -n "${OCR_API_KEY:-}${NELLI_API_KEY:-}" ]` without printing it |
| Approved remote paper OCR | `ocr_api_job.py INPUT.pdf --output-dir DIR --base-url URL --allow-remote` |
| Paper, no key | `liteparse_to_md.py INPUT.pdf --output-dir DIR` |
| Any PDF, fast | `liteparse_to_md.py INPUT.pdf --output-dir DIR --no-ocr` |
| Section audit | `build_section_audit.py DIR/<stem>.md` |
| Article JSON | `populate_article_json.py DIR/<stem>.md` |
| Figure PNGs | `render_pdf_pages_to_png.py INPUT.pdf --output-dir DIR/figure_review` |
| Validate paper | `validate_article_json.py DIR/<stem>.article.json --scientific-paper --section-audit DIR/<stem>.section_audit.json` |

Commands resolve from `$PDF_TO_MD_SKILL`, which defaults to the shared installed skill directory.
`liteparse_to_md.py` and `render_pdf_pages_to_png.py` carry PEP 723 inline
dependencies (`liteparse`, `pypdfium2`) that `uv run` installs automatically; the
remaining scripts are standard-library only.

## Input Requirements

- A PDF, or a format LiteParse converts to PDF first (DOCX/PPTX/XLSX/ODT/CSV via
  LibreOffice; JPG/PNG/TIFF/etc. via ImageMagick).
- For Mode A from existing Markdown: a `.md` with a clear `# Title`, an
  author/affiliation block, recognizable section headings (Abstract, Introduction,
  Methods, Results, Discussion, Conclusion, References), and figure/table captions
  starting with `Fig.`/`Figure`/`Table`.
- For the OCR API engine: `OCR_API_KEY` or `NELLI_API_KEY`, plus `curl`.
- A writable `--output-dir` (keep it outside this repository).

## Output

- **Mode B:** `<stem>.md`, plus `<stem>.ocr.json` and `<stem>.job.json` provenance.
- **Mode A:** the above plus `<stem>.section_audit.json` and `<stem>.article.json`;
  optionally `figure_review/` PNGs. `csag-extraction` consumes `<stem>.md` and
  `<stem>.article.json`; everything else is provenance.
- The article JSON has exactly these keys: `title`, `authors`, `affiliations`,
  `abstract`, `main`, `methods`, `figure_legends` (list), `figure_interpretation`,
  `references` (list). See `references/article_schema.md` and `references/article.yaml`.

## Quality Gates

- The conversion engine is **LiteParse v2** (or the OCR API); `<stem>.job.json`
  records `tool_version` 2.x for the LiteParse engine.
- When LiteParse was the engine, the Markdown has been **shaped** (title, headings,
  rejoined words, front matter, captions, references) — not handed back raw.
- Mode B Markdown is non-empty, has a sensible `#` title (or none, never a
  watermark), and is free of repeated page furniture.
- Mode A: `validate_article_json.py --scientific-paper` returns `OK`.
- `title`, `authors`, and `main` are populated for a real paper, or their absence
  is confirmed against the source (do not fabricate).
- When figure/table captions exist, `figure_legends` is populated and
  `figure_interpretation` is filled (or an explicit no-interpretation note is
  recorded).
- Provenance (`<stem>.job.json`) records the engine, tool version, and OCR setting.
- No test inputs or outputs are written inside this repository.
- The local paper-bundle fixture proves section audit, schema population, figure-legend handling, and scientific-paper validation; its missing-author companion proves absent metadata is rejected rather than invented.

## Examples

Fast Markdown from a non-paper PDF:

```bash
uv run "$PDF_TO_MD_SKILL/scripts/liteparse_to_md.py" report.pdf --output-dir /tmp/out --no-ocr
# -> /tmp/out/report.md  (+ report.ocr.json, report.job.json)
```

Full paper bundle with no OCR key (LiteParse v2 engine):

```bash
DIR=/tmp/paper
uv run "$PDF_TO_MD_SKILL/scripts/liteparse_to_md.py" paper.pdf --output-dir "$DIR"
uv run "$PDF_TO_MD_SKILL/scripts/populate_article_json.py" "$DIR/paper.md"
uv run "$PDF_TO_MD_SKILL/scripts/validate_article_json.py" \
  "$DIR/paper.article.json" --scientific-paper \
  --section-audit "$DIR/paper.section_audit.json"
```

## Troubleshooting

- **`liteparse is not installed`**: run the script itself with `uv run "$PDF_TO_MD_SKILL/scripts/liteparse_to_md.py"` (not `uv run python ...`) so uv reads the PEP 723 dependency.
- **`pdf-to-md requires LiteParse v2`**: run the PEP 723 script directly with uv; it pins `liteparse>=2,<3` without modifying system Python.
- **Title is a journal banner, watermark, or "Downloaded from…" line**: the converter filters furniture and repeated headers; if one slips through, remove it in the Markdown before step 2, or note that `article_extraction` re-derives the title from the body.
- **`authors`/`methods`/`references` empty on a real paper**: the first-pass heuristics miss superscript-heavy author lines and short note formats. Fill them by hand from the Markdown; this is expected, not a converter failure.
- **Scanned/image-only PDF gives little text**: keep OCR enabled (default) and raise `--dpi`, or point `--ocr-server-url` at EasyOCR/PaddleOCR; for best fidelity use the OCR API engine.
- **`Missing OCR API key`**: set `OCR_API_KEY`/`NELLI_API_KEY`, or use the LiteParse v2 engine instead.
- **Garbled equations or merged columns**: LiteParse is the fast path; for layout-heavy papers prefer the OCR API engine.