Souffle Datalog

---
name: souffle-datalog
description: For reading, writing, debugging, or extending Soufflé Datalog (`.dl`) programs in this repo — the layered ontology QC program under `src/util/qc/`, the `basic-cycles.tsv` cycle check, taxon-constraint materialization, and the relation-diff CI report. Covers the RDF fact-file format these programs consume, this repo's Soufflé conventions, the check registry, and the required control-ontology self-test for any new or changed rule.
---

Soufflé is a Datalog engine (<https://souffle-lang.github.io/>). GO uses it for
graph queries that are awkward in SPARQL and too slow in OWL reasoning:
transitive closure over the class graph, cycle detection with path
reconstruction, and taxon-constraint propagation.

The language is small but has several traps that produce **silently empty or
silently wrong results rather than errors**. Read `reference.md` in this skill
directory before writing rules — especially the `match` and aggregate sections.

## Where Soufflé is used

| Program | Invoked by | Reads | Writes |
| --- | --- | --- | --- |
| `src/util/ontology-qc.dl` (+ `src/util/qc/`) | `make datalog-check`, part of `test` and `travis_test` | `go-edit.facts`, `../resources/obsolete_ec.txt` | `datalog-violations.tsv` |
| `src/util/ontology-qc-selftest.dl` | `make qc-selftest`, part of `test` and `travis_test` | `src/util/qc/control.dl` | `qc-missing-expected.tsv`, `qc-unexpected-violations.tsv`, `qc-unexpected-checks.tsv`, `qc-unregistered-checks.tsv` |
| `src/util/cycles.dl` | `make basic-cycles.tsv` | `basic.facts` | `cycle.csv` → `basic-cycles.tsv` |
| `src/util/materialize-taxon-constraints.dl` | `make imports/go-computed-taxon-constraints.owl` | `taxonconstraintsrelationgraph.facts` (from relation-graph) | `computed_only_in_taxon.csv`, `computed_never_in_taxon.csv` |
| `src/util/relation-diff.dl` | `.github/workflows/relation-diff.yml` (runs from the repo root) | `left.facts`, `right.facts`, `ontrdf.facts` | `lost.csv`, `gained.csv`, `fill.csv` |

`datalog-check`, `qc-selftest`, and `basic-cycles.tsv` are QC gates: **the build
fails if the output file is non-empty.**

## The fact-file format

`go-edit.facts` and `basic.facts` are generated by `src/util/ontology-to-facts.sc`,
which renders the ontology as N-Triples and replaces the two field separators with
tabs, dropping the trailing ` .`. The other fact files reach the same format via
equivalent `sed` pipelines (`Makefile:457`, `relation-diff.yml:13,28,41`). So a
fact file is three tab-separated columns
of **raw N-Triples terms, with their syntax intact**. This is the single thing
to internalize:

| RDF term | Appears in the facts file as |
| --- | --- |
| IRI | `<http://purl.obolibrary.org/obo/GO_0008150>` — angle brackets included |
| Plain literal | `"neuron differentiation"` — double quotes included |
| Typed literal | `"true"^^<http://www.w3.org/2001/XMLSchema#boolean>` |
| Language literal | `"foo"@en` |
| Blank node | `_:genid1234` |

Consequences that bite:

- **Comparisons must include the punctuation.** `o = "true"` never matches; the
  boolean-true object is
  `"\"true\"^^<http://www.w3.org/2001/XMLSchema#boolean>"` in Soufflé source.
  Compare with the `obsolete` rule in `qc/vocabulary.dl`.
- `match("<.+>", x)` is the repo idiom for "x is an IRI" — it excludes blank
  nodes and literals. `match("\".*", o)` is the idiom for "o is a literal".
  Restriction bnodes are everywhere in the class graph, so these guards are not
  optional.
- Resource files compared against fact values must carry the same quoting.
  `src/resources/obsolete_ec.txt` stores `"EC:1.1.1.1"` **with** the quote
  characters for exactly this reason.
- **Imports are not followed.** `ontology-to-facts.sc` maps every import to
  /dev/null, so `go-edit.facts` contains the axioms of `go-edit.obo` alone — no
  CHEBI, no NCBITaxon, no imported relations. A rule that expects an imported
  label or hierarchy will find nothing.
- Axiom annotations are reified. Definition and synonym xrefs are reachable only
  via `owl:Axiom` / `owl:annotatedSource` / `owl:annotatedProperty`, as the
  `annotated`, `definition_xref` and `synonym_xref` rules in `qc/vocabulary.dl`
  show.

Declare IRIs as `#define` constants at the top of the file, as every existing
program does, rather than repeating string literals.

## Running it

Fact-file generation and the QC targets are Makefile recipes, so per CLAUDE.md
and the `/odk-make` skill they run in the pinned ODK image:

```bash
.claude/skills/odk-make/odk-run.sh make datalog-check
```

For iterating on a rule, a host `souffle` against a small synthetic fact file is
far faster and touches no ontology tooling. Host and ODK builds agree on
everything documented in this skill, down to the exact error text; they differ
only in word size (`number` is 64-bit in ODK, 32-bit in a stock Homebrew build).
Run the final check through `odk-run.sh` regardless — that is what CI runs.

Soufflé resolves `.input`/`.output` paths against the **current working
directory** unless `-F`/`-D` are given, which is why the Makefile runs
`souffle ../util/ontology-qc.dl` from `src/ontology`. Outputs land as
`<relation>.csv` and are **tab-separated despite the extension**; the recipes
rename and re-head them afterwards.

## The QC program's structure

`ontology-qc.dl` is an entry point only. The program is split so that no check
has to touch RDF:

| File | Holds |
| --- | --- |
| `src/util/qc/rdf.dl` | N-Triples punctuation — quote stripping, datatype tails, IRI→CURIE. The only file allowed to mention that syntax. |
| `src/util/qc/vocabulary.dl` | The ontology as an editor talks about it: `class`, `obsolete`, `label`, `definition`, `synonym`, `xref`, `parent`, `relationship`, `merged_term`. |
| `src/util/qc/registry.dl` | `check`/`violation`/`error`, plus `unregistered_check`. |
| `src/util/qc/selftest.dl` | The self-test gates. Included only by the self-test entry point — in the build program its relations would have no facts, and that warning is this repo's best typo signal. |
| `src/util/qc/checks/*.dl` | One file per topic. |
| `src/util/qc/program.dl` | The include list, shared by both entry points. |
| `src/util/qc/control.dl` | The control ontology for the self-test. |

`#include` resolves **relative to the including file**, not the working
directory, so `souffle ../util/ontology-qc.dl` from `src/ontology` works
unchanged. Both entry points include `qc/program.dl`, so the build check and
the self-test can never disagree about which checks exist.

Every check registers itself and reports through `violation/3`:

```
check("unknown-ec", "An EC xref must cite an EC entry that exists").
violation("unknown-ec", t, cat("has ", kind, " to an unknown EC: ", e)) :-
    ec_citation(t, kind, e), !known_ec(e).
```

`registry.dl` projects that into the three-column `error` relation the build
reads — check name, the subject as a curator would read it (CURIE plus label),
and the message. Write the message as the predicate of a sentence whose subject
is the term; `described/2` supplies the subject.

Pass `violation/3` the **raw** node in the term position and let the registry
render it. `shown/2` is the manual version, for a node interpolated mid-sentence
— `shown(p, prop)` in the `empty-literal` rule turns the offending property into
readable text. Reach for it only where the node is known to be an IRI or a
literal, because it silently drops anything else; see the Gotchas below.

To add a check:

1. Pick or add a topic file under `src/util/qc/checks/`.
2. Write one `check(name, summary)` fact and one or more `violation` rules
   against the vocabulary layer. If a rule needs `ontrdf/3` or an escaped
   literal, add the missing concept to `vocabulary.dl` instead — every other
   check gets it too.
3. Plant a violating term in `src/util/qc/control.dl`.
4. Exclude obsolete terms with `live_class(t)` unless the check is specifically
   about obsoletes; exclude `merged_term(t)` for term-level requirements.

A relation declared in two included files is a hard compile error naming both
files and lines, so a split tree cannot silently collide.

## Required: positive-control test

A wrong QC rule produces an **empty** relation, and an empty relation means the
build passes. A check that never fires is indistinguishable from a clean
ontology. So a new or modified rule is not done until it has been shown to fire.

For `ontology-qc.dl` this is mechanised. `src/util/qc/control.dl` is a small
hand-built ontology that plants a violation of every *arm* of every registered
check and declares each one with an `expected_violation(check, term)` fact.
`make qc-selftest` runs the identical check set against it:

```bash
.claude/skills/odk-make/odk-run.sh make qc-selftest
```

Four gates, all of which must come out empty:

| Gate | Means |
| --- | --- |
| `unregistered_check` | A `violation` under a name no `check` declares — a typo. |
| `missing_expected` | An expectation that did not fire — the rule, or one arm of it, is dead. |
| `check_without_expectation` | A registered check with nothing planted for it. |
| `unexpected_violation` | A violation nobody expected — a rule has grown too broad. |

**The gate is keyed on `(check, term)`, not on the check name.** That matters:
several checks have more than one rule — `obsolete-reference` has six arms —
and a name-keyed gate stays satisfied as long as any one arm fires, so a dead
arm hides behind a live one. Give every arm its own control term. The last gate
is what lets the control ontology assert that its well-formed terms trigger
*nothing*, which catches a rule that has become too broad rather than too
narrow. The run takes well under a second.

The control ontology is written as Datalog facts rather than a `.facts` file
because Soufflé's fact reader rejects comment lines. It uses raw triples rather
than OBO on purpose: an OBO round-trip normalises away several of the
malformations being tested.

For the other programs, which have no registry, build a handful of synthetic
facts containing a known violation and run the real program against them:

```bash
mkdir -p /tmp/dl/ontology /tmp/dl/resources && cd /tmp/dl/ontology
printf '"EC:1.1.1.2"\ttrue\n' > ../resources/obsolete_ec.txt
printf '<http://purl.obolibrary.org/obo/GO_0000001>\t<http://www.w3.org/1999/02/22-rdf-syntax-ns#type>\t<http://www.w3.org/2002/07/owl#Class>\n' > go-edit.facts
souffle /path/to/repo/src/util/cycles.dl && cat cycle.csv
```

Confirm both directions: the violating fact **does** produce the expected
message, and a corrected version of the same fact produces **none**.

Then run the real target through `odk-run.sh` and check the violation count
against the full ontology. A new check that fires on thousands of existing terms
is a signal about the check, not about the ontology — bring the count to the
issue before committing.

## Gotchas

- **`match` is a full-string match, not a search.** `match("GO_0005", iri)` is
  false for every real IRI. Terminate patterns with `.*`, or use `contains`.
  This is the most common cause of a rule that silently matches nothing.
- **`Warning: No rules/facts defined for relation X` exits 0.** It is the
  clearest sign of a typo'd relation name. Read Soufflé's stderr; never pass
  `-w`.
- **Regex backslashes must be doubled** — `"\\d"`, not `"\d"`, which is a
  compile error.
- **Aggregates need the grouping key grounded outside the aggregate**, or you
  get the global aggregate with no warning. See `reference.md`.
- **`substr` is 0-based** and clamps or empties on out-of-range input instead of
  failing.
- **A guard constraint beside a `substr` does not protect it.** Soufflé may
  evaluate the `substr` before the constraint that was written to keep its index
  in range, producing a run full of range warnings. Clamp inside the call —
  `substr(l, max(0, n-k), k)`, not `substr(l, n-k, k), n-k >= 0`.
- **`shown/2` is a filter as well as a renderer.** Its four arms cover OBO
  CURIEs, OBO fragments, bare IRIs and literal text — a blank node matches none
  of them, so `shown(x, txt)` in a rule body drops every bnode binding with no
  error. An empty literal on a reified axiom annotation has a `_:genid` subject,
  so pre-rendering a subject through `shown` rather than handing `violation/3`
  the raw node loses exactly those cases. The same applies to `described/2`,
  which is why `registry.dl` carries a `!described(t, _)` fallback arm.
- **Restrictions are blank nodes, so an `iri()` guard hides them.** A rule that
  walks `rdfs:subClassOf` and filters targets to IRIs skips every
  `owl:Restriction`; anything built on top of it is silently empty. Keep the
  unfiltered relation and project the named-class view from it, as
  `superclass_expr`/`parent` in `vocabulary.dl` do.
- **`#define` does not concatenate adjacent string literals.** C's
  `"a" "b"` → `"ab"` is a preprocessor-plus-C-lexer behaviour that Soufflé's
  lexer does not share; `#define GO(n) "<...GO_" n ">"` is a syntax error at the
  use site. Give each constant its own full-string `#define`.
- **Negation cannot appear inside recursion** (stratification), and negated
  literals ground nothing.
- **Don't commit generated artifacts.** `*.facts`, `*.csv`, `datalog-violations.tsv`,
  and `computed_*_taxon.*` are build outputs.

## Further detail

`reference.md` in this directory: full functor and constraint tables, aggregate
syntax, I/O directive options, CLI flags, and a table of Soufflé's error and
warning messages with fixes, verified against both the host and ODK builds.