Portaljs Check Data Quality

---
name: portaljs-check-data-quality
description: Audit a local or remote tabular file (CSV/TSV) for common data quality issues — schema, nulls, types, duplicates. Read-only. Use when a dataset needs a quality check before publishing, or a showcase renders wrong (blank cells, garbled numbers, an unsortable date column) and the cause needs isolating.
allowed-tools: Bash(curl:*), Bash(awk:*), Bash(sort:*), Bash(head:*), Bash(wc:*)
version: 1.0.0
author: Datopian <hello@datopian.com>
license: MIT
compatibility: Claude Code with PortalJS portals (Next.js 14, React 18, Node 18+). Runs from any project via the plugin, a personal ~/.claude/commands install, or a portaljs clone.
tags:
  - portaljs
  - data-portal
  - data-quality
  - audit
  - csv
  - validation
---

# PortalJS — Check Data Quality

## Overview

Run a read-only quality audit of one CSV or TSV file, local or remote, and return a
structured JSON report. The audit profiles every column — null/blank counts, inferred
value types, numeric ranges, likely year/date fields — and flags duplicate rows,
duplicate values in identifier-like columns, ambiguous overlapping year columns (e.g.
`calendar year` vs `fiscal year`), and mixed-type columns. It never edits the source
file, `datasets.json`, or any other project file; it only reads the target file (a
remote URL is downloaded to a temp file that is deleted before the run ends) and
prints a report. Use it before publishing a dataset with `portaljs-add-dataset`, or to
diagnose why a showcase renders wrong.

## Prerequisites

- `python3` on `PATH` — the audit logic runs as an embedded Python script; nothing is
  installed.
- One CSV or TSV file, given as a local path or an `http`/`https` URL. Only one file
  per run.

## Instructions

The canonical, full step-by-step workflow is
[`.claude/commands/portaljs-check-data-quality.md`](https://github.com/datopian/portaljs/blob/main/.claude/commands/portaljs-check-data-quality.md) —
the single source of truth. Read and follow it when executing. Summary:

1. Gather input — the file path or URL to audit. If missing, ask for it; never dead-end.
2. Resolve the source: if it's an `http`/`https` URL, download it to a temp file first;
   otherwise use the local path as given.
3. Validate the extension is `.csv` or `.tsv`. If not, or the file is missing, or the
   header row is empty, stop and surface the error JSON as-is — do not guess a fix.
4. Profile every column: null/blank counts, distinct values, sample values, inferred
   per-value type (boolean/integer/float/date/string), numeric min/max, and year
   range for columns whose name looks year-like.
5. Derive findings from the profiles — duplicate rows, missing-value ratios, invalid
   year values, mixed types, suspect negative values, duplicate identifier values, and
   ambiguous overlapping year columns — each tagged `critical`, `warning`, or `info`.
6. Assemble the JSON report (`status`, file metadata, `findings`, `recommendations`,
   `column_profiles`), print it, and clean up the temp file if one was created.
7. Relay the report to the user as-is; do not modify the source file, `datasets.json`,
   or any other project file based on the findings — that's a separate, explicit step.

## Output

A single JSON object printed to stdout:

- `status` — `ok`, `warning`, or `critical`.
- `file`, `file_name`, `source_type` (`local` or `url`), `row_count`, `column_count`.
- `findings` — structured issues, most severe first.
- `recommendations` — de-duplicated suggested next steps.
- `column_profiles` — per-column summary (nulls, blanks, distinct count, sample
  values, inferred types, numeric/year ranges).

No files are created or modified. A remote URL's temp download is removed on exit,
success or failure alike.

## Error Handling

| Symptom | Cause | Fix |
| --- | --- | --- |
| `"File ... is not available."` | Local path is wrong, or the URL download failed | Verify the path or URL is reachable and retry. |
| `"Only CSV and TSV files are supported right now."` | File extension isn't `.csv`/`.tsv` | Convert the file, or point to its tabular source instead. |
| `"... does not contain tabular headers."` | File is empty or the header row is malformed | Open the file and confirm it has a valid, non-empty header line. |
| Command hangs on a URL | Remote host is slow or blocks non-browser requests | Download the file manually and audit the local copy instead. |
| `python3: command not found` | Python 3 isn't installed or not on `PATH` | Install Python 3, or run the audit where it's available. |
| Report looks truncated in the terminal | Large report wrapped/paginated by the shell | Redirect to a file (`> report.json`) and open it separately. |

## Examples

### Example 1 — Audit a local CSV before publishing

```
/portaljs-check-data-quality ./public/data/trash.csv
```

### Example 2 — Audit a remote CSV over HTTPS

```
/portaljs-check-data-quality https://example.com/trash.csv
```

### Example 3 — Audit a TSV and save the report for review

```bash
bash scripts/check-data-quality.sh ./data/emissions.tsv > /tmp/emissions-quality.json
```

### Example 4 — Read a `critical` status report

```json
{
  "status": "critical",
  "findings": [
    { "severity": "critical", "check": "duplicate_rows", "message": "42 duplicate rows found." }
  ],
  "recommendations": ["Review and deduplicate repeated rows if they are not intentional."]
}
```
Fix the flagged rows/columns, then re-run the audit before publishing.

## Resources

- Full workflow: [`.claude/commands/portaljs-check-data-quality.md`](https://github.com/datopian/portaljs/blob/main/.claude/commands/portaljs-check-data-quality.md)
- Detailed check catalog and troubleshooting: [`references/reference.md`](references/reference.md)
- Related skills: `portaljs-add-dataset`, `portaljs-define-schema`
- Python `csv` module (parsing behavior this audit relies on): <https://docs.python.org/3/library/csv.html>