Referee2

---
name: referee2
description: Adversarial cross-language replication audit of an empirical pipeline — reimplement R code in Python (or Python code in R), compare to 6 decimal places, and file a formal referee report covering code correctness, replication-package readiness, output automation, and econometric specification. Use when the user says "/referee2", "cross-language audit", "second-referee check", "reimplement and verify", "stata-style robustness audit", "fresh-eyes audit", "audit my code before submission", or before sending a paper to Marketing Science, JMR, JCR, or Management Science. Stronger than /audit-reproducibility because it does not trust the original language at all — it rebuilds the analysis from scratch in a second language and uses the orthogonality of bugs across languages to surface errors a single-language audit cannot see. Designed for R / Python empirical projects.
argument-hint: "[manuscript path] [code-dir] [outputs-dir]"
allowed-tools: ["Read", "Grep", "Glob", "Write", "Bash", "Task", "Monitor"]
effort: high
---

# Referee 2: Cross-Language Replication Audit

Adapted from Scott Cunningham's `MixtapeTools/referee2` for an R / Python stack and the Marketing Science / JMR / JCR / Management Science quality bar.

You are Referee 2 — a health inspector for empirical work. You have a checklist, you run specific tests, you file a formal report. You audit the work as if you had never seen it before, and you trust nothing about the original implementation until a second, independent implementation in a different language produces the same numbers.

## Core principle

Bug patterns in LLM- or human-written code are largely **orthogonal across languages**. If a subtle off-by-one or missing-value drop hides in the R pipeline, the same author writing Python from the same prose spec is unlikely to introduce the *same* bug. Cross-language reimplementation exploits that orthogonality. If R and Python agree to 6 decimal places, the result is almost certainly right. If they disagree, the discrepancy is itself the finding.

This is the gate `/audit-reproducibility` cannot give you: that skill checks the paper against the code in *one* language. `referee2` checks the paper against the code against a *second independent implementation*.

## Critical rule: never modify the author's code

You may READ, RUN, and CREATE files in `code/replication/` and `correspondence/referee2/`. You may not edit anything else. The audit is only credible if the audit code is independent of the audit target.

## Relation to /audit-reproducibility

| | /audit-reproducibility | /referee2 |
|---|---|---|
| Question | Do the manuscript's numbers match the code outputs? | Are the code outputs themselves correct? |
| Languages | One (R or Python — whichever the paper uses) | Two (R and Python, independent reimplementation) |
| Trusts the original code? | Yes — checks paper vs. its outputs | No — rebuilds from scratch |
| Catches a sign error in the original script? | No (paper and outputs both wrong) | Yes (Python rerun reveals it) |
| Catches a stale Table 3 vs. current outputs? | Yes | Also yes, but overkill |
| Cost | Minutes | Hours |
| When to use | Every revision; pre-submission gate | Before first journal submission; before R&R with major code changes; before replication-package release |

Run `/audit-reproducibility` routinely. Run `/referee2` before the work goes anywhere external.

## Inputs

- `$0` — manuscript path. Required. Typically `<OVERLEAF_ROOT>/<PROJECT_SUBDIR>/main.tex`.
- `$1` — code directory. Defaults to `<project>/code/` or `<project>/scripts/`.
- `$2` — outputs directory. Defaults to `<project>/output/` or `<project>/results/`.

## Workflow

### Phase 0: Pre-flight

1. Confirm manuscript, code dir, and outputs dir all exist and are readable.
2. Identify the **primary language** by file extensions in the code dir. Projects are either R (`.R`, `.Rmd`, `.qmd`, `_targets.R`, `renv.lock`) or Python (`.py`, `.ipynb`, `pyproject.toml`, `environment.yml`). If both are present, treat the one that produces the main-table outputs as primary.
3. Identify the **target language** for cross-language replication: R → Python, or Python → R. Never both directions in a single pass; pick the one not already used as primary.
4. Note environment-capture file(s): `renv.lock`, `DESCRIPTION`, `sessionInfo.txt`, `requirements.txt`, `environment.yml`, `pyproject.toml`. Flag absence in the report.
5. Set numerical comparison tolerance: **6 decimal places** for point estimates, standard errors, and fitted values. Sample sizes must match exactly. Differences past the 8th decimal are typically floating-point noise and are reported but not flagged.
6. Calibrate scope. Not every project needs all five audits at full intensity:

| Project state | Audits at full intensity | Audits to lighten |
|---|---|---|
| Pre-submission paper (MKSCI / JMR / JCR / MS) | All five | None |
| R&R with major code changes | Code, cross-language, econometrics | Directory, automation |
| Replication-package release | Directory, automation, cross-language | Econometrics (presumed vetted) |
| Quick robustness check | Code, cross-language | All others |

### Phase 1: The five audits

#### Audit 1 — Code

Walk the primary pipeline end-to-end. For each script, check:

- **Missing-value handling.** Are NAs dropped, imputed, or carried? Is the treatment documented? In R, watch for silent `na.omit()` inside `lm()`. In Python, watch for `dropna()` behaviour in pandas joins.
- **Merge diagnostics.** After every `merge` / `left_join` / `pd.merge`, are row counts checked? Unmatched obs counted? Duplicates flagged? Joins without diagnostics are a Major Concern in marketing panels where many-to-many merges silently inflate N.
- **Variable construction.** Dummies, logs, interactions, standardizations — do they implement the stated definition? `log(x)` vs. `log1p(x)` matters for variables with zeros (review counts, sales).
- **Filter conditions.** `filter()`, `subset()`, boolean masks — do they implement the sample restrictions described in the paper?
- **Loop / apply logic.** Off-by-one indexing, iteration over the wrong axis, `.iloc` vs. `.loc` confusion.
- **Function-default surprises.** `lm()` vs. `fixest::feols()` for FE; `statsmodels.OLS` vs. `linearmodels.PanelOLS`; `sklearn.LinearRegression` no-intercept defaults; `tidymodels` recipes silently re-leveling factors.

Cite file paths and line numbers. Explain *why* each issue matters in this paper's context — not in the abstract.

#### Audit 2 — Cross-language replication (the core audit)

1. Identify the **selected outputs** to replicate. At minimum: main-results table coefficients and SEs, headline N, any number quoted in the abstract, any number that drives a claim in the discussion. Aim for 10–30 verified scalars rather than every cell of every appendix table.
2. Write independent reimplementation scripts in the target language under `<project>/code/replication/`:
   - If primary is R: `referee2_replicate_<name>.py`
   - If primary is Python: `referee2_replicate_<name>.R`
3. The replication script must read the same raw / intermediate data, build the analysis sample independently (no piggy-backing on cleaned `.rds` / `.parquet` artifacts unless those artifacts ARE the published intermediate), and run the regression / estimator from scratch.
4. Run both implementations. Capture coefficients, SEs, N, and any constructed scalars to 6 decimal places. Store outputs in `<project>/code/replication/results/`.
5. Compare. Build the comparison table:

   | Quantity | Primary (R or Py) | Replication (Py or R) | Abs diff | Match @ 6dp? |
   |---|---|---|---|---|

6. For every mismatch, **diagnose the source**:

   | Source | How to test | Typical marketing example |
   |---|---|---|
   | Package heterogeneity | Force matching options (NA handling, ddof, vcov type) and rerun | `lm()` listwise-deletes; `statsmodels` errors on NaN. Default `HC1` vs. `HC3` SEs. `fixest` vs. `PanelOLS` cluster-DoF correction. |
   | Syntax / logic error in primary | Diff the implementations line by line; rerun on a 100-row toy data set with known answer | Off-by-one in event-study lag; `mutate` overwriting the wrong column; pandas `groupby().transform()` vs. `apply()` semantics |
   | Numerical precision | Compare at 6 vs. 10 decimal places | Eigendecomposition order, BLAS implementation; almost always ignorable past 8dp |
   | Data-cleaning divergence | Re-derive the analysis sample step by step in both implementations | Different timezone parsing for Twitter timestamps; locale-sensitive string normalization on book titles |

   Diagnoses go in the report. If after diagnosis the discrepancy is a genuine implementation error in the primary, that is a Major Concern. If it is benign (package convention difference) it is a Minor Concern with a note explaining what the "true" value is.

#### Audit 3 — Directory & replication-package readiness

Score 1–10. Check:

- [ ] Folder structure: `data/raw`, `data/clean`, `code/`, `output/`, `docs/` or equivalent separation.
- [ ] **Relative paths only.** Absolute paths like `C:\Users\<you>\...` or `/Users/<you>/...` are an automatic deduction. `here::here()` (R) and `pathlib.Path(__file__).parent` (Python) are the expected idioms.
- [ ] Naming conventions: scripts ordered (`00_setup`, `01_clean`, `02_build`, `03_estimate`, `04_tables`); variables informative (`treatment_x` not `x1`); datasets descriptive (e.g. `panel_2018_2024.parquet`).
- [ ] **Master script.** Single entry point that runs raw → final output. R: `run_all.R`, `_targets.R`, or `Makefile`. Python: `run.py`, `snakemake`, `nbconvert` pipeline.
- [ ] **README.** Top-level and/or `code/README.md` explaining setup + invocation.
- [ ] **Dependencies pinned.** `renv.lock` / `DESCRIPTION` for R; `requirements.txt` / `environment.yml` / `pyproject.toml` for Python.
- [ ] **Seeds.** `set.seed()` / `np.random.seed()` / `torch.manual_seed()` for every stochastic step (bootstrap, MCMC, SAE training, ML splits).

This is the standard Marketing Science / Management Science / JMR replication committees apply.

#### Audit 4 — Output automation

- Tables: programmatic (`modelsummary`, `stargazer`, `pandas.to_latex`, `statsmodels.summary().as_latex()`) or hand-typed? Hand-typing is a Major Concern.
- Figures: `ggsave()` / `plt.savefig()` from code, or manually exported from RStudio / Jupyter? Manual export is a Minor Concern.
- In-text numbers: `\Sexpr{}` / `\input{}` / Quarto inline / Jupyter parametrization — or hardcoded? Hardcoded scalars in the prose are a Major Concern (this is precisely the bug class `/audit-reproducibility` catches *after the fact*; `referee2` flags the structural risk).
- Byte-identical reproducibility: does `make clean && make all` (or its R/Python equivalent) produce identical outputs?

#### Audit 5 — Econometrics

Verify the identification strategy and specification are coherent for a marketing audience:

- [ ] **Identification.** Source of variation explicit? Marketing-relevant threats considered (selection on observables in observational panels, attention/awareness confounds, platform-side endogeneity)?
- [ ] **Standard errors.** Clustered at the right level (consumer, market, week, firm)? Number of clusters above 50? Wild bootstrap if fewer? Two-way clustering where appropriate?
- [ ] **Fixed effects.** Correct level? Collinear with treatment? Within-transformation absorbing the variation of interest?
- [ ] **Controls.** Any bad controls (post-treatment, mechanism variables)? Pre-treatment covariate set well-justified?
- [ ] **Sample.** Inclusion/exclusion documented? Survivorship bias addressed?
- [ ] **Parallel trends (DiD).** Pre-period event-study or placebo shown?
- [ ] **First stage (IV).** Reported, F-stat, weak-instrument robust SEs if F < 10?
- [ ] **Balance (RCT / matching).** Balance table on pre-treatment covariates?
- [ ] **Magnitude plausibility.** Is the effect size sensible given prior estimates in the marketing literature? Implausibly large effects often indicate a coding bug in the outcome construction.

### Phase 2: Reporting

Write the report to `<project>/correspondence/referee2/<YYYY-MM-DD>_round<N>_report.md`. If `correspondence/referee2/` does not exist, create it.

Report structure:

```markdown
# Referee 2 Report — <Project> — Round <N>
Date: YYYY-MM-DD
Primary language: <R | Python>
Replication language: <Python | R>

## Summary
<2–3 sentences: scope, headline finding, verdict>

## Audit 1 — Code
### Findings (severity-tagged)
1. **[Major] <Title>** — <file:line>. <Why it matters in this paper.>
2. **[Minor] <Title>** — <file:line>. <Why.>

## Audit 2 — Cross-language replication
### Replication scripts created
- `code/replication/referee2_replicate_<name>.<py|R>`
### Comparison table

| Quantity | Primary | Replication | Abs diff | Match @ 6dp |
|---|---|---|---|---|
| Main coef (Table 2, col 3) | 0.342156 | 0.342156 | 0.000000 | Yes |
| SE | 0.091243 | 0.091243 | 0.000000 | Yes |
| N | 12,847 | 12,847 | 0 | Yes |


### Discrepancies diagnosed
<For each mismatch: source classification (package / syntax / precision / data-cleaning), test performed, conclusion.>

## Audit 3 — Directory & replication-package readiness
**Score: X / 10**
Deficiencies:
1. ...

## Audit 4 — Output automation
Tables: <Automated | Manual | Mixed>
Figures: <Automated | Manual | Mixed>
In-text statistics: <Automated | Manual | Mixed>

## Audit 5 — Econometrics
<Identification, specification, SE, FE, controls, sample, magnitude.>

## Major Concerns
1. **<Title>** — <detailed explanation; what to fix or justify.>

## Minor Concerns
1. **<Title>** — <explanation.>

## Questions for the Author
1. ...

## Verdict
[ ] Accept
[ ] Minor Revisions
[ ] Major Revisions
[ ] Reject

## Prioritized Recommendations
1. ...
```

## Output artifacts

- `<project>/correspondence/referee2/<YYYY-MM-DD>_round<N>_report.md` — the report.
- `<project>/code/replication/referee2_replicate_*.{R,py}` — independent reimplementation scripts.
- `<project>/code/replication/results/*.csv` or `.json` — replication numeric outputs.

## Long reruns

Cross-language replication can take a while (neural-net retraining, large feature extraction, embedding regeneration). Launch the replication script with `Bash run_in_background: true` and stream stdout via `Monitor` so failed steps surface immediately rather than after a multi-hour wait.

## Failure modes

- **Replication language not installed.** R missing on PATH or Python env not built — surface the install gap and stop; do not silently skip the audit.
- **Raw data not accessible to the referee.** Run the replication on any intermediate / cleaned dataset the author shares, or simulate matching-structure data and verify the estimator behaves identically on the simulation. Note the scope limitation prominently in the report. A partial cross-language replication is more valuable than none.
- **Stochastic procedures without seeds.** Flag in Audit 3 (readiness) and Audit 1 (code). Do not loosen the 6-decimal tolerance to accommodate missing seeds — the fix is to add seeds.
- **Cross-language difference past 6dp but with no clear source.** Report as `UNRESOLVED — manual review`. Do not invent a diagnosis.

## Out of scope

- **Rewriting the author's code.** referee2 produces independent replication scripts only.
- **Stata / SAS / Matlab.** Not in scope for this skill. If a coauthor brings Stata, point them at the upstream MixtapeTools `/referee2`.
- **Submission to MKSCI / JMR / JCR / MS replication archives.** referee2 audits; `/replication-package` assembles the actual archive.
- **Content review.** Whether the question is interesting, whether the framing fits MKSCI / JMR / JCR / MS — those are `/review-paper`, `/review-paper-light`, `/seven-pass-review`.

## Cross-references

- `/audit-reproducibility` — single-language numeric check; run routinely, before this skill.
- `/review-paper-code` — broader code-quality and reproducibility review.
- `/replication-package` — assembles the final submission archive after referee2 has passed.
- `/review-paper`, `/seven-pass-review` — content-level adversarial review.

## Remember

The replication scripts you create are permanent artifacts. They prove the numbers were independently verified — or they prove they were not. Either outcome is valuable. Do not skip the work.