Uxr Synthetic User Check

---
name: uxr-synthetic-user-check
description: Runs a Synthetic User Check that splits "synthetic user" or AI persona answers into claims, matches each to real evidence, keeps what may stay a hypothesis, rejects what was passed off as a finding and names the real study that would answer it. Use for "run uxr-synthetic-user-check", "just ask the AI personas", "synthetic users", "AI persona interviews", "can we use synthetic research", "check these AI user answers", "someone put synthetic quotes in the deck", "synthetic participants", part of the UX Research with Claude Pack by Polar Bear.
---

# Synthetic User Check

## When To Use
Someone says "just ask the AI personas" or pastes synthetic interview answers into the deck. Run it before those answers shape a decision or a readout. It answers: which of these claims does real evidence support, which does it contradict, which have none, and what real study would settle the rest?

## When Not To Use
If the deck or report is finished and you need every claim in it traced, synthetic or not, run Evidence Trace Audit. If no synthetic answers exist yet and the team wants them, do not create them: list the beliefs in Assumption Map and plan the study in UX Research Plan.

## Inputs
- The synthetic output as it was produced (persona answers, simulated interviews, AI "user feedback"), with the prompt if you have it
- Any real evidence you hold: transcripts, debriefs, codebook, survey data, past reports
- Anonymise first: replace names with P1, P2, remove contact details, employers and anything that identifies a person.
If you have no real evidence, I still check the claims, every verdict reads "no real evidence", and the output says so at the top.

## Approach
Where synthetic users may and may not be used, from the Nielsen Norman Group article Synthetic Users: If, When, and How (21 Jun 2024): at most a source of hypotheses, never validation, behaviour or niche groups. The checks come from two preprints by Kuric, Demcak and Krajcovic: "Distorted Perspectives of LLM-Simulated Preferences" (arXiv, 18 May 2026), and their systematic review of 182 studies, "Synthetic Participants Generated by Large Language Models" (Research Square, 10 Mar 2026). The failure it prevents: a fluent persona quote lands on a slide, gets repeated as "users told us", and nobody can say which user.

## Workflow
1. Ask three questions: where did the synthetic answers come from and with what prompt, what decision are they being used for, and what real evidence exists?
2. Mark the whole input "SYNTHETIC" at the top. Split it into atomic claims, one statement about what people think, do or want per row.
3. Match each claim against the real evidence you pasted: supported (verbatim quote with participant id), contradicted (verbatim quote), or no real evidence. A supported claim is credited to the real quote, not to the synthetic answer.
4. Check each claim for the known failure modes: agreeable or overpraising answers, generic justifications, imagined experiences, believable detail with no source, a group the answer claims to speak for; tag the review's four issue types (cognitive misalignment, distortion, misleading believability, contamination).
5. Verdict per claim: hypothesis to test (marked as such, never as a finding) or rejected as a finding. Contradicted claims are rejected; supported claims cite only the real evidence.
6. For every hypothesis, name the real study that would answer it: method, who, how many [set by user], linked to the research plan.
7. I never generate synthetic answers, even "for comparison", and I never treat a synthetic answer as a participant.

## Output Format
```markdown
# Synthetic Answer Evidence Check
**Input:** SYNTHETIC, [source and prompt] | **Used for:** [decision] | **Real evidence read:** [list or "none"]
## Claims against real evidence
| # | Synthetic claim | Real evidence (quote, participant id) | Match | Failure modes seen |
|---|---|---|---|---|
| 1 | [claim] | [quote, P#] / none | [supported / contradicted / no real evidence] | [overpraise / generic / imagined / unsourced detail / speaks for a group] |
## Hypotheses to test (not findings)
| # | Hypothesis | Real study that would answer it | Who | How many [set by user] |
|---|---|---|---|---|
| [#] | [claim rewritten as a belief] | [method] | [group] | [n] |
## Rejected as findings
| # | Claim | Why rejected |
|---|---|---|
| [#] | [claim] | [contradicted by P# / no evidence and presented as a finding] |
## Decision
[Research lead] removes rejected claims from [deck or doc] and adds the hypotheses to the research plan, with [product lead] agreeing by [date].
```

## Done When
- The input is marked SYNTHETIC and split into single claims
- Every claim has a match against real evidence, or reads "no real evidence"
- No synthetic claim appears anywhere as a finding; every hypothesis names a real study

## Quality Bar
- Real quotes are verbatim with participant ids; synthetic text is never shown in quotation marks as a user quote
- No synthetic answers generated, for comparison or otherwise
- Counts refer to real participants only, "[n] of [N]"
- The papers are cited as preprints, not as settled proof
- Synthetic answers are hypotheses at most; Claude never counts them as user evidence or stands in for a user

## Next
Run uxr-research-plan (UX Research Plan) to plan the real study the hypotheses need.

## About the makers

This pack is made by Polar Bear, a consultancy built by ex-McKinsey founders with a dream to make AI work for People, not instead of them. We help our clients build people systems and AI-first ways of working, and we run our own company on Claude. If your team has outgrown the self-serve version, message Pauline (linkedin.com/in/paulinebertry).