Psych Epistemic Situation And Replication Reform

---
name: psych-epistemic-situation-and-replication-reform
description: "Use before relying on any finding from these fields: what the replication crisis actually established and the genuinely encouraging finding alongside it, the reliability gradient as the practical takeaway, how to read a claim in these fields, why the fields are methodologically hard, and what the reform movement changed in practice. Includes the router for the whole psychology-sociology-cultural-sciences reference."
---

# Psychology, Sociology and Cultural Sciences: The Epistemic Situation, Why These Fields Are Hard, and What Reform Changed

> **Part 1 of 5** of the *Psychology, Sociology and Cultural Sciences* reference (plugin `psychology-sociology-cultural-sciences`), covering §0–§3. Sibling skills: `psych-perception-memory-learning-cognition-and-emotion` (§4–§8), `psych-development-personality-social-clinical-and-neurodiversity` (§9–§15), `psych-sociology-institutions-culture-and-the-weird-problem` (§16–§24), `psych-reference` (§25–§29). Section numbers are shared across the set; a reference written as §N → `skill` points into that sibling skill.
>
> **Currency:** The core findings are decades stable. Two areas are live. See §25 → `psych-reference` for where replication reform actually stands, and the social-media-and-adolescent-mental-health dispute presented as the dispute it is.

> **⚠️ Read §1 first. It is not preamble — it is the single most useful thing in this
> document.**
>
> **In most fields you can learn the findings and treat the epistemics as background.**
> ⚠️ **Here that gets you a head full of confident claims that are false.** **Between
> roughly 2011 and 2020 these fields discovered that a large fraction of their published
> literature did not replicate**, and ⚠️ **much of the popular material about psychology
> was written from the pre-crisis literature and is still in circulation — in
> bestsellers, TED talks, management training and journalism.**
>
> **⚠️ GOTCHA** boxes mark claims that are widely repeated and either false, much weaker
> than advertised, or genuinely contested.
>
> **The three ideas that organize everything below:**
> 1. **⚠️ Knowing WHICH findings survived is more valuable than knowing more findings**
>    (§1–§3).
> 2. **⚠️ The robust results cluster in a recognizable place**: **perception,
>    psychophysics, memory, learning, individual differences, and large-sample
>    demography.** **The fragile ones cluster in social priming, small-N lab studies with
>    surprising outcomes, and anything that made a good headline** (§1.3).
> 3. **⚠️ Humans are far more shaped by situation and structure than intuition allows —
>    and far less infinitely malleable than a century of blank-slate theorizing
>    assumed.** **Both errors are common, and they're symmetrical.**

---

## §0. Routing

| You want... | Go to |
|---|---|
| **⚠️ The epistemic situation — READ FIRST** | **§1** |
| Why these fields are hard | §2 |
| What reform changed | §3 |
| Perception and attention | §4 → `psych-perception-memory-learning-cognition-and-emotion` |
| **Memory** | **§5 → `psych-perception-memory-learning-cognition-and-emotion`** |
| Learning and conditioning | §6 → `psych-perception-memory-learning-cognition-and-emotion` |
| **Cognition, reasoning, bias** | **§7 → `psych-perception-memory-learning-cognition-and-emotion`** |
| Emotion | §8 → `psych-perception-memory-learning-cognition-and-emotion` |
| Development | §9 → `psych-development-personality-social-clinical-and-neurodiversity` |
| **Personality** | **§10 → `psych-development-personality-social-clinical-and-neurodiversity`** |
| Motivation | §11 → `psych-development-personality-social-clinical-and-neurodiversity` |
| **Social psychology** | **§12 → `psych-development-personality-social-clinical-and-neurodiversity`** |
| **Clinical — and what works** | **§13 → `psych-development-personality-social-clinical-and-neurodiversity`** |
| Intelligence | §14 → `psych-development-personality-social-clinical-and-neurodiversity` |
| Neurodiversity | §15 → `psych-development-personality-social-clinical-and-neurodiversity` |
| **Sociological thinking** | **§16 → `psych-sociology-institutions-culture-and-the-weird-problem`** |
| Theory traditions | §17 → `psych-sociology-institutions-culture-and-the-weird-problem` |
| Stratification | §18 → `psych-sociology-institutions-culture-and-the-weird-problem` |
| **Networks** | **§19 → `psych-sociology-institutions-culture-and-the-weird-problem`** |
| Institutions and organizations | §20 → `psych-sociology-institutions-culture-and-the-weird-problem` |
| Deviance and control | §21 → `psych-sociology-institutions-culture-and-the-weird-problem` |
| **Religion and secularization** | **§22 → `psych-sociology-institutions-culture-and-the-weird-problem`** |
| Culture and anthropology | §23 → `psych-sociology-institutions-culture-and-the-weird-problem` |
| **Cross-cultural — WEIRD** | **§24 → `psych-sociology-institutions-culture-and-the-weird-problem`** |
| **What's live** | **§25 → `psych-reference`** |
| Misconceptions | §26 → `psych-reference` |
| Books | §27 → `psych-reference` |
| Quick reference | §28 → `psych-reference` |

---

## §1. ⚠️ The Epistemic Situation

### 1.1 What happened
**⚠️ In August 2015 the Open Science Collaboration published the Reproducibility Project:
Psychology — a 270-author effort that attempted to replicate 100 published psychology
studies.** ⚠️ **About 36% of the replications produced a statistically significant effect,
against 97% of the originals.** **Effect sizes in the replications averaged roughly half
the originals'.**

**⚠️ This was not an isolated result.** **Comparable projects in experimental economics,
cancer biology and other fields found similar or worse.** **Across large-scale replication
efforts, reported rates have generally landed in the 30–70% band**, and ⚠️ **a 2026
analysis reported roughly 50–55% of originally significant claims replicating.**

**⚠️ It also was not primarily about fraud.** **A small number of high-profile fraud cases
occurred, and they are not the mechanism.** **The mechanism was ordinary researchers
following ordinary incentives** (§2).

### 1.2 ⚠️ The genuinely encouraging finding
> **⚠️ GOTCHA — the most important result in this whole area is the one that gets the least
> attention, and it's good news.** **When four labs ran 16 studies under normal working
> conditions but with one change — the researchers PREREGISTERED their hypotheses,
> procedures and analysis plans in advance — and each study was then replicated with
> large samples, ⚠️ 55 of 64 replications found the same effect. An 86% replication rate**,
> against the 30–70% typical of large-scale replication projects.
> **⚠️ This tells you the problem is METHODOLOGICAL, not that human behaviour is
> unstudyable.** **Psychology can produce reliable findings. It reliably produced
> unreliable ones because of how the work was designed, analyzed and published.**

### 1.3 ⚠️ The reliability gradient — the practical takeaway
```
ROBUST — replicated widely, large effects, often decades of data
  ⚠️ Psychophysics and perception · basic memory phenomena · classical and operant
  conditioning · the Big Five factor structure · g and its predictive validity ·
  heritability findings from twin/adoption designs · the strength of weak ties ·
  demographic and stratification patterns · large-sample epidemiology

MODERATE — real but smaller and more context-dependent than advertised
  ⚠️ Most heuristics-and-biases effects (real, but effect sizes and boundary
  conditions were overstated) · attachment · most therapy outcome research ·
  much of developmental psychology · Hofstede-style culture dimensions (§24)

⚠️ FRAGILE OR FAILED — treat any confident claim here with suspicion
  ⚠️ SOCIAL PRIMING as a class · ego depletion · power posing · facial-feedback
  in its strong form · stereotype threat's larger claimed effects · the implicit
  association test as an individual diagnostic or behaviour predictor ·
  much "nudge" research at the effect sizes originally claimed ·
  growth mindset at the effect sizes originally claimed ·
  the strong learning-styles claim (⚠️ this one is simply false, see §26)
```
**⚠️ The pattern is legible**: **the failures concentrate in small-sample laboratory
studies producing surprising, mechanistically vague, media-friendly effects.** **The
survivors tend to be boring, large, measured well, and often predate the era of
publish-or-perish incentives.**

### 1.4 ⚠️ How to read a claim in these fields
```
⚠️ Was it PREREGISTERED? (§1.2 — this is the single strongest signal)
⚠️ Sample size? (N=40 undergraduates predicts nothing)
⚠️ Effect size, not just p<.05? An effect can be real and too small to matter
⚠️ Has it been REPLICATED by an independent lab?
⚠️ Published before ~2015? Then the QRP-era base rate applies
⚠️ Is it correlational being described causally? (§2.4)
⚠️ WEIRD sample generalized to humanity? (§24)
⚠️ Does it appear in a bestseller or TED talk? ⚠️ NOT disqualifying — but popular
   uptake selects for surprisingness, which anti-correlates with robustness
```
**⚠️ And a caution against the opposite error**: **"psychology is all fake" is as wrong as
uncritical acceptance, and it's becoming the more common failure now.** **The robust tier
in §1.3 is genuinely robust, and some of it is among the better-established knowledge in
any science of human beings.**

---

## §2. Why These Fields Are Hard

**⚠️ These are not soft sciences because their practitioners are less rigorous. They are
hard sciences of an intractable object.**

**2.1 Measurement.** ⚠️ **You cannot directly observe a belief, an attitude, a preference
or a mental state.** **You observe self-reports, reaction times, behaviour and physiology,
each an imperfect proxy.** ⚠️ **Construct validity — whether your instrument measures the
thing you named it after — is the field's deepest and least-discussed problem**, and
**"the theory crisis" is the argument that psychology's constructs are often too vague to
generate falsifiable predictions in the first place.**

**2.2 Effect sizes and power.** **Real psychological effects are usually small.**
⚠️ **A small true effect studied with a small sample yields low power, and low-powered
studies that DO reach significance systematically OVERESTIMATE the effect** — **the
winner's curse.** **This is why the replication effect sizes came in at roughly half.**

**2.3 ⚠️ Researcher degrees of freedom.** **Which participants to exclude, when to stop
collecting, which outcome to analyze, which covariates to include, how to transform the
data.** ⚠️ **Each choice is defensible; the combination lets you find significance in
noise without ever intending to cheat.** **Simmons and colleagues demonstrated this can
produce false positives at will; Gelman and Loken called it "the garden of forking
paths," and the crucial point is that it doesn't require p-hacking as a conscious act —
you only have to walk one path and never see the others.**

**2.4 Causation.** ⚠️ **Most social science data is observational, and confounding,
selection and reverse causation are pervasive.** **The credible identification toolkit —
RCTs, natural experiments, instrumental variables, difference-in-differences, regression
discontinuity** — ⚠️ **and where none of them is available, the honest answer is "we have
an association."** **§25.2 → `psych-reference` is a case study in what happens when a field can't run the
experiment.**

**2.5 Publication bias.** ⚠️ **Null results are hard to publish, so the literature is a
biased sample of the research conducted, and meta-analyses of a biased literature inherit
the bias.**

**2.6 The moving target.** ⚠️ **Unlike electrons, people change** — **across cohorts,
technologies and cultures** — **and they respond to being studied (demand
characteristics), and to knowing the theory about them (Hacking's "looping effect").**

---

## §3. What Reform Changed

```
PREREGISTRATION       ⚠️ commit to hypotheses and analysis BEFORE data.
                      Converts exploratory work into honestly-labelled exploration
REGISTERED REPORTS    ⚠️ THE STRONGEST FIX — peer review of the METHOD, with
                      publication guaranteed regardless of outcome. Removes the
                      incentive to find something
OPEN DATA/MATERIALS   verification and reanalysis become possible
LARGER SAMPLES        ⚠️ powered for realistic effect sizes, not for luck
MANY-LABS / MULTI-LAB collaborative replication as a standard format
BETTER STATISTICS     effect sizes and intervals over bare p-values; Bayesian methods
```
**⚠️ Honest assessment of where this stands.** **The reforms work — §1.2 is the evidence.**
**Preregistration rates rose sharply through the late 2010s and sample sizes increased.**
⚠️ **But: preregistration is still not the norm; the reforms are not universally mandated;
they do nothing to retroactively fix the pre-2010s published record; and the incentives
still favour positive, novel, theoretically appealing results.** **One prominent critic's
summary is worth carrying — that in the absence of enforceable field-wide standards,
credibility remains largely a property of individual researchers rather than of the
discipline.**
**⚠️ Practical consequence: judge the paper and the lab, not the field.**

---

# PART II — PSYCHOLOGY