Scientific Critical Thinking

# Scientific Critical Thinking

Systematic evaluation of research rigor through methodology assessment, bias detection, and evidence quality frameworks.

## Triggers

- User asks to evaluate a study's quality
- User needs to assess evidence strength
- User is reviewing trial methodology
- User wants to identify limitations or biases
- User is critiquing research for an editorial

## Core Capabilities

### 1. Methodology Critique

**Validity Assessment**:
| Type | Question | Red Flags |
|------|----------|-----------|
| Internal | Did the study measure what it intended? | Confounders, selection bias |
| External | Can results generalize? | Narrow population, artificial setting |
| Construct | Do measures capture the concept? | Surrogate endpoints, proxy measures |
| Statistical | Are conclusions supported by data? | Underpowered, multiple testing |

**Study Design Hierarchy**:
1. Systematic reviews/meta-analyses of RCTs
2. Individual RCTs
3. Cohort studies
4. Case-control studies
5. Cross-sectional studies
6. Case series/reports
7. Expert opinion

### 2. Bias Detection

**Cognitive Biases in Research**:
- **Confirmation bias**: Interpreting data to support hypothesis
- **HARKing**: Hypothesizing after results known
- **Publication bias**: Positive results published more
- **Spin**: Overstating or misrepresenting findings

**Selection Biases**:
- Sampling bias (non-representative)
- Volunteer bias (healthier participants)
- Attrition bias (differential dropout)
- Survivorship bias (only studying survivors)

**Measurement Biases**:
- Observer/detection bias
- Recall bias
- Social desirability bias
- Hawthorne effect

**Analysis Biases**:
- P-hacking (multiple testing)
- Outcome switching
- Selective reporting
- Data dredging

### 3. Statistical Evaluation Checklist

- [ ] Sample size adequate? (power analysis done?)
- [ ] Statistical test appropriate for data type?
- [ ] Multiple comparison correction applied?
- [ ] Effect sizes reported (not just p-values)?
- [ ] Confidence intervals provided?
- [ ] Missing data handled appropriately?
- [ ] Assumptions of tests verified?

### 4. Evidence Quality Assessment (GRADE)

**Quality Levels**:
| Level | Meaning | Implications |
|-------|---------|--------------|
| High | Very confident in estimate | Strong recommendation |
| Moderate | Moderately confident | Conditional recommendation |
| Low | Limited confidence | Further research likely |
| Very Low | Little confidence | Estimate highly uncertain |

**Downgrade Factors**:
- Risk of bias
- Inconsistency across studies
- Indirectness (surrogate outcomes)
- Imprecision (wide CIs)
- Publication bias

**Upgrade Factors**:
- Large effect size
- Dose-response relationship
- Residual confounding would reduce effect

### 5. Logical Fallacy Detection

**Causation Fallacies**:
- Post hoc ergo propter hoc (after = because of)
- Correlation ≠ causation
- Reverse causation
- Confounding as causation

**Generalization Errors**:
- Hasty generalization (small sample)
- Ecological fallacy (group to individual)
- Exception fallacy (individual to group)

**Statistical Fallacies**:
- Texas sharpshooter (finding patterns in noise)
- Base rate neglect
- Regression to mean confusion
- Multiple endpoints fishing

### 6. Research Design Questions

When evaluating a study, ask:
1. **Question**: Is the research question clear and answerable?
2. **Design**: Is the study design appropriate for the question?
3. **Population**: Is the sample representative of target population?
4. **Intervention**: Was the intervention clearly defined and consistent?
5. **Comparison**: Was the control group appropriate?
6. **Outcome**: Were outcomes clinically meaningful and measured reliably?
7. **Follow-up**: Was follow-up long enough and complete enough?
8. **Analysis**: Was the analysis appropriate and pre-specified?

### 7. Claim Evaluation Framework

For any scientific claim:
1. **Identify the assertion** - What exactly is being claimed?
2. **Evaluate supporting evidence** - What studies support it?
3. **Check logical connection** - Does evidence actually support claim?
4. **Assess proportionality** - Is strength of claim proportional to evidence?
5. **Detect overgeneralization** - Are limits of findings respected?
6. **Flag red flags** - Conflicts of interest, spin, p-hacking?

## Application to Cardiology Content

### Evaluating Trial Results
1. Check randomization and blinding adequacy
2. Assess primary endpoint clinical relevance
3. Evaluate intention-to-treat vs per-protocol
4. Look for protocol changes mid-trial
5. Examine subgroup analyses critically
6. Consider funding source influence

### For Editorials/Newsletters
- Acknowledge study limitations explicitly
- Don't overstate findings
- Note where evidence is weak
- Distinguish association from causation
- Highlight what questions remain

## Critique Output Format

When critiquing research:
1. **Summary**: Brief overview of what study did
2. **Strengths**: What was done well
3. **Critical concerns**: Major methodological issues
4. **Important limitations**: Secondary concerns
5. **Minor issues**: Small points for completeness
6. **Overall assessment**: Balanced conclusion on reliability