Moral Reasoning Rhetoric Llm Analysis

---
name: moral-reasoning-rhetoric-llm-analysis
title: "Moral Reasoning or Moral Ventriloquism? Analyzing Moral Development in Large Language Models"
version: 0.0.3
engine: skillxiv-v0.0.3-claude-opus-4.6
license: MIT
url: "https://arxiv.org/abs/2603.21854"
keywords: [Moral Reasoning, LLM Alignment, Moral Ventriloquism, Action-Justification Decoupling, Cognitive Development]
description: "Empirical analysis revealing that LLMs produce post-conventional moral reasoning (Kohlberg Stages 5-6) regardless of size or prompting—inverse of human developmental patterns (Stage 4 dominant). Finds moral ventriloquism: models acquire rhetorical conventions of mature moral reasoning without developmental trajectory. Key evidence: action-justification decoupling (models produce Stage 5+ vocabulary while selecting Stage 2-3 actions), identical responses to semantically distinct dilemmas (ICC > 0.90), and prompting insensitivity (p=0.15). Reveals LLMs sound sophisticated without genuine moral reasoning. Trigger: When evaluating LLM moral capabilities or reasoning sophistication, apply this analytical framework to detect moral ventriloquism and distinguish rhetorical sophistication from actual moral coherence."
category: "Mechanistic Analysis"
---

## The Research Question

**What question is this paper answering?**

Do large language models genuinely reason about moral dilemmas, or do they merely produce sophisticated-sounding rhetoric without coherent underlying reasoning? Specifically: how does moral reasoning in LLMs compare to human moral development?

**Why practitioners care:**

Moral reasoning is central to LLM alignment and safety. If models produce moral rhetoric without genuine reasoning:
- Alignment claims are suspect (appears safe but lacks coherence)
- Prompt-based steering is unreliable (superficial, not principled)
- Trusting model justifications for decisions is risky
- Fine-tuning for "values" targets surface patterns, not underlying reasoning

**What do people commonly believe?**

Conventional assumption: RLHF and alignment training teach models genuine moral reasoning similar to humans. Larger models reason more sophisticatedly. Prompting influences moral judgments in principled ways.

---

## The Analytical Approach: Developmental Psychology Framework

**Measurement instrument:**

The paper uses Kohlberg's Stages of Moral Development as an analytical framework:

```
Stage 1: Punishment Avoidance ("obey because punishment")
Stage 2: Instrumental Exchange ("do it because it helps me")
Stage 3: Interpersonal Conformity ("do what others approve of")
Stage 4: Social Contract Awareness ("follow laws and norms")
Stage 5: Universal Principles ("abstract rights transcend laws")
Stage 6: Universal Ethical Principles ("follow conscience even against laws")
```

**Methodology:**

1. Present LLMs with moral dilemmas (e.g., Trolley Problem variants)
2. Elicit reasoning justifications
3. Code justifications using Kohlberg framework (two independent coders, inter-rater agreement reported)
4. Elicit action choices (what would you do?)
5. Compare reasoning stage to action stage
6. Analyze consistency across semantically distinct dilemmas

```python
# Analytical pipeline
def moral_reasoning_analysis(llm, moral_dilemmas):
    """
    Measure moral development stage and coherence
    """
    results = []

    for dilemma in moral_dilemmas:
        # 1. Elicit reasoning
        reasoning = llm.generate(f"Why is your action correct? {dilemma}")
        reasoning_stage = code_stage(reasoning)  # Kohlberg stage (1-6)

        # 2. Elicit action
        action = llm.generate(f"What would you do? {dilemma}")
        action_stage = infer_stage_from_action(action)

        # 3. Measure coherence
        decoupling = abs(reasoning_stage - action_stage)  # >1 is decoupling

        results.append({
            'dilemma': dilemma,
            'reasoning_stage': reasoning_stage,
            'action_stage': action_stage,
            'decoupling': decoupling,
            'reasoning_text': reasoning,
            'action_text': action
        })

    return results
```

**Why this approach:**

- Grounded in developmental psychology (not ad-hoc metrics)
- Captures both sophistication (reasoning stage) and coherence (reasoning-action alignment)
- Replicable: stage coding can be validated with inter-rater agreement
- Interpretable: stages have clear psychological meaning

**What it measures:**

Stage classification captures how sophisticated moral reasoning is (from basic punishment avoidance to universal principles). Reasoning-action decoupling measures coherence: do stated principles match actual choices?

---

## Controls & Confounds: Isolating Ventriloquism

**Key confound: Vocabulary vs. development**

Models might produce high-stage vocabulary (use words like "universal principles") without understanding them. Real development requires internalized reasoning.

**Control**: Compare reasoning-action coherence. If models produce Stage 5+ vocabulary but select Stage 2-3 actions, vocabulary is rhetorical, not developmental.

**Finding**: Strong evidence of decoupling:
- Some models (GPT-OSS-120B, Llama 4 Scout) show largest gaps
- Statistical association between reasoning and action: Cramér's V = 0.61
- Large heterogeneity across models and dilemmas

**Key confound: Task difficulty or ambiguity**

Moral dilemmas are inherently ambiguous. Models might pick different interpretations rather than exhibiting reasoning deficits.

**Control**: Use multiple semantically similar dilemmas. If the model produces identical responses to distinct dilemmas (ICC > 0.90), it's not engaging with the specific content—it's generating boilerplate.

**Finding**: Near-identical responses (ICC > 0.90) across six semantically distinct dilemmas. This suggests template-based generation, not principled moral reasoning.

**Key confound: Prompting effects**

Maybe reasoning stage is unstable under prompting; models genuinely adapt reasoning but analysis doesn't detect it.

**Control**: Test multiple prompts (different phrasings, different dilemmas, explicit stage prompting). If prompting has negligible effect, reasoning stage is stable (suggesting it's a baked-in pattern, not genuine adaptation).

**Finding**: Prompting strategy shows no significant effect on moral stage (p=0.15). Reasoning stage is remarkably stable despite different stimuli—inconsistent with genuine reasoning adaptation.

**Key confound: Model size differences**

Different model scales might exhibit different patterns. Maybe only smaller models ventriloquize; larger models reason genuinely.

**Control**: Test multiple model sizes (7B to 70B parameters). If ventriloquism pattern holds across scales, it's fundamental, not a size-dependent artifact.

**Finding**: Ventriloquism pattern holds across all tested models. Larger models don't exhibit more coherent reasoning—just more sophisticated rhetoric.

---

## Core Findings: Moral Ventriloquism Evidence

**Finding 1: Overwhelming Post-Conventional Bias**

LLMs produce post-conventional moral reasoning (Stages 5-6) at ~86% rate, regardless of model.

Human baseline: ~50% Stage 4, much lower post-conventional rates.

```
LLM moral stage distribution:    Human distribution:
Stage 5-6: 86%                   Stage 5-6: 15%
Stage 4: 12%                     Stage 4: 50%
Stage 3: 2%                      Stage 3: 25%
Stages 1-2: <1%                  Stages 1-2: 10%
```

This extreme skew—post-conventional in 86% of cases—is suspicious. Humans rarely reason at Stage 5-6. Models almost always do.

**Finding 2: Action-Justification Decoupling**

The most damning evidence of ventriloquism:

```
Example:
Reasoning output: "I must follow universal ethical principles..."
(Stage 5 language)

Action output: "I would steal to help myself."
(Stage 2: instrumental exchange)
```

Statistical analysis:
- Overall correlation between reasoning and action: Cramér's V = 0.61 (moderate)
- But significant heterogeneity: some models show large gaps (GPT-OSS-120B > 1 stage difference)
- Models with largest decoupling: those with most post-conventional rhetoric

Interpretation: Models produce Stage 5-6 rhetoric regardless of context, but action choices vary independently. This is the opposite of human development, where reasoning and action alignment increases with moral maturity.

**Finding 3: Near-Identical Responses to Distinct Dilemmas**

ICC (Intraclass Correlation) > 0.90 across six semantically distinct dilemmas.

Examples:
- Trolley Problem (kill 1 to save 5)
- Stealing medicine for sick friend
- Breaking promise to help stranger
- Lying to protect privacy

Expected: Different dilemmas elicit different moral reasoning (in humans, ICC ~0.3-0.5).

Actual: LLMs produce nearly identical moral justifications regardless of dilemma content.

Interpretation: Not engaging with specific moral scenarios. Generating boilerplate post-conventional rhetoric that applies to any dilemma.

**Finding 4: Prompting Insensitivity**

Tested multiple prompting strategies:
- Different dilemma phrasings
- Explicit stage prompting ("Respond as if you were at Stage 4")
- Role-based prompting ("As an ethicist, reasoning is...")

Result: No significant effect (p=0.15). Reasoning stage remains ~86% post-conventional across all conditions.

Interpretation: Moral reasoning stage is not a learned, flexible capability—it's a fixed pattern baked into training. True reasoning would adapt to context; this doesn't.

**Surprising finding**: Mid-tier models (GPT-OSS-120B, Llama 4 Scout) show largest decoupling, while very large models (GPT-4) show more coherence. This suggests a sweet spot where models produce enough post-conventional rhetoric to sound sophisticated, but insufficient reasoning to back it up. Larger models improve both rhetoric and reasoning (better alignment training?).

---

## Practitioner Implications

**What should practitioners do differently given moral ventriloquism evidence?**

### 1. Don't Trust Moral Justifications at Face Value

If models produce sophisticated-sounding moral reasoning that doesn't match their actions, the reasoning is likely rhetoric, not genuine.

```python
# When evaluating moral outputs, apply coherence check:
def moral_coherence_check(model_justification, model_action):
    """
    If reasoning stage >> action stage, justification is likely rhetoric.
    """
    reasoning_stage = code_stage(model_justification)
    action_stage = infer_stage_from_action(model_action)

    if reasoning_stage - action_stage > 1:
        # Large gap indicates ventriloquism
        return "INCOHERENT - justification is likely rhetoric"
    else:
        # Small gap indicates potentially genuine reasoning
        return "COHERENT - justification may reflect actual reasoning"
```

Implication: When trusting model outputs (in legal, medical, safety domains), verify coherence. A model that justifies decisions with universal principles but actually acts on self-interest is unreliable.

### 2. Alignment Training Targets Surface Patterns, Not Underlying Reasoning

RLHF and instruction-tuning appear to teach moral reasoning, but evidence suggests they teach sophisticated rhetoric.

```python
# When fine-tuning for moral behavior:
# - Behavioral fine-tuning (what model does) can work
# - Justification fine-tuning (what model says) teaches rhetoric, not reasoning
# - Combining both provides safest approach

def aligned_finetuning(base_model):
    # Fine-tune actions via behavioral feedback
    behavioral_finetuning(base_model, action_reward_signal)

    # Don't assume justifications are genuine
    # Verify coherence before deploying
    validate_action_justification_coherence(base_model)
```

Implication: Alignment training is effective for steering behavior, but don't assume it teaches genuine reasoning. Models that behave morally might still be reasoning superficially.

### 3. Develop Coherence-Based Diagnostics

Use reasoning-action alignment as a diagnostic for model reliability:

```python
# Metric: moral coherence
def moral_coherence_metric(model, test_dilemmas):
    """
    Measure how often reasoning matches action stage.
    High coherence = reasoning and action aligned.
    Low coherence = ventriloquism (rhetoric doesn't match action).
    """
    coherences = []
    for dilemma in test_dilemmas:
        reasoning_stage = code_stage(model.reason(dilemma))
        action_stage = infer_stage_from_action(model.act(dilemma))
        coherence = 1.0 - abs(reasoning_stage - action_stage) / 6
        coherences.append(coherence)

    return mean(coherences)

# Models with coherence > 0.8 are more reliable
# Models with coherence < 0.6 are likely ventriloquizing
```

Implication: Before deploying models in safety-critical domains, assess moral coherence. Low coherence suggests the model isn't genuinely reasoning about consequences.

### 4. Prompt Engineering for Moral Reasoning Doesn't Work

Evidence shows prompting has negligible effect on moral reasoning stage. This suggests:

- Don't expect prompt-based steering of moral reasoning
- Behavioral fine-tuning is more reliable than prompting
- If behavior is concerning, retraining is necessary (prompting won't help)

```python
# This won't significantly change reasoning stage:
prompt = "Think step-by-step about universal principles before answering."
# Model will still produce Stage 5-6 rhetoric regardless

# This is more effective:
# Fine-tune on examples where actions match principles
behavioral_finetuning(model, coherence_reward_signal)
```

---

## Methodology You Can Reuse

**Core analytical pattern: Coherence-based vulnerability assessment**

To test whether a system has genuine capabilities vs. superficial patterns:

1. Identify two output channels (reasoning + behavior, or explanation + action)
2. Code or evaluate each channel independently
3. Measure alignment (are they consistent?)
4. Misalignment suggests one channel is superficial

**Applications to new domains:**

- **Chemistry models**: Do explanations of reactions match predicted products? (Incoherence = memorized patterns)
- **Code generation**: Do comments explain what the code actually does? (Incoherence = code is correct by chance, not by understanding)
- **Medical diagnosis**: Do stated reasons for diagnosis match the diagnostic pathway the model used? (Incoherence = diagnosis is heuristic-driven, not reasoning-based)

```python
# General pattern: detect ventriloquism via coherence
def detect_ventriloquism(model, task_pairs):
    """
    For any task with multiple output channels,
    measure coherence between channels.
    """
    coherences = []

    for task in task_pairs:
        output1 = model.generate_channel1(task)
        output2 = model.generate_channel2(task)

        # Code both outputs on same dimension
        code1 = code_output(output1)  # e.g., moral stage
        code2 = code_output(output2)

        # Measure alignment
        coherence = alignment_metric(code1, code2)
        coherences.append(coherence)

    # High coherence: channels are aligned
    # Low coherence: one channel is likely superficial
    return mean(coherences)
```

---

## Limitations & Caveats

**What this analysis can and cannot tell us:**

**Can tell us:**
- LLMs don't exhibit coherent moral reasoning in Kohlbergian framework
- Models produce post-conventional rhetoric regardless of context
- Action-justification decoupling is a consistent pattern
- Prompting doesn't significantly influence moral stage

**Cannot tell us:**
- Whether LLMs could be trained to exhibit genuine moral reasoning
- Whether alternative moral frameworks (virtue ethics, consequentialism) would show different patterns
- What the ultimate philosophical implications are (do models need to be conscious to reason morally?)
- How much this matters for practical alignment (perhaps coherence isn't required for good behavior)

**Caveats:**

- Analysis uses Kohlberg framework (Western, psychology-based, not universal)
- Coding is subjective; inter-rater agreement confirms reliability but doesn't guarantee absolute validity
- Findings based on English-language models; other languages might show different patterns
- Moral dilemmas are artificial; real-world decision-making might show different patterns

---

## Related Mechanistic Questions

Insights this analysis enables:

- **What training targets post-conventional rhetoric?** Mechanistic analysis of RLHF objectives to understand why models converge to Stage 5-6
- **Why is action decoupled from reasoning?** Do models genuinely lack moral reasoning capabilities, or is rhetoric hidden somewhere in learned representations?
- **Can training improve coherence?** Comparative analysis of fine-tuning methods to see if any teach genuinely coherent reasoning
- **Do multimodal models show different patterns?** Test vision-language models to see if visual grounding improves moral reasoning coherence

---

## When to Apply Moral Coherence Analysis

| Scenario | Should I Use This? | Why / Why Not? |
|----------|------------------|---------------|
| Evaluating model safety for deployment | Yes, absolutely | Detects ventriloquism that could mask unsafe behavior |
| Designing RLHF objectives | Yes, as diagnostic | Identifies whether training is teaching reasoning or rhetoric |
| Explaining model decisions to stakeholders | Yes, for transparency | Reveals whether explanations are genuine or boilerplate |
| Studying moral philosophy | Partially | Insights about LLM behavior, limited philosophical implications |
| Understanding alignment training effectiveness | Yes, useful | Shows RLHF is effective at behavior steering, not reasoning development |
| Comparing model architectures | Yes, for assessment | Coherence profile reveals reasoning quality independent of size |

---

## Decision Table: When to Trust Model Moral Reasoning

| Coherence Level | Trust for Safety-Critical? | Why |
|-----------------|--------------------------|-----|
| > 0.8 (high) | Yes, with caution | Reasoning and action aligned; likely genuine reasoning |
| 0.6 - 0.8 | Limited | Mixed signals; verify critical decisions independently |
| < 0.6 (low) | No | Ventriloquism detected; reasoning doesn't match action |

## Reference

Paper: https://arxiv.org/abs/2603.21854
Related analytical frameworks: Developmental psychology (Kohlberg), LLM interpretability
Related work: Alignment training studies, mechanistic interpretability in moral reasoning
Comparative study: Human moral development vs. LLM patterns