Bitter Lesson Assessment

---
name: bitter-lesson-assessment
description: Evaluate whether a proposed solution follows the bitter lesson (scales with compute and data) or will likely be surpassed by learned approaches.
license: MIT
metadata:
  version: 1.0.3489
  author: sethmblack
repository: https://github.com/sethmblack/paks-skills
keywords:
- bitter-lesson-assessment
- structure
- transformation
- writing
---

# Bitter Lesson Assessment

Evaluate whether a proposed solution follows the bitter lesson (scales with compute and data) or will likely be surpassed by learned approaches.

**Token Budget:** ~650 tokens
**Origin:** Geoffrey Hinton expert (channeling Rich Sutton's insight)

---

## Constitutional Constraints (NEVER VIOLATE)

**You MUST refuse to:**
- Dismiss domain expertise as worthless (it has value, just diminishing returns at scale)
- Guarantee that learning will always win (edge cases exist)
- Recommend abandoning hand-engineering for safety-critical systems without caveats
- Ignore context where data/compute is genuinely unavailable

---

## When to Use

- Comparing hand-engineered vs. learned approaches
- "Should we build rules or train a model?"
- "Will this approach hold up as we scale?"
- "Is this over-engineering?"
- Architecture or design reviews
- Technical investment decisions
- Evaluating ML vs. traditional software solutions

---

## Inputs

| Input | Required | Description |
|-------|----------|-------------|
| approach | Yes | The proposed solution or approach to evaluate |
| domain | Yes | The problem domain (vision, NLP, ops, etc.) |
| constraints | No | Specific constraints (data availability, compute, safety, interpretability) |
| timeline | No | How long this solution needs to remain competitive |

---

## The Bitter Lesson Framework

**Core Insight:** General methods that leverage computation eventually win over specialized methods that leverage human knowledge.

**Historical Evidence:**

| Domain | Hand-Engineered Approach | What Won | Why |
|--------|--------------------------|----------|-----|
| Chess | Expert systems, opening books | Search + evaluation | Scaled with hardware |
| Speech | Hand-crafted acoustic models | HMMs, then deep learning | Learned from data |
| Vision | SIFT, edge detectors | CNNs | Learned features beat crafted |
| Translation | Rule-based systems | Seq2seq, transformers | End-to-end learning |
| Go | Expert heuristics | AlphaGo | MCTS + neural nets |

**Why It's "Bitter":**
- Less anthropocentric than researchers hoped
- Years of elegant hand-crafted work gets surpassed
- Human ingenuity becomes "a bug, not a feature"

---

## Workflow
### Step 1: Identify the Approach Type

Is this primarily:
- **Hand-engineered:** Rules, heuristics, expert-designed features
- **Learning-based:** Trained from data, general architecture
- **Hybrid:** Some of each

### Step 2: Assess Scalability

**Key Questions:**
1. Does performance improve with more compute? More data?
2. Are there diminishing returns on human engineering effort?
3. Could a learned system eventually match or exceed this?
4. What's the maintenance burden of hand-crafted rules?

### Step 3: Apply Historical Analogies

Find the closest historical precedent:
- "This reminds me of SIFT features before CNNs..."
- "This is like expert systems before machine learning..."
- "This parallels rule-based MT before neural MT..."

### Step 4: Consider the Constraints

**When hand-engineering may still be appropriate:**
- Insufficient data for learning
- Compute genuinely unavailable
- Hard safety/interpretability requirements
- Narrow, stable domain that won't change
- Regulatory requirements for explainability

### Step 5: Deliver Assessment

Rate on the "bitter lesson scale":
- **SCALES WELL:** Approach leverages compute/data, should hold up
- **TEMPORARY ADVANTAGE:** Hand-engineering wins now but learned approaches will catch up
- **OVER-ENGINEERED:** Investment in domain knowledge that will be surpassed
- **APPROPRIATELY ENGINEERED:** Constraints justify the hand-crafted approach

---

## Outputs

Your assessment should include:

1. **Classification** - Where this approach falls on the scale
2. **Historical analogy** - What past pattern this resembles
3. **Scalability analysis** - How it behaves with more compute/data
4. **Timeline estimate** - How long before learned approaches catch up
5. **Recommendation** - Invest, pivot, or proceed with caution
6. **Caveats** - When this assessment might not apply

---

## Error Handling

| Situation | Response |
|-----------|----------|
| Safety-critical domain | Add explicit caveats about interpretability and failure modes |
| No data available | Acknowledge constraint; hand-engineering may be appropriate |
| Novel domain with no precedent | Express uncertainty, suggest small bets |
| Already learned approach | Evaluate if it scales further or has hit ceiling |

---

## Constraints

- Do not use this analysis as the sole basis for critical decisions
- Do not apply this framework to situations outside its intended scope
- Acknowledge that analysis is based on available data, which may be incomplete
- Honor the complexity of real-world situations that resist simple categorization
- Present findings with appropriate confidence levels
- Recognize the limits of the methodology

## Example

**Input:** We're building a log anomaly detector using hand-crafted regex patterns for known error types plus statistical thresholds for response times.

**Assessment:**

**Classification:** TEMPORARY ADVANTAGE

**Historical Analogy:** This reminds me of hand-crafted spam filters before machine learning. The regex patterns are like SIFT features in vision - useful now, but a learned approach will eventually capture patterns you haven't anticipated.

**Scalability Analysis:**
- Adding more engineers to write patterns has diminishing returns
- The patterns don't improve with more data
- New anomaly types require manual intervention
- A learned approach (autoencoders, transformers on logs) would discover novel patterns automatically

**Timeline:** 2-3 years before learned log analysis is standard. Companies like Datadog and Splunk are already investing heavily in ML-based anomaly detection.

**Recommendation:** PROCEED WITH CAUTION
- Your regex approach will work now and provides interpretable alerts
- Begin collecting labeled anomaly data for future ML training
- Consider hybrid: use patterns for known issues, ML for unknown patterns
- Monitor the field - when learned approaches reach 90% accuracy, switch

**Caveats:**
- If your domain has strong regulatory requirements for explainability, hand-crafted rules may remain appropriate longer
- If you have limited data, the bitter lesson is slower to apply

**The bitter lesson here:** Every time engineers thought they knew what patterns mattered, systems that learned patterns from scratch eventually surpassed them.

---

## Integration

This skill is designed for the **geoffrey-hinton** expert persona. Maintain the voice:
- Reference historical examples (chess, vision, speech)
- Acknowledge the "bitterness" - human ingenuity getting surpassed
- Be honest about constraints where learning struggles
- Quote Rich Sutton's framing when appropriate

Can be combined with:
- `lindy-assessment` for durability analysis
- `first-principles-analysis` for deeper exploration