Chapter Evaluator

---
name: chapter-evaluator
description: "Evaluate educational chapters from dual student and teacher perspectives. This skill should be used when analyzing chapter quality, identifying content gaps, or planning chapter improvements. Reads all lessons in a chapter directory and provides structured analysis with ratings, gap identification, and prioritized recommendations."
---

# Chapter Evaluator Skill

Evaluate educational chapters through dual lenses: the **Student Experience** (engagement, clarity, confidence) and the **Teacher Perspective** (pedagogy, objectives, assessment). Output structured analysis with ratings, gaps, and actionable improvements.

## When to Use

- Analyzing a chapter's overall quality before publication
- Identifying why content "feels off" (too short, boring, disconnected)
- Planning improvements to existing chapters
- Comparing chapters against quality standards
- User asks to "evaluate", "review", "analyze", or "assess" a chapter

## Evaluation Process

### Step 1: Gather Chapter Content

Read all lesson files in the chapter directory:

```bash
ls -la <chapter-path>/*.md | grep -v summary | grep -v README | grep -v quiz
```

For each lesson file, extract:
- YAML frontmatter (learning objectives, cognitive load, skills, layer)
- Word count
- Section structure (headings)
- Try With AI prompts
- Hands-on exercises
- Code examples

### Step 2: Student Perspective Analysis

Evaluate as a beginner encountering this content for the first time.

#### 2.1 Engagement Score (1-10)

| Score | Criteria |
|-------|----------|
| 9-10 | Compelling hook, real-world relevance clear, I want to keep reading |
| 7-8 | Interesting enough, some engaging moments, minor dry spots |
| 5-6 | Functional but forgettable, reads like documentation |
| 3-4 | Boring, walls of text, no compelling reason to continue |
| 1-2 | Would abandon after first section |

**Check for:**
- Opening hook (does first paragraph grab attention?)
- Real-world scenarios (why does this matter to ME?)
- Story/narrative flow vs disconnected facts
- Visual breaks (diagrams, tables, code blocks)
- Pacing variety (concept → hands-on → concept)

#### 2.2 Length Assessment

| Verdict | Criteria |
|---------|----------|
| Too Short | Missing examples, concepts unexplained, abrupt endings, "I don't understand" |
| Just Right | Each concept has sufficient depth, examples clarify, natural flow |
| Too Long | Repetitive explanations, over-elaborated points, could cut 30%+ |

**Word count benchmarks:**
- Conceptual lesson: 1,000-1,400 words
- Hands-on lesson: 1,200-1,600 words
- Installation/setup: 800-1,200 words (focused)
- Capstone: 1,400-1,800 words

#### 2.3 Clarity Score (1-10)

| Score | Criteria |
|-------|----------|
| 9-10 | Crystal clear, no re-reading needed, "aha" moments |
| 7-8 | Mostly clear, occasional re-read for complex parts |
| 5-6 | Understandable with effort, some confusing sections |
| 3-4 | Frequently confused, missing context, jargon unexplained |
| 1-2 | Cannot follow, assumes knowledge I don't have |

**Check for:**
- Jargon introduced before defined
- Logical flow between paragraphs
- Transitions between sections
- Prerequisites assumed vs stated

#### 2.4 Hands-On Effectiveness (1-10)

| Score | Criteria |
|-------|----------|
| 9-10 | Clear steps, achievable, builds confidence, "I did it!" |
| 7-8 | Mostly clear, minor ambiguity, successful completion likely |
| 5-6 | Workable but confusing steps, may need to troubleshoot |
| 3-4 | Missing steps, unclear what to do, likely to get stuck |
| 1-2 | Cannot complete without external help |

**Check for:**
- Step-by-step instructions (numbered, clear)
- Expected output/results shown
- Troubleshooting guidance
- Connection to concepts just learned

#### 2.5 Progression Clarity (1-10)

| Score | Criteria |
|-------|----------|
| 9-10 | Clear path from start to mastery, each lesson builds on previous |
| 7-8 | Generally progressive, minor jumps between lessons |
| 5-6 | Some logical progression, noticeable gaps |
| 3-4 | Disconnected lessons, unclear how they relate |
| 1-2 | Random ordering, no clear learning path |

**Check for:**
- Opening connections ("In Lesson N-1, you learned X. Now...")
- Running example threaded through chapter
- Skills building on each other
- Clear "what's next" at lesson end

#### 2.6 Confidence Score (1-10)

| Score | Criteria |
|-------|----------|
| 9-10 | "I can definitely do this now" - ready to apply independently |
| 7-8 | "I mostly understand and could figure out the rest" |
| 5-6 | "I kind of get it but would need help applying it" |
| 3-4 | "I'm confused about when/how to use this" |
| 1-2 | "I have no idea what I just read" |

**Check for:**
- Practice opportunities before moving on
- Verification steps ("you should see X")
- Real-world application examples
- "Try it yourself" prompts

### Step 3: Teacher Perspective Analysis

Evaluate as an instructional designer assessing pedagogical soundness.

#### 3.1 Learning Objectives Quality (1-10)

| Score | Criteria |
|-------|----------|
| 9-10 | SMART objectives, measurable, aligned to content and assessment |
| 7-8 | Clear objectives, mostly measurable, good alignment |
| 5-6 | Objectives present but vague or partially aligned |
| 3-4 | Weak objectives, not measurable, poor alignment |
| 1-2 | Missing or meaningless objectives |

**Check for:**
- Bloom's taxonomy verb alignment (Remember → Create)
- Measurable criteria ("can explain", "can create", "can distinguish")
- Assessment method specified
- Objectives actually taught in lesson content

#### 3.2 Cognitive Load Management (1-10)

| Score | Criteria |
|-------|----------|
| 9-10 | Appropriate concepts for level, well-scaffolded, no overload |
| 7-8 | Generally appropriate, minor overload moments |
| 5-6 | Some cognitive overload, too many concepts at once |
| 3-4 | Significant overload, concepts piled without consolidation |
| 1-2 | Overwhelming, no chance of retention |

**Benchmarks by proficiency:**
- A1-A2: 3-5 new concepts per lesson
- B1-B2: 5-7 new concepts per lesson
- C1-C2: 7-10 new concepts per lesson

**Check for:**
- New concepts counted in frontmatter
- Concepts introduced one at a time
- Practice before new concept introduced
- Chunking of complex procedures

#### 3.3 Scaffolding Quality (1-10)

| Score | Criteria |
|-------|----------|
| 9-10 | Perfect progression, each concept builds on previous, no gaps |
| 7-8 | Good scaffolding, minor jumps that students can bridge |
| 5-6 | Some scaffolding gaps, requires prior knowledge not taught |
| 3-4 | Significant gaps, assumes knowledge not in prerequisites |
| 1-2 | No scaffolding, concepts appear randomly |

**Check for:**
- Prerequisites listed and actually prerequisite
- Concepts introduced before used
- Increasing complexity curve
- Prior knowledge activated before new content

#### 3.4 Pedagogical Layer Appropriateness (1-10)

| Layer | Expected Characteristics |
|-------|-------------------------|
| L1 (Foundation) | Manual-first, understand before automate, no AI shortcuts |
| L2 (Collaboration) | AI as Teacher/Student/Co-Worker, learning through interaction |
| L3 (Intelligence) | Pattern recognition, creating reusable intelligence (skills/subagents) |
| L4 (Orchestration) | Capstone, combining components, spec-driven development |

**Check for:**
- Layer declared in frontmatter
- Content matches layer expectations
- Layer progression through chapter (L1 → L2 → L3 → L4)
- No premature automation (L3 content in early lessons)

#### 3.5 Try With AI Effectiveness (1-10)

| Score | Criteria |
|-------|----------|
| 9-10 | Prompts directly extend lesson, specific, build skills |
| 7-8 | Good prompts, mostly connected to content |
| 5-6 | Generic prompts, loosely connected |
| 3-4 | Copy-paste prompts, don't match lesson |
| 1-2 | Missing or irrelevant prompts |

**Check for:**
- 2-3 prompts per lesson (not 1, not 5+)
- Prompts reference lesson content specifically
- Progressive difficulty across prompts
- "What you're learning" explanations present

#### 3.6 Assessment/Verification Quality (1-10)

| Score | Criteria |
|-------|----------|
| 9-10 | Clear verification at each step, students know if they succeeded |
| 7-8 | Good verification for most exercises |
| 5-6 | Some verification, students may be unsure of success |
| 3-4 | Weak verification, students can't tell if they're on track |
| 1-2 | No verification, students have no idea if they succeeded |

**Check for:**
- "Expected output" shown for commands
- "You should see X" confirmations
- Error states explained
- End-of-lesson checkpoint

### Step 4: Gap Analysis

After scoring, identify specific missing elements:

#### Content Gaps
- Missing examples (concept taught but not demonstrated)
- Missing hands-on (theory without practice)
- Missing "why" (what but not why it matters)
- Missing troubleshooting (happy path only)
- Missing transitions (lessons don't connect)

#### Structural Gaps
- Missing opening hook
- Missing running example continuity
- Missing "What's Next" closure
- Missing visual elements (all text, no diagrams/tables)
- Missing code examples for technical content

#### Pedagogical Gaps
- Objectives not assessed
- Cognitive overload unaddressed
- Layer mismatch (content doesn't match declared layer)
- Prerequisites not actually prerequisite
- Try With AI prompts disconnected from content

### Step 5: Generate Improvement Recommendations

For each gap, provide:

1. **Problem**: What's missing or wrong
2. **Impact**: How it affects learning (high/medium/low)
3. **Fix**: Specific action to address
4. **Effort**: Estimated work (low: <30min, medium: 30-90min, high: >90min)
5. **Priority**: 1 (critical), 2 (important), 3 (nice-to-have)

## Output Format

Generate analysis in this structure:

```markdown
# Chapter Evaluation: [Chapter Name]

## Executive Summary

[1 paragraph: Overall quality assessment, key strengths, critical issues, recommendation]

## Student Analysis

### Scores

| Dimension | Score | Verdict |
|-----------|-------|---------|
| Engagement | X/10 | [One-line summary] |
| Length | [Short/Right/Long] | [One-line summary] |
| Clarity | X/10 | [One-line summary] |
| Hands-On | X/10 | [One-line summary] |
| Progression | X/10 | [One-line summary] |
| Confidence | X/10 | [One-line summary] |

**Overall Student Experience**: X/10

### Detailed Findings

[Specific observations per dimension with examples from content]

### Student Pain Points

1. [Specific issue from student perspective]
2. [Specific issue from student perspective]
...

## Teacher Analysis

### Scores

| Dimension | Score | Verdict |
|-----------|-------|---------|
| Learning Objectives | X/10 | [One-line summary] |
| Cognitive Load | X/10 | [One-line summary] |
| Scaffolding | X/10 | [One-line summary] |
| Layer Appropriateness | X/10 | [One-line summary] |
| Try With AI | X/10 | [One-line summary] |
| Assessment | X/10 | [One-line summary] |

**Overall Pedagogical Quality**: X/10

### Detailed Findings

[Specific observations per dimension with examples from content]

### Pedagogical Concerns

1. [Specific issue from teacher perspective]
2. [Specific issue from teacher perspective]
...

## Gap Analysis

### Content Gaps

| Gap | Lesson(s) | Impact |
|-----|-----------|--------|
| [Missing element] | L0X | High/Med/Low |
...

### Structural Gaps

| Gap | Lesson(s) | Impact |
|-----|-----------|--------|
| [Missing element] | L0X | High/Med/Low |
...

### Pedagogical Gaps

| Gap | Lesson(s) | Impact |
|-----|-----------|--------|
| [Missing element] | L0X | High/Med/Low |
...

## Improvement Recommendations

### Priority 1 (Critical)

| # | Problem | Fix | Effort | Lesson(s) |
|---|---------|-----|--------|-----------|
| 1 | [Issue] | [Action] | Low/Med/High | L0X |
...

### Priority 2 (Important)

| # | Problem | Fix | Effort | Lesson(s) |
|---|---------|-----|--------|-----------|
| 1 | [Issue] | [Action] | Low/Med/High | L0X |
...

### Priority 3 (Nice-to-Have)

| # | Problem | Fix | Effort | Lesson(s) |
|---|---------|-----|--------|-----------|
| 1 | [Issue] | [Action] | Low/Med/High | L0X |
...

## Summary Metrics

| Metric | Value |
|--------|-------|
| Total Lessons | X |
| Average Word Count | X |
| Student Score | X/10 |
| Teacher Score | X/10 |
| Overall Score | X/10 |
| Critical Issues | X |
| Estimated Fix Time | X hours |
```

## Quality Reference

Compare evaluated chapters against high-quality reference lessons. The skill should automatically identify and read a reference lesson from Part 1 or Part 6 for comparison when available.

Reference lesson patterns to look for:
- `01-agent-factory-paradigm/01-digital-fte-revolution.md`
- `33-introduction-to-ai-agents/01-what-is-an-ai-agent.md`

## Resources

### references/

See `references/` for detailed rubrics:
- `student-rubric.md` - Detailed student perspective evaluation criteria
- `teacher-rubric.md` - Detailed teacher perspective evaluation criteria
- `word-count-benchmarks.md` - Word count guidelines by lesson type