Assessment Engineering

---
name: assessment-engineering
description: "An assessment engineering skill used by the quiz-maker agent. Provides item type design guides, distractor psychology, rubric construction, and formative/summative assessment strategies. Used for 'quiz design,' 'assessment items,' 'rubrics,' 'exam creation,' and related topics."
---

# Assessment Engineering — Assessment Engineering Methodology

Expert knowledge used by the quiz-maker agent when designing formative and summative assessments.

## Why Assessment Engineering

Good assessment **accurately measures** learner understanding while simultaneously **reinforcing learning**. Poor assessment rewards memorization; good assessment rewards understanding.

## Item Type Design Guide

### 1. Multiple Choice

| Element | Rule |
|---------|------|
| **Stem** | Must be a clear question when read independently |
| **Key (Correct Answer)** | Must be unambiguously the best answer |
| **Distractors** | Plausible but clearly incorrect (see below) |
| **Number of Options** | 4 (5 adds meaningless wrong answers; 3 gives a 33% guess rate) |
| **Option Length** | Keep similar length (longest option being correct is a clue leak) |

### Distractor Design Psychology

| Distractor Type | Purpose | Example (Q: "Which HTTP method deletes a resource in REST?") |
|----------------|---------|------|
| **Common Misconception** | Diagnose incorrect knowledge | "POST" (confusing create and delete) |
| **Partial Answer** | Diagnose superficial understanding | "PUT" (confusing update and delete) |
| **Related Concept** | Diagnose concept discrimination | "PATCH" (partial update, similar concept) |
| **Correct**: | | "DELETE" |

### 2. Code Writing Problems

```
[Scenario Setup]
Write a function that meets the following requirements.

[Input/Output Specification]
- Input: [Type and range]
- Output: [Type and conditions]
- Constraints: [Time/space complexity, restrictions]

[Example]
Input: [Example input]
Output: [Example output]

[Grading Criteria]
- [ ] Correctness: Passes all test cases
- [ ] Code Quality: Readability, naming
- [ ] Efficiency: Time/space complexity
```

### 3. Scenario-Based Problems

```
[Scenario]
You are in the following situation. [Concrete situation description]

[Question]
What is the most appropriate [action/design/solution] in this situation?

[Evaluation Perspective]
- Problem identification ability
- Solution strategy appropriateness
- Trade-off awareness
```

## Formative vs. Summative Assessment

| Attribute | Formative | Summative |
|-----------|----------|-----------|
| **Timing** | During learning (every lesson) | After learning (end of module) |
| **Purpose** | Check understanding, immediate correction | Measure achievement, assign grades |
| **Atmosphere** | Low-stakes, practice | Formal, final |
| **Item Count** | 3-5 | 10-20 |
| **Feedback** | Immediate, detailed | Post-completion, aggregate |
| **Retakes** | Unlimited | 1-2 |
| **Bloom's Level** | 1-3 (Remember, Understand, Apply) | 3-6 (Apply through Create) |

### Formative Assessment Design Principles

1. **3-5 items at the end of each lesson**: Confirm key concept understanding
2. **Immediate feedback**: Correct/incorrect + "Why this is the answer" explanation
3. **Learning-directed wrong answers**: "Review this section: [link]"
4. **Difficulty ordering**: Easy to hard (builds confidence)

### Summative Assessment Design Principles

1. **100% learning objective coverage**: Every objective mapped to at least one item
2. **Bloom's distribution**: Remember 20% + Understand 20% + Apply 30% + Analyze/Evaluate/Create 30%
3. **Time allocation**: 2-3 minutes per item, 30-60 minutes total
4. **Practical weighting**: Labs/projects should account for 40-60% of the total score

## Rubric Construction

### Analytic Rubric Template

| Criterion | Excellent (4) | Good (3) | Satisfactory (2) | Needs Improvement (1) |
|-----------|-------------|---------|-----------------|---------------------|
| **Functionality** | All requirements met | 90% met | 70% met | Below 50% |
| **Code Quality** | Clean, consistent style | Mostly clean | Some inconsistency | Low readability |
| **Error Handling** | All edge cases | Major cases | Basic cases only | No error handling |
| **Documentation** | Detailed comments + README | Key function comments | Minimal comments | No comments |

### Rubric Writing Rules

1. **Describe observable behaviors**: Instead of "understood well," use "can explain..."
2. **Clear distinctions between levels**: Each level must be unambiguously different
3. **3-5 criteria**: Too many makes assessment impractical
4. **Share with learners in advance**: Showing the rubric before assessment sets learning direction

## Feedback Design

### Effective Feedback Formula: EEC

```
E (Evidence): "Your code lacks error handling"
E (Effect): "In this case, the program will crash if the user enters an empty value"
C (Change): "Wrap it in a try-except block and show the user a guidance message"
```

### Feedback Rules

- **Specific**: Instead of "Good job," say "Your variable names clearly describe their purpose"
- **Immediate**: As soon as possible after submission (automated tests preferred)
- **Action-oriented**: Tell the learner what to do next
- **Positive first**: What was done well -> Areas for improvement -> Next steps