Systematically elicits evaluation criteria across technical, team, ecosystem, security, deployment, integration, cost, and viability dimensions to prevent hype-driven framework selection. Produces structured requirement matrices, weighted scoring frameworks, and decision records with documented trade-offs.
Scanned 9/4/2026
Install to Claude Code
Skills Directorynpx -y skills add paulpas/agent-skill-router --skill framework-evaluation-criteria --agent claude-codeInstalls into .claude/skills of the current project.
Are you the author of Framework Evaluation Criteria?
Add the live security badge to your README — it updates automatically with every re-scan.
[](https://www.skillsdirectory.com/skills/paulpas-framework-evaluation-criteria)More formats (shields.io, HTML) on the badges page.
---
name: framework-evaluation-criteria
description: Systematically elicits evaluation criteria across technical, team, ecosystem, security, deployment, integration, cost, and viability dimensions to prevent hype-driven framework selection. Produces structured requirement matrices, weighted scoring frameworks, and decision records with documented trade-offs.
license: MIT
compatibility: opencode
metadata:
version: "1.0.0"
domain: coding
triggers: evaluation criteria, requirements elicitation, weighted scoring matrix, ADR decision record, technology selection framework, failure mode analysis, decision sustainability, tech comparison checklist
archetypes:
- tactical
- diagnostic
anti_triggers:
- brainstorming
- vague ideation
- code golf
response_profile:
verbosity: medium
directive_strength: high
abstraction_level: operational
role: implementation
scope: implementation
output-format: code
content-types: [code, guidance, examples, do-dont]
related-skills: framework-selection, requirement-driven-selection, framework-utilization, tool-evaluation-workflow
---
# Framework Evaluation Criteria
Structures and elicits evaluation criteria across eight dimensions to prevent hype-driven framework selection. When loaded, this skill makes the model act as a senior engineering evaluator — systematically extracting requirements, separating non-negotiables from scored criteria, building weighted evaluation matrices, analyzing failure modes, and producing decision records with documented trade-offs for any technology comparison.
## TL;DR Checklist
- [ ] Elicit requirements across all 8 dimensions (technical, team skills, ecosystem, security, deployment, integration, cost, viability) — never skip a dimension
- [ ] Separate knock-out criteria from scored criteria before evaluating any candidate
- [ ] Normalize weights so they sum to exactly 1.0 across all scoring criteria
- [ ] Run failure mode analysis on the top candidate(s) before finalizing
- [ ] Document the decision with an ADR including context, drivers, weighted scores, and consequences
- [ ] Involve at least two stakeholders in the evaluation — no solo framework selections
- [ ] Record "why not" for rejected candidates, not just "why" for the winner
- [ ] Validate knock-out criteria by running concrete verification steps (not opinions)
---
## When to Use
Use this skill when:
- Starting a framework comparison before scoring any candidates (this is always step zero)
- Selecting a new library or framework for a project with long-term maintenance needs
- During a technology spike where you need structured evaluation criteria up front
- Leading an architectural review that includes technology stack decisions
- The team has conflicting preferences and needs objective, requirement-driven criteria to resolve disagreements
- Before adopting a framework recommended solely because it is popular, new, or used at another company
- When migrating from an existing framework and need criteria to evaluate replacement candidates
- For build-vs-buy decisions where you must assess whether a framework fits your specific constraints
---
## When NOT to Use
Avoid this skill for:
- During active implementation of a framework already selected — use `framework-utilization` instead
- For throwaway prototypes or weekend projects where framework choice is irrelevant
- When requirements are already fully defined by another process or skill (go back to those, do not re-elicitation)
- For one-off script decisions where the same tool would work equally well (overhead outweighs benefit)
---
## Core Workflow
### Step 1: Elicit Requirements Across All 8 Dimensions
Systematically go through each evaluation dimension. For each dimension, write down specific, verifiable requirements — not preferences. Use the questions listed in the Evaluation Dimensions section below to guide elicitation. **Checkpoint:** Every dimension must have at least one requirement written down before proceeding. If a dimension is genuinely irrelevant (e.g., cost for an internal open-source tool), document why it was excluded explicitly.
### Step 2: Identify Knock-out Criteria
From the requirements collected in Step 1, identify which ones are non-negotiables. A knock-out criterion disqualifies any framework that fails it — there is no partial credit. Examples: "Must support Python 3.10+", "Must use an OSI-approved license", "Must run on Linux x86_64". **Checkpoint:** Every knock-out criterion must have a concrete, testable verification method (e.g., `python -c "import framework; print(framework.__version__)"`).
### Step 3: Construct Weighted Evaluation Matrix
Assign normalized weights to all non-knock-out criteria. Weights must sum to exactly 1.0. Use the Importance enum (LOW=1 through ESSENTIAL=5) to assign raw importance, then normalize. Score each candidate framework against every criterion on a 1–10 scale with evidence — no gut feelings. **Checkpoint:** The weighted sum for each candidate must be calculated and all candidates must pass knock-out checks before scoring is considered complete.
### Step 4: Analyze Failure Modes
For the top-scoring candidate(s), run failure mode analysis against all six documented failure modes. For each failure mode, determine whether the risk is present, its likelihood, and what mitigation applies. **Checkpoint:** If any high-severity failure mode is identified for the leading candidate with no viable mitigation, reconsider the selection before proceeding.
### Step 5: Document Decision with ADR
Produce a structured Architecture Decision Record in MADR format (see Decision Documentation Patterns below). The ADR must include context, decision drivers, knock-out criteria results, weighted scores for all candidates, and positive/negative consequences of the chosen direction. **Checkpoint:** All stakeholders have reviewed the ADR before status moves from `proposed` to `accepted`.
---
## Evaluation Dimensions (The Core Content)
Each dimension below describes what to assess, key questions to answer, common pitfalls, and a real example drawn from actual framework selection decisions.
### 1. Technical Capability Fit
Assess how well the framework's features align with your project's functional requirements. This is the most commonly evaluated dimension but also the most often done incompletely — teams compare feature lists without considering depth of capability.
**Key questions:**
- Does the framework support all required use cases, or only a subset?
- How deep is the API surface for each feature you need? (Can it do simple things easily AND complex things if needed?)
- What is the framework's extensibility model? Can you add features not built in?
- Are there performance characteristics that could become bottlenecks?
**Common pitfalls:**
- "Feature parity" thinking — assuming a feature exists because other frameworks have it
- Not testing the specific use case under realistic conditions (a POC exercise prevents this)
- Over-valuing "batteries included" when your project needs only one or two components
**Example:** When the Playwright team evaluated migrating from Selenium, they didn't just check "can it automate a browser?" — they assessed specific capabilities: auto-wait mechanisms, network interception, multi-tab handling, and mobile emulation. The deep API surface for Playwright's locator strategies (CSS, text, role, test-id) was a decisive factor because their tests needed multiple reliable targeting strategies.
### 2. Team Skills & Learning Curve
Evaluate the team's existing knowledge of the framework and estimate the time-to-productivity for unfamiliar ones. This dimension is frequently underestimated — it is one of the top causes of framework selection failure. A technically superior framework that your team cannot use effectively is worse than a good-enough framework they can master quickly.
**Key questions:**
- What fraction of the team has production experience with this framework?
- How many weeks until a mid-level developer on the team is productive (not expert)?
- Are there training resources, books, or community support for learning gaps?
- Is knowledge concentrated in one person (bus factor risk)?
**Common pitfalls:**
- Assuming "everyone can pick it up quickly" without evidence
- Ignoring the learning curve impact on delivery timelines for the current sprint/quarter
- Not distinguishing between initial learning and mastery depth
**Example:** When a team chose Go over Rust for a networking service, the deciding factor was that several engineers already had Go experience from internal tooling. The decision sustainability criteria explicitly noted: "The team can contribute meaningfully within 1–2 weeks without external training." A framework with zero team familiarity would require 40–80 hours of ramp-up per developer.
### 3. Ecosystem & Maintainership
Assess the health and trajectory of the framework's ecosystem — not just current activity but sustainability. A framework with a dead repository is a ticking time bomb regardless of how good it is today.
**Key questions:**
- What is the release cadence? (Weekly, monthly, quarterly, annually?)
- How many contributors are active in the last 90 days?
- How large and healthy is the third-party package/plugin ecosystem?
- Is documentation current and does it include migration guides from popular alternatives?
**Common pitfalls:**
- Looking at total GitHub stars instead of recent activity (stars don't equal health)
- Assuming a well-known framework name guarantees active maintenance
- Not checking if the primary maintainer is employed by a company that could abandon it
**Example:** The ThoughtWorks Technology Radar categorizes frameworks into "Assess," "Trial," and "Adopt" rings precisely to surface ecosystem maturity. A framework in "Trial" might have strong technical merit but insufficient production validation or community breadth. The radar approach explicitly rewards frameworks that demonstrate real-world adoption beyond the core team.
### 4. Security Posture
Evaluate the framework's security track record, dependency graph risk, and licensing compliance. This dimension should not be an afterthought — security flaws in a foundational framework cascade into every module that uses it.
**Key questions:**
- What is the CVE history? Are there known unpatched vulnerabilities?
- How deep is the dependency tree? (More dependencies = larger attack surface)
- Does the framework verify signatures on published artifacts?
- Is the license OSI-approved and compatible with your project's distribution model?
**Common pitfalls:**
- Assuming well-known frameworks have no security issues ("big projects can't be vulnerable")
- Ignoring transitive dependency risks (the framework itself may be clean, but its dependencies might not be)
- Not checking if the license has copyleft clauses that affect your distribution
**Example:** When selecting a CSS framework for a public-facing application, the team evaluated whether Tailwind CSS had any known XSS vectors in its utility class processing. They also verified that its MIT license was compatible with distributing compiled stylesheets without source disclosure — a concern with GPL-licensed alternatives.
### 5. Deployment Environment Fit
Assess how well the framework integrates with your deployment infrastructure, resource constraints, and operational requirements. A framework that works beautifully on your local machine may fail in production due to platform incompatibilities or resource demands.
**Key questions:**
- Does the framework support your target platforms (Linux, Windows, macOS, containerized)?
- What are the minimum and recommended resource requirements?
- Does it support containerization (Docker images, multi-stage builds)?
- Are there built-in monitoring/integration hooks (OpenTelemetry, Prometheus metrics, structured logging)?
**Common pitfalls:**
- Assuming cross-platform because marketing says so (verify with actual CI tests)
- Ignoring startup time requirements for serverless or edge deployment scenarios
- Not evaluating whether the framework's logging format matches your log aggregation pipeline
**Example:** A team evaluated Next.js versus a custom React SSR setup for deployment to Vercel and AWS Lambda. The decision hinged on Next.js's built-in support for edge runtime, automatic code splitting, and Image component optimization — all critical for their serverless deployment target with cold-start sensitivity.
### 6. Integration Requirements
Evaluate how the framework connects with your existing systems, data formats, authentication patterns, and CI/CD pipelines. Frameworks don't exist in isolation — integration friction is a real cost that compounds over time.
**Key questions:**
- Does the framework's API align with existing service contracts (REST, GraphQL, gRPC)?
- What data formats does it natively support (JSON, Protocol Buffers, Avro)?
- Can it integrate with your authentication system (OAuth2, JWT, SAML, LDAP)?
- Is there CI/CD plugin or CLI tooling for automated testing and deployment?
**Common pitfalls:**
- Assuming standard protocols "just work" without verifying specific version compatibility
- Overlooking auth pattern alignment (a framework may support OAuth2 but not the specific grant flow your identity provider uses)
- Not evaluating whether the framework's test runner integrates with your existing CI pipeline
**Example:** When selecting a GraphQL client, Apollo Client won partly because it integrated cleanly with their existing JWT-based auth flow and had a caching strategy compatible with their Redux state management. The alternative frameworks either required significant adapter code or lacked the specific cache configuration patterns the team needed.
### 7. Cost & Total Cost of Ownership
Go beyond licensing fees to calculate the true cost: infrastructure, training, support contracts, and opportunity cost of developer time spent on framework-related work versus business logic.
**Key questions:**
- What is the licensing model? (MIT, Apache 2.0, GPL, commercial, dual-license?)
- Does the framework increase infrastructure costs (e.g., requires more compute, specialized hardware)?
- Will you need paid support contracts or managed service subscriptions?
- What is the opportunity cost — how much developer time will framework mastery vs. business development consume?
**Common pitfalls:**
- Focusing only on license fees while ignoring infrastructure and training costs
- Assuming all open-source licenses are equally "free" (GPL copyleft has real legal costs)
- Not estimating support burden (a framework with no commercial backing means your team absorbs all support)
**Example:** PostgreSQL was selected over a proprietary database partly because the total cost of ownership — including zero licensing fees, existing team expertise, and active community support — was dramatically lower despite comparable performance. The absence of vendor lock-in meant future negotiation leverage on any managed service pricing.
### 8. Long-Term Viability
Assess whether the framework is likely to remain relevant, maintained, and compatible with your roadmap over the expected lifecycle of the project (typically 3–7 years). A framework's trajectory matters as much as its current features.
**Key questions:**
- Is there a public roadmap aligned with your projected needs?
- What is the adoption trend — growing, plateaued, or declining?
- How complex would an exit strategy be if you need to migrate away later?
- Is the license stable (no history of license changes)?
**Common pitfalls:**
- Choosing a framework because it has 50k GitHub stars when the star count peaked two years ago
- Ignoring the migration cost — a framework with deep coupling makes exit difficult
- Assuming current maintainer commitment guarantees future stability (check who pays for development)
**Example:** The adr/adr decision sustainability criteria explicitly evaluate: "Will this framework likely still be maintained in 3 years?" and "What is our exit path?" A framework backed by a single startup with no alternative funding source received lower viability scores than one with multiple contributing companies, regardless of feature parity.
---
## Criteria Data Structure
Represent evaluation criteria programmatically to enable consistent scoring and automated matrix calculation:
```python
from dataclasses import dataclass, field
from enum import IntEnum
from typing import Optional
class Importance(IntEnum):
"""Importance levels for evaluation criteria. Higher = more weight."""
LOW = 1
MODERATE = 2
HIGH = 3
CRITICAL = 4
ESSENTIAL = 5
@dataclass
class EvaluationCriterion:
"""A single evaluation criterion for framework selection."""
name: str
dimension: str # One of the 8 dimensions
importance: Importance
weight: float # Normalized weight (sums to 1.0 across all scoring criteria)
is_knockout: bool = False
description: str = ""
validation_method: Optional[str] = None # How to verify compliance
def __post_init__(self):
if self.is_knockout and self.weight != 0.0:
raise ValueError(
f"Knock-out criterion '{self.name}' must have weight=0. "
"Knock-outs disqualify; scoring criteria receive weights."
)
if self.is_knockout and not self.validation_method:
raise ValueError(
f"Knock-out criterion '{self.name}' requires a validation_method. "
"A knock-out must be concretely testable."
)
@dataclass
class FrameworkCandidate:
"""A framework being evaluated against all criteria."""
name: str
version: str
license_type: str
criteria_scores: dict[str, float] = field(default_factory=dict) # criterion_name -> score (1-10)
knockout_failures: list[str] = field(default_factory=list)
@property
def weighted_score(self) -> float:
"""Calculate total weighted score. Only scoring criteria contribute."""
return sum(
self.criteria_scores.get(c.name, 0.0) * c.weight
for c in FrameworkEvaluator.all_criteria
)
@property
def is_viable(self) -> bool:
"""Check if framework passes all knock-out criteria."""
return len(self.knockout_failures) == 0
def add_score(self, criterion_name: str, score: float, max_score: float = 10.0) -> None:
"""Add a scored criterion result (clamped to valid range)."""
if not (1.0 <= score <= max_score):
raise ValueError(f"Score {score} for '{criterion_name}' must be between 1 and {max_score}")
self.criteria_scores[criterion_name] = round(score / max_score * 10, 2)
def add_knockout_failure(self, criterion_name: str, reason: str) -> None:
"""Record a knock-out failure with explanatory context."""
self.knockout_failures.append(f"{criterion_name}: {reason}")
class FrameworkEvaluator:
"""Framework evaluation engine implementing the weighted matrix pattern."""
# Dimension groupings for structured elicitation
DIMENSIONS = {
"technical_capability": [
"Core feature completeness",
"API design quality",
"Performance characteristics",
"Extensibility points",
],
"team_skills": [
"Current team familiarity",
"Learning curve estimate",
"Training resource availability",
"Knowledge distribution risk",
],
"ecosystem": [
"Active development velocity",
"Community size",
"Third-party package ecosystem",
"Documentation quality",
],
"security": [
"CVE history",
"Dependency graph depth",
"Code signing practices",
"Supply chain security",
],
"deployment_fit": [
"Platform compatibility",
"Resource requirements",
"Containerization support",
"Monitoring integrations",
],
"integration": [
"API compatibility with existing systems",
"Data format support",
"Auth pattern alignment",
"CI/CD pipeline integration",
],
"cost": [
"Licensing model",
"Infrastructure cost impact",
"Training cost",
"Support contract needs",
],
"long_term_viability": [
"Roadmap alignment",
"Adoption trend",
"Exit strategy complexity",
"License stability",
],
}
all_criteria: list[EvaluationCriterion] = []
@classmethod
def normalize_weights(cls, raw_importances: dict[str, Importance]) -> dict[str, float]:
"""Convert importance scores to normalized weights summing to 1.0."""
total = sum(imp.value for imp in raw_importances.values())
if total == 0:
raise ValueError("Sum of importance values cannot be zero")
return {name: round(imp.value / total, 4) for name, imp in raw_importances.items()}
@classmethod
def run_evaluation(
cls,
criteria: list[EvaluationCriterion],
candidates: list[FrameworkCandidate],
) -> dict[str, FrameworkCandidate]:
"""Run complete evaluation and return scored candidates sorted by score descending."""
cls.all_criteria = criteria
results: dict[str, FrameworkCandidate] = {}
for candidate in candidates:
# Knock-out check first — non-negotiables are binary pass/fail
if not candidate.is_viable:
continue # Excluded entirely
# Calculate weighted score from scoring criteria only
candidate.weighted_score = sum(
candidate.criteria_scores.get(c.name, 0.0) * c.weight
for c in criteria
)
results[candidate.name] = candidate
return dict(
sorted(results.items(), key=lambda item: item[1].weighted_score, reverse=True)
)
@classmethod
def generate_summary(cls, candidates: list[FrameworkCandidate]) -> str:
"""Generate a human-readable evaluation summary table."""
lines = [
"=" * 70,
"FRAMEWORK EVALUATION SUMMARY",
"=" * 70,
f"{'Candidate':<25} {'Version':<12} {'Score':>8} {'Viable':>6}",
"-" * 70,
]
for candidate in sorted(candidates, key=lambda c: getattr(c, "weighted_score", 0), reverse=True):
score = getattr(candidate, "weighted_score", 0)
viable = "Yes" if candidate.is_viable else "No"
lines.append(f"{candidate.name:<25} {candidate.version:<12} {score:>8.3f} {viable:>6}")
if not candidate.is_viable:
for failure in candidate.knockout_failures:
lines.append(f" ❌ Knock-out failed: {failure}")
lines.append("=" * 70)
return "\n".join(lines)
```
### Example: Evaluating Three Framework Candidates
```python
# Define knock-out criteria (weight=0, disqualify on failure)
knockout_criteria = [
EvaluationCriterion(
name="OSI-approved license",
dimension="security",
importance=Importance.ESSENTIAL,
weight=0.0,
is_knockout=True,
description="Must use an OSI-approved open-source license (MIT, Apache 2.0, BSD)",
validation_method="Check LICENSE file in repository for SPDX identifier",
),
EvaluationCriterion(
name="Python 3.10+ support",
dimension="technical_capability",
importance=Importance.CRITICAL,
weight=0.0,
is_knockout=True,
description="Must support Python 3.10 or higher (project minimum)",
validation_method="pip install <framework> && python -c 'import framework; print(framework.__version__)'",
),
]
# Define scoring criteria (with normalized weights)
scoring_criteria = [
EvaluationCriterion(
name="Core feature completeness",
dimension="technical_capability",
importance=Importance.HIGH,
weight=0.15, # Normalized: 3 / (3+2+2+...+1) = 0.15
description="Does the framework cover all required use cases with sufficient depth?",
),
EvaluationCriterion(
name="Team familiarity",
dimension="team_skills",
importance=Importance.HIGH,
weight=0.12,
description="Current team's production experience level with this framework",
),
EvaluationCriterion(
name="Active development velocity",
dimension="ecosystem",
importance=Importance.CRITICAL,
weight=0.18,
description="Release cadence and active contributor count in last 90 days",
),
EvaluationCriterion(
name="Documentation quality",
dimension="ecosystem",
importance=Importance.MODERATE,
weight=0.08,
description="Clarity, completeness, and searchability of documentation",
),
EvaluationCriterion(
name="Deployment compatibility",
dimension="deployment_fit",
importance=Importance.HIGH,
weight=0.13,
description="Supports containerization and target deployment platforms",
),
EvaluationCriterion(
name="Integration ease",
dimension="integration",
importance=Importance.MODERATE,
weight=0.09,
description="Fits with existing auth, data formats, and CI/CD pipeline",
),
EvaluationCriterion(
name="Total cost of ownership",
dimension="cost",
importance=Importance.HIGH,
weight=0.12,
description="Licensing + infrastructure + training costs over 3-year horizon",
),
EvaluationCriterion(
name="Long-term viability",
dimension="long_term_viability",
importance=Importance.CRITICAL,
weight=0.13,
description="Adoption trend, roadmap stability, and exit strategy complexity",
),
]
# Verify weights sum to 1.0
total_weight = sum(c.weight for c in scoring_criteria)
assert total_weight == 1.0, f"Weights must sum to 1.0, got {total_weight}"
# Define candidates
candidates = [
FrameworkCandidate(name="FastAPI", version="0.104.1", license_type="MIT"),
FrameworkCandidate(name="Django", version="4.2.3", license_type="BSD-3-Clause"),
FrameworkCandidate(name="Flask", version="3.0.0", license_type="BSD-3-Clause"),
]
# Score each candidate (1-10 scale, evidence-based)
# FastAPI scores
fastapi = candidates[0]
fastapi.add_score("Core feature completeness", 9.0)
fastapi.add_score("Team familiarity", 5.0) # Team new to it
fastapi.add_score("Active development velocity", 9.0)
fastapi.add_score("Documentation quality", 9.0)
fastapi.add_score("Deployment compatibility", 8.0)
fastapi.add_score("Integration ease", 7.0)
fastapi.add_score("Total cost of ownership", 8.0)
fastapi.add_score("Long-term viability", 8.0)
# Django scores
django = candidates[1]
django.add_score("Core feature completeness", 9.5)
django.add_score("Team familiarity", 7.0) # Moderate experience
django.add_score("Active development velocity", 8.0)
django.add_score("Documentation quality", 10.0)
django.add_score("Deployment compatibility", 8.0)
django.add_score("Integration ease", 8.0)
django.add_score("Total cost of ownership", 6.0) # Heavier, more infra cost
django.add_score("Long-term viability", 9.5)
# Flask scores
flask = candidates[2]
flask.add_score("Core feature completeness", 7.0)
flask.add_score("Team familiarity", 8.0) # High experience
flask.add_score("Active development velocity", 7.0)
flask.add_score("Documentation quality", 7.0)
flask.add_score("Deployment compatibility", 9.0)
flask.add_score("Integration ease", 9.0) # Minimal, fits anything
flask.add_score("Total cost of ownership", 7.0)
flask.add_score("Long-term viability", 8.5)
# Run evaluation
results = FrameworkEvaluator.run_evaluation(scoring_criteria, candidates)
print(FrameworkEvaluator.generate_summary(candidates))
```
---
## Decision Documentation Patterns
Use the MADR (Milestone-driven Architecture Decision Records) template adapted for framework selection decisions. Every framework comparison must produce an ADR — even if the decision is to stay with the current framework.
### ADR Generation Class
```python
from datetime import date
from dataclasses import dataclass, field
from typing import Optional
@dataclass
class FrameworkDecisionADR:
"""Architecture Decision Record for framework selection."""
title: str
status: str = "proposed" # proposed | rejected | accepted | deprecated | superseded
deciders: list[str] = field(default_factory=list)
date: str = field(default_factory=lambda: date.today().isoformat())
context_description: str = ""
decision_drivers: list[str] = field(default_factory=list)
# Evaluation data
knock_out_criteria: list[str] = field(default_factory=list)
weighted_criteria: dict[str, float] = field(default_factory=dict)
# Results
candidates_evaluated: list[str] = field(default_factory=list)
selected_framework: Optional[str] = None
positive_consequences: list[str] = field(default_factory=list)
negative_consequences: list[str] = field(default_factory=list)
@property
def markdown(self) -> str:
"""Generate the ADR in MADR-compatible markdown format."""
sections = [
f"# {self.title}",
"",
f"* Status: {self.status}",
f"* Deciders: {', '.join(self.deciders)}",
f"* Date: {self.date}",
"",
"---",
"",
"## Context and Problem Statement",
self.context_description,
"",
]
if self.decision_drivers:
sections.append("## Decision Drivers")
for driver in self.decision_drivers:
sections.append(f"* {driver}")
sections.append("")
if self.knock_out_criteria:
sections.append("## Knock-out Criteria")
for criterion in self.knock_out_criteria:
sections.append(f"* [ ] {criterion}")
sections.append("")
if self.weighted_criteria:
sections.append("## Weighted Evaluation Criteria")
total_weight = sum(self.weighted_criteria.values()) or 1.0
for criterion, weight in sorted(
self.weighted_criteria.items(), key=lambda x: x[1], reverse=True
):
normalized_pct = (weight / total_weight) * 100
marker = " **KNOCK-OUT**" if criterion in self.knock_out_criteria else ""
sections.append(
f"* **{criterion}**: {normalized_pct:.0f}% weight{marker}"
)
sections.append("")
if self.candidates_evaluated:
sections.append("## Candidates Evaluated")
for candidate in self.candidates_evaluated:
marker = " <-- SELECTED" if candidate == self.selected_framework else ""
sections.append(f"* {candidate}{marker}")
sections.append("")
if self.positive_consequences or self.negative_consequences:
sections.append("## Consequences")
sections.append("")
if self.positive_consequences:
sections.append("### Positive (Benefits)")
for consequence in self.positive_consequences:
sections.append(f"+ {consequence}")
sections.append("")
if self.negative_consequences:
sections.append("### Negative (Drawbacks / Risks)")
for consequence in self.negative_consequences:
sections.append(f"- {consequence}")
sections.append("")
sections.append("---")
return "\n".join(sections)
# Example: Generate an ADR from evaluation results
adr = FrameworkDecisionADR(
title="Select FastAPI for Internal API Service Framework",
status="proposed",
deciders=["alice (tech lead)", "bob (backend engineer)", "carol (platform engineer)"],
context_description=(
"Our internal services require a lightweight, high-performance HTTP framework "
"for building REST APIs and microservice endpoints. The current service uses "
"a legacy framework with slow startup times and limited async support. We need "
"a framework that supports Python 3.10+, runs efficiently in containers, and "
"integrates with our existing OpenTelemetry observability stack."
),
decision_drivers=[
"Must support async/await natively for high-concurrency workloads",
"Must run on Linux x86_64 in Docker containers",
"Must use an OSI-approved license (MIT or Apache 2.0 preferred)",
"Team needs to achieve productivity within 2 weeks of introduction",
"Must integrate with existing JWT-based authentication",
],
knock_out_criteria=[
"OSI-approved license",
"Python 3.10+ support",
"Linux container compatibility",
],
weighted_criteria={
"Core feature completeness": 0.15,
"Team familiarity": 0.12,
"Active development velocity": 0.18,
"Documentation quality": 0.08,
"Deployment compatibility": 0.13,
"Integration ease": 0.09,
"Total cost of ownership": 0.12,
"Long-term viability": 0.13,
},
candidates_evaluated=["FastAPI", "Django", "Flask"],
selected_framework="FastAPI",
positive_consequences=[
"4x faster request throughput compared to current legacy framework",
"Automatic OpenAPI/Swagger documentation generation reduces API docs burden",
"Built-in dependency injection simplifies testing and mocking",
"Async-native design enables efficient concurrent request handling",
],
negative_consequences=[
"Smaller ecosystem of third-party extensions compared to Django",
"Team has limited prior async experience — requires learning sprint",
"No built-in ORM — must select and integrate a separate database layer",
"Rapidly evolving API may introduce breaking changes between minor versions",
],
)
print(adr.markdown)
```
### ADR Workflow for Framework Selection
1. **Draft the ADR before evaluation begins** — Set context and decision drivers upfront so they don't change based on results (prevents confirmation bias).
2. **Fill in knock-out criteria with validation methods** — Each must be concretely testable.
3. **Populate weighted criteria and scores during evaluation** — Use the matrix from Step 3 above.
4. **Document consequences explicitly** — Both positive and negative. The MADR pattern treats consequences as equally important to the decision itself.
5. **Set status to `accepted` only after stakeholder review** — Keep it `proposed` until all deciders have reviewed and agreed.
---
## Failure Mode Analysis
Research into real-world framework selection failures identifies six common failure modes. For each, this section provides detection heuristics, prevention strategies, and a real example from ADR repositories.
### 1. Hype-Driven / Trend-Following Selection
**Detection heuristics:**
- The primary justification is "it's popular" or "everyone is using it"
- No specific project requirements were elicited before comparing candidates
- The recommendation comes from outside the team without stakeholder validation
- GitHub stars are cited as the primary evidence of quality
**Prevention strategy:**
Anchor every evaluation decision to requirement-derived criteria (Step 1 above). A framework that scores highly on requirement-weighted criteria may differ significantly from one chosen based on popularity. If you cannot articulate three specific project requirements that this framework satisfies better than alternatives, the selection is likely hype-driven.
**Real example:** An adr/adr decision record shows a team that chose React because it was trending in 2014, then documented in retrospective: "We adopted React before validating that its component model fit our data-intensive dashboard needs. We spent six months refactoring when we realized a simpler templating approach would have been more productive." The ADR later became a reference for "hype detection" in framework selection decisions.
### 2. Team Skill Mismatch
**Detection heuristics:**
- Zero team members have production experience with the framework
- The estimated learning curve is not factored into sprint timelines
- No training budget or resources are allocated for ramp-up
- The framework's documentation assumes prior knowledge of concepts the team lacks
**Prevention strategy:**
Conduct a skill gap analysis matrix. For each team member, rate their familiarity (none / basic / intermediate / expert) with every candidate framework. Summarize as an average and compare against the estimated productivity timeline. If the weighted average familiarity is "none" for the leading candidate, mandate a learning sprint or select a more familiar alternative.
**Real example:** A Go language selection ADR documented: "The team chose Go over Rust even though Rust had superior memory safety guarantees. Three engineers had production Go experience from internal CLI tooling, while zero had shipped Rust in production. The decision sustainability criteria rated Go's 'team productivity within 1–2 weeks' as critical and Rust's as uncertain."
### 3. Ecosystem Immaturity
**Detection heuristics:**
- Fewer than 50 contributors with activity in the last 90 days
- Releases are spaced more than 6 months apart
- Documentation is incomplete or contains outdated examples
- The framework is newer than 12 months without significant production adoption evidence
**Prevention strategy:**
Track GitHub metrics over time (not just current snapshot). Look at release cadence history, issue resolution times, and contributor retention. Use the ThoughtWorks Technology Radar classification as an external validation — frameworks in "Assess" should be treated as higher risk than those in "Trial" or "Adopt." Require at least one proof-of-concept exercise that stresses real-world usage before adoption.
**Real example:** A CSS framework comparison ADR noted: "We evaluated three modern CSS frameworks. One had an innovative architecture but only 12 contributors and releases every 8 months. We rated it 'high risk' on ecosystem maturity despite superior technical design. The selected framework had a larger team and weekly release cadence, which mattered for long-term support."
### 4. Vendor Lock-In
**Detection heuristics:**
- The framework's proprietary features are central to the architecture (not optional plugins)
- Data formats or APIs are not publicly documented
- Migration would require significant rewrites because of deep coupling
- The licensing terms change unexpectedly or include usage-based fees
**Prevention strategy:**
Conduct a portability assessment. Map every proprietary feature used in the architecture to a potential alternative. If you cannot describe an exit path with concrete steps, flag it as high lock-in risk. Prefer frameworks with open standards and well-documented data formats. For managed services, verify that data export tools exist and are maintained independently of the service itself.
**Real example:** An adr/adr record documented: "We chose PostgreSQL over a proprietary time-series database because PostgreSQL's license (PostgreSQL License) allows unrestricted use, modification, and redistribution. The proprietary alternative required annual per-node licensing and had no documented data export path for migration. Even though the proprietary option had slightly better query performance, lock-in risk was rated 'critical' in our decision matrix."
### 5. Over-Engineering vs Under-Specification
**Detection heuristics:**
- The selected framework includes features you will never use (bloat)
- OR: the framework requires building too much custom infrastructure that should be built-in
- Architecture decisions are made based on anticipated future needs rather than current requirements
- There is no "good enough" threshold defined for any evaluation dimension
**Prevention strategy:**
Define a "good enough" threshold per criterion — the minimum acceptable level that satisfies your current needs. Any framework meeting this threshold receives maximum score regardless of how much it exceeds it. Conversely, frameworks that force you to build custom infrastructure for basic functionality should be penalized. Balance is achieved when the framework's capability surface matches your requirement surface without significant overhang or gaps.
**Real example:** A team evaluating backend frameworks documented: "Django offered a complete ORM, admin panel, and authentication system out of the box — we only needed the REST API layer. The extra features added 200MB to the deployment image. Flask required building our own ORM, auth, and admin tools from scratch. We selected FastAPI because its capability surface matched our actual requirements: fast HTTP framework with async support, no unnecessary baggage."
### 6. Ignoring Consequences
**Detection heuristics:**
- The ADR or decision record contains only positive consequences (or none at all)
- No negative consequences or trade-offs are documented
- Stakeholders who dissented were not recorded
- There is no plan for monitoring the decision's effectiveness over time
**Prevention strategy:**
The MADR ADR template explicitly requires a Consequences section with both benefits and drawbacks. Enforce this as a review gate: if an ADR lacks negative consequences, it cannot be marked `accepted`. Additionally, schedule a 90-day retrospective to validate whether the decision's outcomes match the documented expectations. Record actual outcomes against expected consequences to improve future decisions.
**Real example:** The adr/adr repository explicitly includes decision sustainability criteria that evaluate "What happens when this framework stops being maintained?" and "Are consequences documented for both adoption and rejection?" A well-documented example from the repository shows a team choosing an alternative even though it scored lower on technical merit, because they documented: "Negative consequence: We accept slower initial development velocity for significantly lower long-term maintenance burden."
---
## Constraints
### MUST DO
- Cover all 8 evaluation dimensions during elicitation (do not skip team skills or security because they feel subjective)
- Separate knock-out criteria from scored criteria clearly (knock-outs disqualify entirely; scores enable comparison)
- Normalize weights so they sum to exactly 1.0 across all scoring criteria (validate with an assertion)
- Include at least one failure mode check before finalizing any selection decision (run the full six-mode analysis on the top candidate)
- Document the decision with an ADR that includes both positive AND negative consequences
- Involve at least two stakeholders in the evaluation and record all deciders in the ADR
- Score each criterion with evidence — not opinions ("documentation is good" → "documentation covers 18 of 20 features with code examples")
### MUST NOT DO
- Start comparison without completing requirements elicitation first (Step 1 is mandatory, not optional)
- Use unweighted scoring when dimension importance clearly varies across your project (always use weights)
- Skip the knock-out criteria check — frameworks failing knock-out criteria must be excluded entirely, no exceptions
- Document only the final decision without trade-off analysis (the "why not" for rejected candidates matters as much as "why" for the winner)
- Let a single person make the framework selection without documented stakeholder input
- Confuse framework features with evaluation criteria (features are what you evaluate; criteria are how you decide they matter)
---
## Output Template
When this skill is active, your output must contain all five sections below in order. Do not skip any section or combine them — each serves a distinct purpose in the decision-making process.
1. **Requirements Elicitation Report** — Structured summary covering all 8 dimensions with specific findings per dimension. For each dimension, list the requirements gathered, their importance level (LOW through ESSENTIAL), and whether they are knock-out or scored criteria. If any dimension was excluded, document why explicitly.
2. **Knock-out Criteria List** — Explicit list of non-negotiables with: criterion name, description, validation method, and pass/fail status for each candidate framework. Any candidate failing a knock-out must be immediately disqualified and noted.
3. **Weighted Evaluation Matrix** — Table showing candidates × criteria with scores (1–10), weights, and weighted sub-scores. Include total weighted score per candidate sorted descending. Weights must sum to 1.0 — state the sum explicitly. Scores must include evidence citations (e.g., "9/10: covers 18 of 20 features, verified via docs audit").
4. **Failure Mode Assessment** — For the top candidate(s), assess all six failure modes with: risk present (yes/no), likelihood (low/medium/high), and mitigation strategy (specific actions to reduce risk). If any high-severity risk has no viable mitigation, flag this prominently for reconsideration.
5. **ADR Draft** — Complete Architecture Decision Record in MADR-compatible markdown format, ready for team review. Include context, decision drivers, knock-out results, weighted scores, all candidates evaluated with selection marker, and both positive and negative consequences. Status should be `proposed` pending stakeholder sign-off.
---
## Related Skills
| Skill | Purpose |
|---|---|
| `framework-selection` | After criteria are defined, use this skill to apply AHP and advanced scoring methods for nuanced multi-criteria decisions |
| `requirement-driven-selection` | Alternative methodology — focuses on measurable criteria with evidence-based validation instead of subjective scoring |
| `framework-utilization` | After selection, use this skill for learning and leveraging the chosen framework effectively in production |
| `tool-evaluation-workflow` | Broader tool evaluation including CI/CD tools, testing frameworks, and infrastructure — not limited to application frameworks |
---
## Live References
- [MADR ADR Project — Architecture Decision Records](https://github.com/joelparkerhenderson/architecture_decision_record) — The definitive ADR template repository with real-world examples
- [arc42 — Template for Software Architecture Documentation](https://www.arc42.org/) — Section 1.2 (Quality Goals) and decision sustainability criteria
- [Decision Sustainability Criteria — adr/adr Repository](https://github.com/adr/adr/blob/master/doc/5-decision-sustainability-criteria.md) — Formal criteria for evaluating long-term framework viability
- [Technology Radar — ThoughtWorks](https://www.thoughtworks.com/radar) — Ring-based framework maturity classification (Assess → Trial → Adopt → Hold)
- [ADR Examples — Real-world Technology Selection Decisions](https://github.com/joelparkerhenderson/architecture_decision_record/tree/main/examples) — Actual ADRs for Playwright vs Selenium, Go vs Rust selection, CSS framework comparisons
- [Decision Records as Architectural Knowledge Management](https://goalkicker.com/ArchitectureDecisionRecordBook/) — Practical guide to writing effective ADRs with the MADR format
---
*This skill is designed to be loaded before any framework comparison begins. Its output becomes the structured foundation upon which scoring, trade-off analysis, and decision documentation are built.*
Is this your skill, or is something wrong with this listing? Request removal or report an issue. Author removals are honored within 72 hours.
No comments yet. Be the first to comment!
