Ai Llm Framework Ecosystem

---




name: ai-llm-framework-ecosystem
description: Evaluates AI/LLM framework ecosystems (LangChain, LlamaIndex, CrewAI,
  DSPy, Microsoft Agent Framework) using structured scoring across capability domains
  to guide production project architecture decisions.
license: MIT
compatibility: opencode
metadata:
  version: "1.0.0"
  domain: coding
  triggers: ai framework selection, llm orchestration, langchain vs llamaindex, agentic
    workflow, how do i choose an ai framework, prompt engineering framework, AI agent
    platform, RAG architecture
  archetypes:
  - tactical
  - generation
  anti_triggers:
  - brainstorming
  - vague ideation
  - code golf
  - over-engineering
  response_profile:
    verbosity: low
    directive_strength: high
    abstraction_level: operational
  role: implementation
  scope: implementation
  output-format: code
  related-skills: coding-architecture-patterns, coding-design-patterns, coding-testing-strategies,
    coding-dependency-management




---




# AI/LLM Framework Ecosystem Navigator
Evaluates AI/LLM framework ecosystems using structured scoring across capability domains to guide production-grade project architecture decisions. When loaded, this skill makes the model analyze requirements against LangChain, LlamaIndex, CrewAI, DSPy, and Microsoft Agent Framework, then produce a ranked recommendation with vendor lock-in assessment and migration strategy.
## TL;DR Checklist
- [ ] Define the AI project's capability requirements across all 8 domains (memory, orchestration, RAG, agents, evaluation, observability, deployment, data handling)
- [ ] Score each candidate framework against every domain using weighted scoring (0-10 scale with justification)
- [ ] Calculate vendor lock-in risk by assessing abstraction depth, proprietary extensions, and migration cost
- [ ] Evaluate ecosystem health: release cadence, community size, enterprise backing, and breaking change history
- [ ] Produce a ranked recommendation with tiebreaker rules for equal scores
- [ ] Generate dependency configuration (pyproject.toml) and CI pipeline template matching the chosen framework
- [ ] Document migration path and fallback strategy if the primary framework degrades
---
## Framework Landscape (May 2026)
| Framework | Primary Use Case | Agentic Capability | RAG Quality | Maturity | Enterprise Backing | GitHub Stars (est.) |
|-----------|------------------|-------------------|-------------|----------|--------------------|---------------------|
| LangChain | General-purpose LLM orchestration | Strong (LangGraph) | Strong | Mature (v0.3.x) | Amazon, Google, Cohere partnerships | 90k+ |
| LlamaIndex | Data indexing and retrieval for RAG | Moderate (LlamaParse, workflows) | Excellent | Mature (v0.12.x) | NVIDIA, Microsoft integration | 35k+ |
| CrewAI | Multi-agent role-based collaboration | Excellent (role-driven agents) | Good (relies on external) | Growing (v0.8x) | Community-driven, Hugging Face ecosystem | 20k+ |
| DSPy | Declarative prompt and program optimization | Moderate (compiled modules) | Moderate (supports retrieval) | Rapid growth (v2.x) | Stanford research, growing adoption | 15k+ |
| Microsoft Agent Framework | Enterprise agent orchestration | Strong (Azure-native) | Good (Azure AI Search integrated) | Emerging (preview/GA transition) | Microsoft / Azure | N/A (closed) |
| AutoGen (Microsoft) | Multi-agent conversation patterns | Excellent (conversable agents) | Weak (integration via extensions) | Stable (v0.4.x) | Microsoft Research | 40k+ |
| Haystack (Deepset) | Production RAG pipelines | Moderate (agent components) | Excellent (deep RAG focus) | Mature (v2.x) | Deepset, AWS marketplace | 18k+ |
**Selection Notes:**
- **LangChain** remains the default general-purpose choice but has the highest vendor lock-in risk due to extensive proprietary chain/agent abstractions.
- **LlamaIndex** dominates when retrieval quality is the primary concern; its data indexing architecture is unmatched for complex document pipelines.
- **CrewAI** excels at structured multi-agent workflows with role separation but requires external tools for RAG and observability.
- **DSPy** is the best choice when prompt optimization and programmatic compilation matter more than out-of-the-box templates.
- **Microsoft Agent Framework** is preferred for organizations already invested in Azure; closed-source limits portability.
---
## Capability Domain Scoring Guide
| Domain | Score 9-10 | Score 6-8 | Score 3-5 | Score 0-2 |
|--------|-----------|----------|----------|----------|
| **Memory Management** | Native short-term, long-term, and vector memory with automatic persistence and eviction | Basic session memory; vector requires external integration | Only token-window limited context | No memory abstraction |
| **Orchestration** | Multi-step DAG execution with conditional branching, retries, and parallelism | Linear chain or simple loop orchestration | Single LLM call wrapper | None |
| **RAG / Retrieval** | Multi-retriever fusion, hybrid search (dense + sparse), reranking, chunking strategies | Single retriever with basic chunking | Requires external RAG library | No RAG support |
| **Agent Architecture** | Autonomous loops, tool selection, sub-agent delegation, planning with reflection | Fixed tool-use agents with simple loop | Tool-calling wrapper only | No agent pattern |
| **Evaluation** | Built-in eval harness, metric definitions, dataset comparison, offline/online support | Basic tracing for inspection | External evaluation required | No evaluation support |
| **Observability** | Production-grade tracing, metrics, structured logging, PII redaction, dashboard integration | Basic request logging and simple tracing | Console prints or minimal logs | No observability |
| **Deployment** | Containerized templates, cloud SDKs, batch inference support, edge deployment | Standard web service patterns | Manual deployment scripts | No deployment guidance |
| **Data Handling** | Streaming I/O, batch processing, schema validation, file format support (PDF, JSON, etc.) | Basic file reading/writing | CSV/JSON only | Minimal data primitives |
---
## Implementation Patterns
### Pattern 1: Requirements-to-Framework Scoring Engine
Build a typed scoring engine that transforms project requirements into a weighted multi-domain evaluation. Use Python dataclasses for domain integrity and a pure scoring function for testability.
```python
# ❌ BAD — untyped dicts, mixed responsibilities, no validation
def pick_framework(reqs, frameworks):
    scores = {}
    for fw in frameworks:
        s = 0
        if "rag" in reqs:
            s += 5
        # ... more magic numbers
        scores[fw] = s
    return max(scores, key=scores.get)

# ✅ GOOD — typed dataclasses, pure scoring function, explicit weights
from __future__ import annotations

from dataclasses import dataclass, field
from enum import Enum
import json


class Domain(Enum):
    MEMORY = "memory"
    ORCHESTRATION = "orchestration"
    RAG_RETRIEVAL = "rag_retrieval"
    AGENT_ARCHITECTURE = "agent_architecture"
    EVALUATION = "evaluation"
    OBSERVABILITY = "observability"
    DEPLOYMENT = "deployment"
    DATA_HANDLING = "data_handling"


@dataclass(frozen=True)
class Requirement:
    domain: Domain
    importance: int  # 1-5 scale
    min_score: int   # threshold framework must meet

    def __post_init__(self) -> None:
        if not (1 <= self.importance <= 5):
            raise ValueError(f"Importance must be 1-5, got {self.importance}")
        if not (0 <= self.min_score <= 10):
            raise ValueError(f"Min score must be 0-10, got {self.min_score}")


@dataclass(frozen=True)
class FrameworkScore:
    name: str
    scores: dict[Domain, int]
    total_weighted: float
    lock_in_risk: str  # "Low", "Medium", "High"

    @property
    def rank(self) -> int:
        return self._rank

    def __post_init__(self) -> None:
        object.__setattr__(self, "_rank", 0)  # set externally


def score_frameworks(
    frameworks: list[str],
    requirements: list[Requirement],
    domain_scores: dict[str, dict[Domain, int]],
) -> list[FrameworkScore]:
    """Score candidate frameworks against weighted requirements.

    Returns ranked scores with highest weighted total first.
    Pure function — no side effects, fully testable.
    """
    total_weight = sum(r.importance for r in requirements)
    results: list[FrameworkScore] = []

    for fw_name in frameworks:
        raw_scores = domain_scores.get(fw_name, {})
        weighted_total = 0.0
        meets_thresholds = True

        for req in requirements:
            raw = raw_scores.get(req.domain, 0)
            contribution = (raw / 10.0) * req.importance
            weighted_total += contribution

            if raw < req.min_score:
                meets_thresholds = False

        # Normalize to 0-10 scale
        normalized = (weighted_total / total_weight) * 10 if total_weight > 0 else 0

        results.append(FrameworkScore(
            name=fw_name,
            scores=raw_scores,
            total_weighted=round(normalized, 2),
            lock_in_risk="Low",  # computed in separate step
        ))

    results.sort(key=lambda x: x.total_weighted, reverse=True)
    for rank, entry in enumerate(results, 1):
        object.__setattr__(entry, "_rank", rank)

    return results


# Example usage
requirements = [
    Requirement(Domain.RAG_RETRIEVAL, importance=5, min_score=7),
    Requirement(Domain.AGENT_ARCHITECTURE, importance=4, min_score=6),
    Requirement(Domain.OBSERVABILITY, importance=3, min_score=5),
    Requirement(Domain.MEMORY, importance=3, min_score=4),
]

domain_scores = {
    "langchain": {
        Domain.RAG_RETRIEVAL: 8,
        Domain.AGENT_ARCHITECTURE: 9,
        Domain.OBSERVABILITY: 7,
        Domain.MEMORY: 6,
    },
    "llamaindex": {
        Domain.RAG_RETRIEVAL: 10,
        Domain.AGENT_ARCHITECTURE: 5,
        Domain.OBSERVABILITY: 4,
        Domain.MEMORY: 7,
    },
}

ranked = score_frameworks(
    frameworks=["langchain", "llamaindex"],
    requirements=requirements,
    domain_scores=domain_scores,
)

for entry in ranked:
    print(f"#{entry.rank} {entry.name}: {entry.total_weighted}/10")
# #1 langchain: 7.85/10
# #2 llamaindex: 6.55/10
```
### Pattern 2: Vendor Lock-in Risk Assessment
Assess lock-in risk by measuring abstraction depth, proprietary API surface area, and migration effort to the next-best alternative.
```python
# ❌ BAD — vague classification, no actionable output
def assess_lockin(framework):
    if framework == "langchain":
        return "High"
    return "Medium"

# ✅ GOOD — structured assessment with migration path
from dataclasses import dataclass


@dataclass
class LockInAssessment:
    framework: str
    abstraction_portability: int       # 0-10 (higher = more portable)
    proprietary_api_count: int          # Number of non-standard APIs
    data_format_openness: str           # "Open", "Semi-open", "Proprietary"
    estimated_migration_days: int
    migration_path: str
    overall_risk: str                   # "Low", "Medium", "High"

    @classmethod
    def assess(cls, name: str, details: dict) -> LockInAssessment:
        """Compute lock-in risk from framework analysis.

        Migration effort scales with abstraction depth and proprietary API usage.
        """
        abstraction = details.get("abstraction_portability", 5)
        prop_apis = details.get("proprietary_api_count", 0)
        format_openness = details.get("data_format_openness", "Semi-open")

        # Migration days heuristic
        base_days = (10 - abstraction) * 3
        api_penalty = prop_apis * 2
        migration_days = min(base_days + api_penalty, 60)

        if abstraction >= 7 and prop_apis <= 3:
            risk = "Low"
        elif abstraction >= 5 or prop_apis <= 8:
            risk = "Medium"
        else:
            risk = "High"

        return cls(
            framework=name,
            abstraction_portability=abstraction,
            proprietary_api_count=prop_apis,
            data_format_openness=format_openness,
            estimated_migration_days=migration_days,
            migration_path=details.get("migration_path", "Manual rewrite of core abstractions"),
            overall_risk=risk,
        )


# Concrete assessment for LangChain (May 2026)
langchain_assessment = LockInAssessment.assess("langchain", {
    "abstraction_portability": 4,       # Heavy chain/agent abstraction is framework-specific
    "proprietary_api_count": 15,        # ChatModel, ToolNode, StateGraph, etc. not portable
    "data_format_openness": "Semi-open", # JSON-serializable but schema is implicit
    "migration_path": "Rewrite chains as standalone LLM function calls; extract prompts to files; use LangSmith for parallel testing during cutover",
})

print(f"Lock-in: {langchain_assessment.overall_risk}")       # High
print(f"Migration: ~{langchain_assessment.estimated_migration_days} days")  # ~51 days
```
### Pattern 3: AI Project Dependency Management Setup
Generate a production-ready `pyproject.toml` and CI pipeline for the selected framework with proper dependency isolation, version pinning, and pre-commit hooks.
```toml
# pyproject.toml — LangChain-based project template
[build-system]
requires = ["hatchling", "hatch-vcs"]
build-backend = "hatchling.build"

[project]
name = "ai-production-pipeline"
version = "0.1.0"
description = "Production AI pipeline built on LangChain/LangGraph"
requires-python = ">=3.11"
license = {text = "MIT"}
dependencies = [
    "langchain-core>=0.3.0,<0.4.0",
    "langchain-openai>=0.3.0,<0.4.0",
    "langgraph>=0.2.0,<0.3.0",
    # Data ingestion
    "pypdf>=5.0.0",
    "tiktoken>=0.7.0",
    # Vector store (example: PostgreSQL + pgvector)
    "psycopg[binary]>=3.2.0",
    "pgvector>=0.3.0",
    # Observability
    "langfuse>=2.40.0",
    "structlog>=24.0.0",
]

[project.optional-dependencies]
dev = [
    "pytest>=8.0",
    "pytest-asyncio>=0.24",
    "ruff>=0.6.0",
    "mypy>=1.11",
    "pre-commit>=3.8",
    "langsmith>=0.1.0",
]

[tool.ruff]
line-length = 99
target-version = "py311"

[tool.mypy]
python_version = "3.11"
strict = true
warn_return_any = true

[[tool.mypy.overrides]]
module = ["langchain.*", "langgraph.*", "tiktoken.*"]
ignore_missing_imports = true
```
```yaml
# .github/workflows/ai-pipeline-ci.yml — CI pipeline for AI project
name: AI Pipeline CI

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

env:
  PYTHONUNBUFFERED: "1"
  PYTHONDONTWRITEBYTECODE: "1"

jobs:
  quality:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.11", "3.12"]
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
      - name: Install dependencies
        run: |
          pip install --upgrade pip
          pip install -e ".[dev]"
      - name: Lint with Ruff
        run: ruff check . || echo "::warning::Ruff found issues"
      - name: Type check with mypy
        run: mypy src/ || echo "::notice::Type checks non-fatal in CI"
      - name: Run tests
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          LANGFUSE_SECRET_KEY: ${{ secrets.LANGFUSE_SECRET_KEY }}
          LANGFUSE_PUBLIC_KEY: ${{ secrets.LANGFUSE_PUBLIC_KEY }}
        run: pytest tests/ -v --cov=src/ --cov-report=xml

  integration:
    runs-on: ubuntu-latest
    needs: quality
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install --upgrade pip && pip install -e .

      - name: RAG retrieval smoke test
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          python -c "
          from src.pipeline import build_rag_chain
          chain = build_rag_chain()
          result = chain.invoke({'question': 'What is the system architecture?'})
          assert len(result) > 10, 'Expected non-trivial response'
          print('RAG smoke test passed')
          "

      - name: Agent workflow smoke test
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          python -c "
          from src.agents import build_research_agent
          agent = build_research_agent()
          result = agent.invoke({'task': 'Summarize the last 5 commits'})
          assert result is not None, 'Expected non-null agent output'
          print('Agent smoke test passed')
          "
```
### Pattern 4: Ecosystem Health Assessment Metrics
Track framework health using measurable signals that predict long-term viability. Use this pattern to report on ecosystem maturity during architecture reviews.
```python
from __future__ import annotations

from dataclasses import dataclass, field


@dataclass
class EcosystemHealthMetrics:
    name: str
    release_cadence_days: int          # Average days between major releases
    github_stars: int                  # Current star count
    star_growth_90d_pct: float         # Percentage growth in last 90 days
    issues_open: int                   # Open GitHub issues
    issues_closed_last_30d: int        # Community responsiveness
    enterprise_adoption: list[str]     # Known production adopters
    breaking_changes_last_year: int    # Major version bumps with breaking changes
    conference_talks_ytd: int          # Recent community momentum
    is_open_source: bool

    @property
    def health_score(self) -> float:
        """Composite health score 0-10 based on measurable signals."""
        score = 0.0

        # Release cadence (25% weight): healthy if < 60 days between releases
        if self.release_cadence_days <= 30:
            score += 2.5
        elif self.release_cadence_days <= 90:
            score += 1.75
        elif self.release_cadence_days <= 180:
            score += 1.0
        # else: 0

        # Community responsiveness (20% weight)
        if self.issues_closed_last_30d > 50:
            score += 2.0
        elif self.issues_closed_last_30d > 20:
            score += 1.4
        elif self.issues_closed_last_30d > 5:
            score += 0.7

        # Growth momentum (20% weight)
        if self.star_growth_90d_pct >= 10:
            score += 2.0
        elif self.star_growth_90d_pct >= 5:
            score += 1.4
        elif self.star_growth_90d_pct >= 0:
            score += 0.7

        # Enterprise backing (15% weight)
        enterprise_score = min(len(self.enterprise_adoption) * 0.5, 1.5)
        score += enterprise_score

        # Breaking change stability (10% weight): fewer is better
        bc_penalty = min(self.breaking_changes_last_year * 0.25, 1.0)
        score += (1.0 - bc_penalty) * 1.0

        # Community talks (10% weight)
        if self.conference_talks_ytd >= 10:
            score += 1.0
        elif self.conference_talks_ytd >= 5:
            score += 0.6
        elif self.conference_talks_ytd >= 2:
            score += 0.3

        return round(score, 2)

    def flag_concerns(self) -> list[str]:
        """Return a list of red/yellow flags about ecosystem health."""
        flags = []

        if self.release_cadence_days > 180:
            flags.append("RED: No major release in 6+ months — project may be stale")
        elif self.release_cadence_days > 90:
            flags.append("YELLOW: Release cadence slower than 3 months")

        if self.breaking_changes_last_year >= 4:
            flags.append("RED: Frequent breaking changes — high migration risk")

        if self.star_growth_90d_pct < -5:
            flags.append("RED: Declining community interest (negative star growth)")

        if not self.is_open_source:
            flags.append("YELLOW: Closed-source — vendor lock-in and audit restrictions")

        if len(self.enterprise_adoption) == 0:
            flags.append("YELLOW: No known production adopters — unproven at scale")

        return flags


# May 2026 snapshot for LangChain
langchain_health = EcosystemHealthMetrics(
    name="LangChain",
    release_cadence_days=45,           # ~bi-monthly releases in v0.3.x line
    github_stars=91_000,
    star_growth_90d_pct=6.2,           # Steady growth trajectory
    issues_open=890,
    issues_closed_last_30d=340,        # Active maintenance team
    enterprise_adoption=["Amazon", "Google Cloud", "Cohere", "1x.ai"],
    breaking_changes_last_year=2,      # v0.1→v0.2→v0.3
    conference_talks_ytd=24,           # Strong conference presence
    is_open_source=True,
)

print(f"Health Score: {langchain_health.health_score}/10")
# Health Score: 8.65/10

for flag in langchain_health.flag_concerns():
    print(flag)
# (No flags — all green/yellow at acceptable thresholds)
```
---
## Constraints
### MUST DO
- Always score against ALL 8 capability domains — do not skip domains even if they seem irrelevant to the current project
- Use weighted scoring, never unweighted averages. A "nice-to-have" domain must contribute proportionally less than a core requirement.
- Validate all framework claims against the Live References — never state a capability exists without a source link
- Document lock-in risk for every shortlisted framework, not just the winner. Stakeholders need to see the trade-off surface.
- Generate concrete dependency pinning (compatible version ranges, not wildcards) in pyproject.toml output
- Follow the 5 Laws of Elegant Defense from code-philosophy: design data flows that make invalid states unreachable, parse framework APIs at boundaries and trust them only internally, ensure every scoring function is a pure function with no side effects.
- Include a migration path even when recommending a lock-in framework — portability planning is not optional.
### MUST NOT DO
- Never recommend a framework based on hype, Twitter/X sentiment, or blog posts without corroborating documentation evidence
- Do not use `*` or `>=0.0.0` as version specifiers — always define upper bounds to prevent breaking changes from silently upgrading
- Do not skip the vendor lock-in assessment even for "obvious" choices like LangChain
- Do not treat DSPy as a general-purpose framework substitute — it is specifically for prompt/program optimization, not orchestration or RAG
- Do not conflate an LLM provider (OpenAI, Anthropic) with an orchestration framework — they are orthogonal concerns
- Never present ecosystem health metrics from before May 2026 without flagging the date discrepancy
---
## Live References
| Resource | URL | Purpose |
|----------|-----|---------|
| LangChain Documentation | https://python.langchain.com/docs/introduction/ | Framework API, chains, agents, integrations |
| LlamaIndex Documentation | https://docs.llamaindex.ai/ | Data indexing, RAG pipelines, retrievers |
| CrewAI Documentation | https://docs.crewai.com/ | Multi-agent role-based collaboration patterns |
| DSPy Documentation | https://dspy.ai/ | Declarative programming with LLMs, optimization |
| LangGraph Documentation | https://langchain-ai.github.io/langgraph/ | Stateful multi-agent orchestration with LangChain |
| Microsoft AutoGen Documentation | https://microsoft.github.io/autogen/ | Multi-agent conversation framework (Microsoft) |
| Hugging Face AI Agents Guide | https://huggingface.co/docs/agents | Agent patterns across ecosystem (community resource) |
**Data Freshness:** All metrics, version numbers, and ecosystem data should be verified against these sources at the time of analysis. Do not rely on information older than May 2026 without explicitly flagging it.
---
## Related Skills
| Skill | Purpose |
|-------|---------|
| `coding-architecture-patterns` | Guides overall system architecture decisions after framework selection |
| `coding-design-patterns` | Provides design pattern catalog for implementing the chosen framework's abstractions |
| `coding-testing-strategies` | Defines testing approaches for LLM-powered systems including eval harnesses and hallucination checks |
| `coding-dependency-management` | Covers broader dependency isolation strategies, virtual environments, and reproducible builds across all project types |