Recsys Pipeline Architect

---
name: recsys-pipeline-architect
description: "Use when design composable recommendation, ranking, and feed pipelines using the six-stage Source→Hydrator→Filter→Scorer→Selector→SideEffect framework popularized by xAI's open-sourced For You algorithm. Use this skill whenever the user is buildin... Triggers on \"recsys-pipeline-architect\", \"recsys pipeline architect\", \"architect\"."
metadata:
  origin: community
---

# recsys-pipeline-architect

A spec-and-scaffold skill for building composable recommendation, ranking, and feed pipelines. It encodes the **six-stage pattern** — Source → Hydrator → Filter → Scorer → Selector → SideEffect — popularized by xAI's open-sourced [For You algorithm](https://github.com/xai-org/x-algorithm) (Apache 2.0). This skill is an independent reimplementation of the pattern (MIT) — no code copied from the original.

Upstream: <https://github.com/mturac/recsys-pipeline-architect>

## When to Use

- User wants to build any system that picks "the top K items for a user/context"
- User asks "how should I rank X" or describes a feed/personalization problem
- User has a scoring function and needs the pipeline plumbing around it
- User wants to migrate from a single relevance score to multi-action prediction with tunable weights
- User is wrapping an LLM/ML scorer and needs filters, hydrators, side-effects, and a runnable scaffold in their stack (TypeScript / Go / Python)
- Triggers: "recommendation system", "feed algorithm", "ranking pipeline", "for you feed", "candidate pipeline", "content recommender", "pipeline architecture for recsys", "RAG retrieval reranker"

## When NOT to Use

- Model architecture work (transformer design, two-tower retrieval, embedding training) — this skill is plumbing *around* the model, not the model itself
- Pure ML training pipelines — the scoring function is the user's responsibility
- Operating a deployed pipeline (monitoring, autoscaling) — out of scope

## The six-stage framework

| # | Stage | Job | Parallel? |
|---|---|---|---|
| 1 | **Source** | Fetch candidates from one or more origins | Yes — multiple sources run in parallel |
| 2 | **Hydrator** | Enrich each candidate with metadata needed for filtering and scoring | Yes — independent hydrators run in parallel |
| 3 | **Filter** | Drop candidates that should never be shown (blocked, expired, duplicate, ineligible) | Sequential — each filter sees fewer items |
| 4 | **Scorer** | Assign each surviving candidate one or more scores | Sequential — later scorers see earlier scores |
| 5 | **Selector** | Sort by final score, return top K | Single op |
| 6 | **SideEffect** | Cache served IDs, log impressions, emit events, update counters | Async — must never block the response |

### Why this exact order

- Sources before hydration: know what candidates exist before paying to enrich them
- Hydration before filtering: many filters need metadata the source did not provide
- Filtering before scoring: scoring is the expensive stage; drop the ineligible first
- Scorer chain (not single scorer): real systems compose ML scoring + diversity reranking + business rules
- Selector after scoring: keeps scoring deterministic and cacheable
- SideEffects last and async: side effects must never block the user response

## Workflow when invoked

Walk the user through these eight steps:

1. **Clarify the use case** (one round, three questions): items being ranked? input context? language/runtime?
2. **Identify the candidate sources**: usually in-network (followed/owned/subscribed) + out-of-network (ML retrieval / trending / similar-to-liked)
3. **List required hydrations**: for each filter and scorer, what data does it need that the source did not provide?
4. **List the filters**: duplicate, self, age, block/mute, previously-served, eligibility. Order matters — cheap before expensive.
5. **Design the scorer chain**: primary (ML) → combiner (multi-action with weights) → diversity → business rules
6. **Selector**: sort descending by final score, take top K (or stratified mix for in-network/out-of-network)
7. **SideEffects**: cache served IDs, emit impression events, update counters, log analytics — all fire-and-forget
8. **Generate the scaffold** in the user's stack

## Key trade-offs to surface (don't default silently)

### 1. Single score vs multi-action prediction

## Exemplo

```text
Feed "para você": Source (seguidos + trending) → Hydrator (meta) → Filter (dup/bloq)
→ Scorer (ML + diversidade + regra negócio) → Selector top 20 → SideEffect async (log, cache)
Trade-off explícito: single score (simples) vs multi-action (afinado, mais custo)
```