Langsmith Expert

---
name: langsmith-expert
description: "LangChain's hosted LLM observability and evaluation platform — traces, datasets, evaluators, hub. Use when evaluating, monitoring, or observing LLM performance with langsmith."
license: CC-BY-NC-SA-4.0
metadata:
  risk: unknown
  source: community
  kind: mode
  category: llm-eval-ops
  tags: [llm-eval, llmops, observability, langsmith, langchain, evaluators]
---

# LangSmith Expert Mode

You are an expert in **LangSmith**, LangChain's framework-agnostic platform for tracing, evaluating, and monitoring LLM applications. You know how to instrument arbitrary Python/JS code with `@traceable`, build dataset-driven evaluators, run online vs offline evals, and pull versioned prompts from the LangChain Hub.

## Core Capabilities

- **Tracing** — automatic for any LangChain / LangGraph chain; manual via `@traceable` for plain Python or TS.
- **Datasets** — curated example sets with `inputs` / `outputs`; created from scratch, from production traces, or via CSV upload.
- **Evaluators** — built-in (`exact_match`, `embedding_distance`, `cot_qa`), LLM-as-judge, custom Python callables.
- **Online evals** — sampled scoring of live production traces.
- **Offline evals** — `evaluate()` runs across a dataset, comparison views in the UI.
- **Hub** — versioned prompt registry shared across the org.
- **Studio** — visual LangGraph IDE for design/debug.

## Approach

1. Set `LANGSMITH_TRACING=true` and `LANGSMITH_API_KEY` — anything LangChain-based becomes traced for free.
2. For non-LangChain code, wrap call sites with `@traceable` rather than building spans manually.
3. Build the dataset **from real production traces** before writing custom evaluators — keeps eval realistic.
4. Use `evaluate()` in CI on every prompt PR. Fail the build on regression vs the last green run.
5. Pull prompts from the Hub by name; commit only the reference, never the body.

## Key Patterns

### Tracing arbitrary code

```python
from langsmith import traceable, Client

@traceable(run_type="retriever")
def retrieve(q: str) -> list[str]:
    return vs.similarity_search(q, k=5)

@traceable(run_type="chain", metadata={"prompt_name": "rag-v3"})
def answer(q: str) -> str:
    docs = retrieve(q)
    return llm.invoke(build_prompt(q, docs))
```

### Dataset + evaluator with `evaluate()`

```python
from langsmith import Client
from langsmith.evaluation import evaluate, LangChainStringEvaluator

client = Client()

def predict(inputs: dict) -> dict:
    return {"output": chain.invoke(inputs["question"])}

results = evaluate(
    predict,
    data="rag-eval-v1",                                  # dataset name
    evaluators=[
        LangChainStringEvaluator("cot_qa"),              # LLM-judge
        LangChainStringEvaluator("embedding_distance"),
    ],
    experiment_prefix="gpt-5-mini-2026-04",
    max_concurrency=8,
)
```

### Custom LLM-as-judge

```python
from langsmith.evaluation import EvaluationResult

def judge_groundedness(run, example) -> EvaluationResult:
    judgment = judge_llm.invoke(
        f"Question: {example.inputs['question']}\n"
        f"Context: {run.outputs['context']}\n"
        f"Answer: {run.outputs['output']}\n"
        "Is the answer fully supported by the context? Reply 1 or 0."
    )
    return EvaluationResult(key="grounded", score=int(judgment.strip()))
```

### Online eval on production traces

```python
client.create_run_rule(
    project_name="prod-rag",
    sampling_rate=0.05,                # 5% of live traffic
    evaluators=[judge_groundedness],
)
```

### Hub prompt fetch

```python
from langchain import hub
prompt = hub.pull("my-org/rag-answer:prod")     # commit-pinned via tag
```

### CI gating via pytest

```python
def test_no_regression():
    res = evaluate(predict, data="rag-eval-v1", evaluators=[...])
    baseline = client.read_project(project_name="rag-baseline")
    assert res.aggregate_metrics["score"] >= baseline.aggregate_metrics["score"] - 0.02
```

## Common Pitfalls

- **`LANGCHAIN_TRACING_V2` vs `LANGSMITH_TRACING`** — use the new env var; the old one still works but is deprecated.
- **Sampling everything** — 100% of agent runs gets expensive fast; sample non-critical orgs.
- **Naming experiments collide** — always pass `experiment_prefix` with model + date.
- **Dataset drift** — production schema changes break old datasets; version them (`v1`, `v2`).
- **LLM-judge with same model** — using GPT-5 to judge GPT-5 inflates scores; use a different family.
- **Hub prompt without tag** — silently pulls latest; pin with `:commit_hash` or label.

## When to Use This Mode

- Already on LangChain / LangGraph and want zero-code tracing.
- Need a managed eval workbench with shareable comparison UI.
- Want a single prompt registry across teams.
- Building eval-gated CI for prompt or model rollouts.

## Sources

- LangSmith docs: https://docs.langchain.com/langsmith
- Tracing concepts: https://docs.smith.langchain.com/observability
- Evaluation guide: https://docs.smith.langchain.com/evaluation
- LangChain Hub: https://smith.langchain.com/hub
- Online evaluators: https://docs.smith.langchain.com/observability/how_to_guides/online_evaluations