Liu 2023 Agentbench

---
license: Apache-2.0
name: liu-2023-agentbench
description: Comprehensive benchmark suite for evaluating LLM agents across diverse interactive environments
metadata:
  category: Research & Academic
  tags:
    - benchmarks
    - llm-agents
    - evaluation
    - agent-testing
    - capabilities
  io-contract:
    kind: deliverable
    produces:
      - kind: critique
        description: >-
          Diagnostic analysis of LLM agent failures across interactive environments, identifying failure modes (rubber
          stamp validation, planning amnesia, metacognitive blindness, code training rigidity, executive function
          collapse) with detection rules and root cause explanations
        format: markdown
      - kind: refactor-plan
        description: >-
          Structured remediation strategies for agent failures, including prompt engineering adjustments, validation
          layer additions, state tracking mechanisms, and model selection guidance based on task characteristics
        format: markdown
      - kind: design-doc
        description: >-
          Decision trees and heuristics for model selection (code-trained vs frontier models), environment-specific
          debugging strategies, and verbosity trade-offs for prompt engineering in multi-turn agent tasks
        format: markdown
allowed-tools: Read,Write,Edit,Glob,Grep
---

# SKILL: AgentBench - Evaluating and Debugging LLM Agents

## DECISION POINTS

### Model Selection for Agent Tasks
```
IF task requires procedural execution (web shopping, DB queries, API calls):
  ├─ IF <15 steps AND template-based outputs
  │  └─ Use code-trained models (CodeLlama) - 3x better at format compliance
  └─ IF >15 steps OR requires plan revision
     └─ Use general models (GPT-4) - maintains coherence across turns

IF task requires strategic reasoning (games, puzzles, negotiations):
  ├─ IF model must generate novel hypotheses
  │  └─ Avoid code-trained models - they over-optimize for deterministic paths
  └─ IF model must revise plans based on feedback
     └─ Use frontier models only - others lose state after round 5

IF failure budget <10%:
  ├─ Invalid format/action rate matters more than success rate
  └─ API models (GPT-4: 6% invalid) vs Open source (13.6% invalid)

IF task involves >20 interaction rounds:
  └─ Only GPT-4 tier maintains plan-state binding - others enter loops by round 10
```

### Failure Diagnosis Decision Tree
```
IF agent produces malformed outputs despite clear instructions:
  ├─ Check Rouge-L similarity in last 3 outputs
  │  ├─ High (>0.8): Loop detection failure → Add state tracking
  │  └─ Low (<0.5): Executive function gap → Add format validation
  
IF agent violates environment rules (impossible actions):
  ├─ Count rule violations per environment type
  │  ├─ Code environments: Missing API constraints → Add action space docs
  │  ├─ Game environments: Invalid moves → Add rule reminders each turn  
  │  └─ Web environments: Element targeting → Add DOM structure context

IF agent exceeds task limits without completion:
  ├─ Analyze final 5 rounds for repetition patterns
  │  ├─ Repeating same action: Add "what have I tried?" prompt
  │  ├─ Repeating same reasoning: Add progress checkpoints
  │  └─ Random actions: Escalate to human or abort task
```

### Prompt Engineering Verbosity Trade-off
```
IF environment has complex rule set (>10 constraints):
  ├─ High verbosity: Include full rules every turn
  │  └─ Trade-off: Context bloat but lower invalid action rate
  └─ Low verbosity: Rules in system message only
     └─ Trade-off: Cleaner prompt but higher rule violation risk

IF task requires >10 sequential steps:
  ├─ Include explicit progress tracking: "Step X of Y completed"
  └─ Add loop detection: "Have I done this exact action before?"
```

## FAILURE MODES

### 1. Rubber Stamp Validation ("Format Looks Right")
**Detection Rule**: If model produces structurally valid JSON but semantically invalid actions (e.g., `{"action": "click", "element": "nonexistent_button"}`)

**Symptoms**: 
- Valid syntax, impossible actions
- Model explains action correctly but targets wrong elements
- High invalid action rate (>15%) despite format compliance

**Diagnosis**: Dissociation between linguistic understanding and environmental grounding

**Fix**: Add action pre-validation layer that checks element existence before execution

### 2. Planning Amnesia ("I Had a Plan, What Was It?")
**Detection Rule**: If model generates good initial plan but actions don't follow plan by round 5+ OR model repeats plan generation mid-task

**Symptoms**:
- Rouge-L <0.3 between plan and actual action sequence
- Model re-explains same strategy when asked mid-task
- Good reasoning in `<thought>` tags, contradictory actions

**Diagnosis**: Plan-state binding failure in working memory

**Fix**: Include plan summary in every prompt; add "current plan step" tracking

### 3. Metacognitive Blindness ("I'm Not Stuck")
**Detection Rule**: If Rouge-L ≥0.8 in final 3 rounds AND task incomplete

**Symptoms**:
- Repeating identical actions/reasoning
- No recognition of loop when explicitly asked
- Confidence remains high despite lack of progress

**Diagnosis**: No internal representation of "attempted strategies" or progress monitoring

**Fix**: External loop detection with mandatory strategy pivot after 3 identical rounds

### 4. Code Training Rigidity ("There Must Be One Right Sequence")
**Detection Rule**: If code-trained model fails strategic tasks with success rate <50% of general model performance

**Symptoms**:
- Over-optimization for deterministic execution
- Fails to explore alternative paths when first approach fails
- Strong performance on procedural tasks, weak on exploratory tasks

**Diagnosis**: Code training bias toward single optimal path

**Fix**: Use general models for strategic tasks; add explicit exploration prompts for code-trained models

### 5. Executive Function Collapse ("I Know What To Do But Can't Do It")
**Detection Rule**: If model correctly explains requirements when asked but immediately violates them in output

**Symptoms**:
- Perfect task understanding in conversation mode
- Immediate format violations in action mode
- Can debug own errors but repeats them

**Diagnosis**: Linguistic competence vs. procedural compliance dissociation

**Fix**: Constrained decoding, output validation layer, or format templates with variable substitution

## WORKED EXAMPLES

### Example 1: Web Shopping Failure Analysis
**Scenario**: Agent must purchase specific laptop from e-commerce site. CodeLlama-34b vs GPT-4 comparison.

**Turn 1-3**: Both models navigate homepage correctly, use search function
- CodeLlama: Clean JSON format, efficient navigation
- GPT-4: Slightly verbose but same actions

**Turn 4-6**: Product comparison required
- CodeLlama: Attempts to add first matching product to cart immediately
- GPT-4: Opens multiple products, compares specs against requirements

**Turn 7-12**: CodeLlama hits constraint (wrong specs)
- CodeLlama: Loops on same product, doesn't revise search strategy
- GPT-4: Recognizes mismatch, refines search with different keywords

**Turn 13+**: Task completion
- CodeLlama: Task limit exceeded (TLE) - Rouge-L 0.87 in final rounds
- GPT-4: Completes purchase successfully

**Key Insight**: Procedural task (web navigation) initially favors CodeLlama, but strategic pivot requirement (spec mismatch → search refinement) causes failure. The task grounding shifted from procedural to strategic mid-execution.

### Example 2: Debug TLE Loop with Recovery
**Scenario**: Agent stuck in navigation loop on unfamiliar website

**Detection Phase**:
```
Round 15: {"action": "click", "element": "nav-menu"}
Round 16: {"action": "click", "element": "nav-menu"}  
Round 17: {"action": "click", "element": "nav-menu"}
Rouge-L: 0.92 - LOOP DETECTED
```

**Recovery Steps**:
1. **Interrupt**: Insert "You've clicked nav-menu 3 times. What are you trying to accomplish?"
2. **State Reset**: Agent responds: "Looking for product categories"  
3. **Alternative Prompt**: "List 3 different ways to find product categories besides nav-menu"
4. **Strategy Pivot**: Agent suggests search bar, footer links, homepage scan
5. **Execution**: Agent successfully uses search bar, completes task

**Key Insight**: Loop detection must trigger strategy enumeration, not just "try harder." The fix is metacognitive scaffolding, not better reasoning.

## Reference Files

- `diagrams/01_flowchart_agent_failure_diagnosis_&_reme.md` — Mermaid flowchart mapping agent symptoms (malformed output, rule violations, repetition) to root causes and remediation strategies. **Read when** diagnosing why an agent failed and what to fix.

- `diagrams/02_quadrantChart_model_selection_matrix-_task_t.md` — Quadrant chart positioning CodeLlama, Llama2, GPT-4 by task type (procedural vs strategic) and planning horizon. **Read when** choosing which model to deploy for a specific agent task.

- `diagrams/03_stateDiagram-v2_agent_state_machine-_planning-.md` — State machine showing planning→action→validation→observation loop with failure points (format violations, invalid actions, loop detection). **Read when** understanding where agents get stuck in multi-turn interactions.

- `references/code-training-double-edged-sword.md` — Explains CodeLlama's paradox: excels at web/database tasks (52% vs 5.6%) but fails at games/OS tasks (8.4% vs 21.3%). **Read when** deciding between code-trained and general models.

- `references/decomposition-principles-from-multi-environment-evaluation.md` — Teaches how task decomposability depends on task structure (code/game/web), not just complexity. **Read when** breaking down a complex agent task into subtasks.

- `references/environment-grounding-types-and-skill-design.md` — Defines three grounding types (Code, Game, Web) and shows performance varies by type independent of general intelligence. **Read when** designing agent systems for specific domain types.

- `references/failure-taxonomy-interactive-systems.md` — Systematic categorization of agent failures (rubber stamp validation, planning amnesia, metacognitive blindness, code rigidity, executive collapse) with detection rules. **Read when** classifying an agent failure to identify root cause.

- `references/instruction-following-as-executive-function.md` — Shows why GPT-4 fails to follow explicit format instructions despite high reasoning ability; frames instruction-following as executive function. **Read when** debugging format compliance issues in agent outputs.

- `references/task-limit-exceeded-and-loop-detection.md` — Analyzes the dominant failure mode (24.9% API, 36.9% open-source): agents exhaust rounds via repetition. Includes detection signatures. **Read when** agent gets stuck in loops or exceeds interaction limits.

- `references/the-gap-between-planning-and-execution.md` — Documents planning-execution disconnect: agents articulate correct plans but execute violating actions. **Read when** agent's reasoning contradicts its actions.

## QUALITY GATES

- [ ] Invalid format rate <8% (measure malformed JSON/action structures)
- [ ] Invalid action rate <12% (measure environment rule violations)  
- [ ] Task limit exceeded rate <30% (measure incomplete tasks at round limit)
- [ ] Loop detection triggers activated <15% of runs (Rouge-L ≥0.8 check)
- [ ] Multi-turn coherence maintained: Rouge-L between plan and actions ≥0.4 after round 10
- [ ] Model-task grounding alignment: Code-trained models only on procedural tasks
- [ ] Failure mode distribution logged: Track format/action/TLE breakdown for intervention targeting
- [ ] Human escalation triggers defined: Automatic handoff rules for detected failure patterns
- [ ] Progress validation: State advancement measurable every 5 rounds
- [ ] Environment-specific constraints validated: Action space limitations enforced pre-execution

## NOT-FOR BOUNDARIES

**Do NOT use this skill for**:
- **Single-turn question answering** → Use `question-answering-strategies.md` instead
- **Static text generation** → Use `content-generation-patterns.md` instead  
- **Pure reasoning without environmental interaction** → Use `logical-reasoning-frameworks.md` instead
- **Real-time reactive systems** → Use `streaming-response-handling.md` instead
- **Simple API calls without multi-step planning** → Use `api-integration-patterns.md` instead

**Delegate to other skills when**:
- Task requires domain-specific knowledge → Use `domain-expertise-routing.md`
- Evaluation needs custom metrics → Use `benchmark-design-principles.md`
- Agent architecture design needed → Use `agent-orchestration-patterns.md`
- Production deployment concerns → Use `llm-system-monitoring.md`

**This skill specifically handles**: Multi-round interactive decision-making where environmental constraints, plan revision, and failure recovery are primary concerns.