Systematically train 8B web agents to solve complex information-seeking tasks through model-based exploration and long-to-short query evolution, achieving state-of-the-art performance via SFT+RL training pipeline with progressive context expansion to 128K tokens and 100 tool turns.
Installs into .claude/skills of the current project.
Are you the author of Webexplorer Long Horizon Web Agent Training?
Add the live security badge to your README. It updates with every re-scan.
[](https://www.skillsdirectory.com/skills/adu2021-webexplorer-long-horizon-web-agent-training)
---
name: webexplorer-long-horizon-web-agent-training
title: "WebExplorer: Explore and Evolve for Training Long-Horizon Web Agents"
version: 0.0.2
engine: skillxiv-v0.0.2-claude-opus-4.6
license: MIT
url: "https://arxiv.org/abs/2509.06501"
keywords: [web agents, long-horizon reasoning, data synthesis, query evolution, reinforcement learning, information seeking, multi-step reasoning]
description: "Systematically train 8B web agents to solve complex information-seeking tasks through model-based exploration and long-to-short query evolution, achieving state-of-the-art performance via SFT+RL training pipeline with progressive context expansion to 128K tokens and 100 tool turns."
---
## Train Long-Horizon Web Agents with Challenging Synthetic Data
Building effective web agents requires abundant high-quality training data that reflects real-world complexity. Most agents struggle with information-seeking tasks because they rarely encounter sufficiently challenging examples during training. WebExplorer addresses this fundamental bottleneck by systematically generating difficult query-answer pairs through model-based exploration and iterative query transformation, then training agents with a supervised-fine-tuning-plus-reinforcement-learning pipeline that progressively extends reasoning horizons.
### Problem Context
Open-source web agents historically suffered from two limitations: limited information-seeking ability on complex tasks and lack of transparent, reproducible implementations. The root cause—scarcity of challenging, realistic training data—created a data quality bottleneck. Existing datasets either contained straightforward queries solvable in 5-7 tool calls or relied on rigid knowledge graph construction with complex engineering overhead. WebExplorer-8B demonstrates that an 8B parameter model can outperform larger agents (WebSailor-72B, WebShaper-72B) when trained on properly synthesized, progressively difficult data, achieving 15.7% on BrowseComp-en (versus WebSailor-72B's 12.0%) while supporting up to 100 sequential tool calls and 128K token contexts.
### Core Concept
WebExplorer operates as a two-stage data synthesis pipeline followed by a two-phase training methodology:
**Stage 1: Model-Based Exploration** — Rather than engineering explicit knowledge graphs, leverage an existing capable model (Claude, GPT-4) to autonomously explore the web starting from Wikipedia seed entities. This natural exploration builds comprehensive information landscapes without complex graph heuristics.
**Stage 2: Long-to-Short Query Evolution** — Transform simple questions into hard ones by systematically removing salient information (dates, locations, proper names) and introducing strategic obfuscation across multiple iterations. This contrasts with prior short-to-long approaches that inject new facts; instead, you deliberately hide search entry points to force multi-step exploratory reasoning.
**Training Phase 1: Supervised Fine-Tuning** — Collect high-quality trajectories (13K samples via rejection sampling) using the ReAct framework (thoughts, tool calls, observations) to establish foundational search/browse capabilities.
**Training Phase 2: Reinforcement Learning** — Apply GRPO algorithm on the QA pairs (without requiring trajectory demonstrations), with progressive context expansion (64K→96K→128K tokens, 50→75→100 tool turns) to naturally emerge sophisticated long-horizon behaviors.
### Architecture Overview
The framework consists of interconnected components working in sequence:
- **Exploration Engine**: Queries web with search and browse tools, visits Wikipedia entities and related pages, collects diverse contexts about seed entities (names, dates, organizations, locations, achievements)
- **Evolution Transformer**: Takes initial QA pairs and iteratively removes obvious answer entry points across 5 evolution steps, tracking which queries reach desired difficulty (9+ tool turns needed)
- **SFT Trainer**: Fine-tunes base model (Qwen3-8B) on 13K curated trajectories with ReAct formatting, batch size 32, learning rate 1e-5, 4 epochs
- **RL Optimizer**: Applies GRPO with composite reward function (0.2 × format_reward + accuracy_reward), gradually expands context windows and turn limits, trains on 12K samples with group size 8
- **Evaluation Suite**: Tests on BrowseComp-en/zh, GAIA, WebWalkerQA, FRAMES, XBench-DS, HLE using Avg@4 metric with LLM-as-Judge
### Implementation
#### 1. Data Synthesis Pipeline
Set up the initial exploration phase by seeding Wikipedia entities and generating baseline QA pairs.
```python
# exploration_engine.py
import anthropic
from typing import List, Dict
import json
def explore_entity(entity_name: str, client) -> Dict[str, str]:
"""
Use model-based exploration to gather information about a Wikipedia entity.
Returns contexts suitable for creating information-seeking questions.
"""
system_prompt = """You are a thorough web researcher. Given an entity name,
systematically search for and gather comprehensive information about it,
including biographical details, achievements, dates, locations, and related
entities. Return structured information."""
search_prompt = f"""Search for and compile comprehensive information about '{entity_name}'.
Include: birth/founding date, key locations, major achievements, related organizations/people.
Return as a JSON object with fields: dates, locations, names, achievements, organizations."""
response = client.messages.create(
model="claude-opus-4-1-20250805",
max_tokens=2000,
messages=[
{"role": "user", "content": search_prompt}
],
system=system_prompt
)
try:
info = json.loads(response.content[0].text)
except:
info = {"raw": response.content[0].text}
return info
def generate_initial_qa(entity_info: Dict, client) -> tuple[str, str]:
"""
Generate initial question-answer pair from explored entity information.
Creates straightforward questions that require basic web search.
"""
qa_prompt = f"""Based on this information about a person/entity:
{json.dumps(entity_info, indent=2)}
Create ONE factual question that requires web search to answer completely.
The question should be answerable but require at least 5-7 tool calls (search, browse).
Return as JSON: {{"question": "...", "answer": "..."}}"""
response = client.messages.create(
model="claude-opus-4-1-20250805",
max_tokens=1500,
messages=[{"role": "user", "content": qa_prompt}]
)
qa_data = json.loads(response.content[0].text)
return qa_data["question"], qa_data["answer"]
```
The exploration phase produces approximately 40K initial QA pairs. These form the foundation for the evolution process, which transforms simple questions into genuinely difficult ones.
#### 2. Long-to-Short Query Evolution
Implement the iterative query evolution that makes questions harder by removing obvious search entry points.
```python
# query_evolution.py
def evolve_query(question: str, answer: str, iteration: int, client) -> tuple[str, str]:
"""
Iteratively remove salient information from the question across 5 evolution steps.
Each iteration deliberately obscures search entry points (dates, names, locations).
Iteration 0→1: Remove full names, use descriptions instead
Iteration 1→2: Obscure dates (use "early career" instead of "1985")
Iteration 2→3: Remove organization names, use role descriptions
Iteration 3→4: Combine multiple obfuscations
Iteration 4→5: Make question require significant inference and browsing
"""
evolution_prompts = {
0: """Rewrite the question to be less specific by describing the person without using their full name.
Use role descriptions (e.g., "a prominent diplomat" instead of "David Hackett Souter").""",
1: """Further obscure the question by replacing specific dates with vague temporal references
(e.g., "early 1980s" instead of "1981", "later years" instead of specific decades).""",
2: """Remove references to specific organizations or institutions.
Use generic descriptors (e.g., "a major legal body" instead of "U.S. Supreme Court").""",
3: """Combine multiple obfuscations: remove names, vague dates, and generic organizations.
Make the question require inference about connections between entities.""",
4: """Create an ambiguous question that requires extensive exploration to disambiguate.
Remove context that makes it obvious which entity the question concerns."""
}
prompt_template = f"""Original question: {question}
Answer: {answer}
{evolution_prompts.get(iteration, "Make the question significantly more difficult by removing all obvious search entry points.")}
Rewrite the question to be harder while remaining answerable.
Return JSON: {{"evolved_question": "...", "still_answerable": true/false, "estimated_difficulty": 1-10}}"""
response = client.messages.create(
model="claude-opus-4-1-20250805",
max_tokens=1000,
messages=[{"role": "user", "content": prompt_template}]
)
evolved = json.loads(response.content[0].text)
return evolved["evolved_question"], evolved.get("estimated_difficulty", 5)
def evolve_to_target_difficulty(question: str, answer: str, target_turns: int = 9,
client = None) -> str:
"""
Evolve query through multiple iterations until it requires target number of tool turns.
Uses a sample of existing LLM judges to validate difficulty progression.
"""
current_question = question
for iteration in range(5):
# Evolve the question
evolved_q, difficulty = evolve_query(current_question, answer, iteration, client)
# Verify with LLM judge (simplified; real implementation uses multiple judges)
judge_prompt = f"""Question: {evolved_q}
Answer: {answer}
Estimate the minimum number of web search/browse tool calls needed to answer this.
Return JSON: {{"estimated_turns": <number>}}"""
judge_response = client.messages.create(
model="claude-opus-4-1-20250805",
max_tokens=200,
messages=[{"role": "user", "content": judge_prompt}]
)
estimated = json.loads(judge_response.content[0].text)["estimated_turns"]
if estimated >= target_turns:
# Question is now sufficiently difficult
return evolved_q
current_question = evolved_q
return current_question
```
The evolution process transforms accuracy from 86.6% (on initial pairs) to 67.1% (on evolved pairs) while increasing average tool turns from 7.9 to 9.9, demonstrating substantially increased complexity and reasoning requirements.
#### 3. Supervised Fine-Tuning Pipeline
Collect high-quality trajectories and train the base model with ReAct formatting.
```python
# sft_training.py
from typing import List, Tuple
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments, Trainer
from dataclasses import dataclass
@dataclass
class TrajectoryData:
"""Stores a single search trajectory in ReAct format"""
question: str
trajectory: List[Dict] # [{"thought": "...", "tool_call": {...}, "observation": "..."}]
final_answer: str
def format_trajectory_as_react(trajectory_data: TrajectoryData) -> str:
"""
Format trajectory into ReAct format with <think>, <tool_call>, <tool_response> tags.
This matches the format the model learns during SFT.
"""
formatted = f"Question: {trajectory_data.question}\n\n"
for step in trajectory_data.trajectory:
formatted += f"<think>\n{step['thought']}\n</think>\n\n"
formatted += f"<tool_call>\n{step['tool_call']}\n</tool_call>\n\n"
formatted += f"<tool_response>\n{step['observation']}\n</tool_response>\n\n"
formatted += f"Final Answer: {trajectory_data.final_answer}"
return formatted
def create_sft_dataset(trajectories: List[TrajectoryData], tokenizer, max_length: int = 8192):
"""
Convert trajectories to tokenized dataset for SFT training.
Uses rejection sampling to ensure only correct trajectories are included.
"""
formatted_texts = []
for traj in trajectories:
# Verify trajectory correctness (rejection sampling)
if traj.final_answer and len(traj.trajectory) > 0:
formatted = format_trajectory_as_react(traj)
formatted_texts.append(formatted)
# Tokenize
encodings = tokenizer(formatted_texts, max_length=max_length, truncation=True,
padding="max_length", return_tensors="pt")
class TrajectoryDataset:
def __init__(self, encodings):
self.encodings = encodings
def __len__(self):
return len(self.encodings["input_ids"])
def __getitem__(self, idx):
return {key: val[idx] for key, val in self.encodings.items()}
return TrajectoryDataset(encodings)
def train_sft_model(model_name: str = "Qwen/Qwen3-8B",
trajectory_dataset = None,
output_dir: str = "./webexplorer-sft"):
"""
Fine-tune base model on trajectory data using standard causal language modeling.
Hyperparameters based on WebExplorer paper (13K samples, 4 epochs).
"""
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)
training_args = TrainingArguments(
output_dir=output_dir,
num_train_epochs=4,
per_device_train_batch_size=32,
learning_rate=1e-5,
weight_decay=0.01,
warmup_steps=500,
logging_steps=100,
save_steps=500,
gradient_accumulation_steps=2,
bf16=True, # Use bfloat16 precision
dataloader_pin_memory=True
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=trajectory_dataset,
tokenizer=tokenizer
)
trainer.train()
model.save_pretrained(output_dir)
return model
```
The SFT phase trains on approximately 13K high-quality trajectories collected through rejection sampling from a capable model. After SFT, the model learns to properly invoke search/browse tools, structure its reasoning with explicit thoughts, and synthesize information into coherent answers.
#### 4. Reinforcement Learning with GRPO and Progressive Context Expansion
Implement the RL training loop with the GRPO algorithm and gradual context window expansion.
```python
# grpo_training.py
import torch
from typing import Dict, List
import numpy as np
class GRPOTrainer:
"""
Group Relative Policy Optimization trainer for web agents.
Gradually expands context length and tool calling limits.
"""
def __init__(self, model, tokenizer, device="cuda"):
self.model = model
self.tokenizer = tokenizer
self.device = device
# Progressive expansion schedule
self.context_schedule = [64000, 96000, 128000] # tokens
self.turn_schedule = [50, 75, 100] # max tool calls
self.stage = 0
def composite_reward(self, format_correct: bool, answer_correct: bool) -> float:
"""
Composite reward function: R = 0.2 * R_format + R_correct
Format rewards evaluate whether tool calls and thoughts use proper tags.
Accuracy rewards use automated judge (DeepSeek-V3) to verify if final
response matches ground truth answer.
"""
r_format = 1.0 if format_correct else 0.0
r_correct = 1.0 if answer_correct else 0.0
return 0.2 * r_format + r_correct
def evaluate_format(self, generated_trajectory: str) -> bool:
"""Check that trajectory contains properly formatted tool calls and thoughts."""
has_think_tags = "<think>" in generated_trajectory and "</think>" in generated_trajectory
has_tool_calls = "<tool_call>" in generated_trajectory and "</tool_call>" in generated_trajectory
has_responses = "<tool_response>" in generated_trajectory and "</tool_response>" in generated_trajectory
return has_think_tags and has_tool_calls and has_responses
def check_answer_correctness(self, generated_answer: str, ground_truth: str,
judge_model = None) -> bool:
"""
Use LLM-as-Judge (DeepSeek-V3) to evaluate if generated answer matches truth.
This enables scalable evaluation of complex reasoning.
"""
if not judge_model:
# Fallback to simple string matching (real impl. uses semantic evaluation)
return generated_answer.lower() in ground_truth.lower()
evaluation_prompt = f"""Ground truth: {ground_truth}
Generated answer: {generated_answer}
Are these answers equivalent or does the generated answer correctly answer the question?
Return: true or false"""
# Simplified; real implementation uses actual judge call
return True # Placeholder
def train_grpo_step(self, questions: List[str], answers: List[str],
group_size: int = 8, batch_size: int = 64):
"""
Perform one GRPO training step with group relative advantage estimation.
Each QA pair generates multiple rollouts, policy gradient is computed relative
to group mean.
"""
# Expand context if entering new stage
if self.stage < len(self.context_schedule) - 1:
self.stage += 1
max_context = self.context_schedule[min(self.stage, 2)]
max_turns = self.turn_schedule[min(self.stage, 2)]
optimizer = torch.optim.AdamW(self.model.parameters(), lr=1e-6)
for batch_start in range(0, len(questions), batch_size):
batch_end = min(batch_start + batch_size, len(questions))
batch_questions = questions[batch_start:batch_end]
batch_answers = answers[batch_start:batch_end]
group_returns = []
group_log_probs = []
# Generate multiple rollouts per question
for question, answer in zip(batch_questions, batch_answers):
rollout_returns = []
rollout_log_probs = []
for _ in range(group_size):
# Generate trajectory with current policy
generated = self._generate_trajectory(
question, max_context, max_turns
)
# Compute rewards
format_ok = self.evaluate_format(generated)
answer_ok = self.check_answer_correctness(
self._extract_answer(generated), answer
)
reward = self.composite_reward(format_ok, answer_ok)
# Compute log probability of trajectory
log_prob = self._compute_log_prob(generated)
rollout_returns.append(reward)
rollout_log_probs.append(log_prob)
group_returns.append(rollout_returns)
group_log_probs.append(rollout_log_probs)
# Compute group relative advantages
all_returns = np.array([r for rets in group_returns for r in rets])
group_mean = np.mean(all_returns)
group_std = np.std(all_returns) + 1e-8
# Policy gradient update using group advantages
loss = 0
for returns, log_probs in zip(group_returns, group_log_probs):
for ret, log_prob in zip(returns, log_probs):
advantage = (ret - group_mean) / group_std
loss += -(advantage * log_prob) # Policy gradient
loss = loss / (len(batch_questions) * group_size)
optimizer.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(self.model.parameters(), 1.0)
optimizer.step()
def _generate_trajectory(self, question: str, max_context: int,
max_turns: int) -> str:
"""Generate full trajectory with current policy (simplified)."""
# Real implementation calls model with proper prompting
return f"<think>I need to search for {question}</think>\n<tool_call>search(...)</tool_call>"
def _extract_answer(self, trajectory: str) -> str:
"""Extract final answer from trajectory."""
if "Final Answer:" in trajectory:
return trajectory.split("Final Answer:")[-1].strip()
return ""
def _compute_log_prob(self, trajectory: str) -> float:
"""Compute log probability of trajectory under current policy."""
# Simplified; real implementation tokenizes and scores with model
return float(torch.tensor(0.5))
```
The RL phase trains on approximately 12K samples without requiring trajectory demonstrations. The model learns to optimize its action selection through interaction rewards while the progressive context expansion (64K→96K→128K tokens, 50→75→100 turns) allows emergence of increasingly sophisticated reasoning patterns. Training shows tool calls increase from ~11 to ~16 per trajectory while performance on BrowseComp-en improves from 7.9% to 15.7%.
### Practical Guidance
#### Hyperparameter Reference
| Parameter | SFT Phase | RL Phase | Notes |
|-----------|-----------|---------|-------|
| Learning Rate | 1e-5 | 1e-6 | Standard practice: reduce for RL |
| Batch Size | 32 | 64 | 128 total with group size 8 |
| Epochs / Steps | 4 epochs | Until convergence | Train on 13K/12K samples respectively |
| Max Context Length | 8K-32K | 64K→96K→128K | Progressive expansion |
| Max Tool Turns | N/A | 50→75→100 | Staged increases |
| Group Size (GRPO) | N/A | 8 rollouts | Groups for relative advantage |
| Format Reward Weight | N/A | 0.2 | Balance structure vs. accuracy |
| Warm-up Steps | 500 | Implicit | 5-10% of training steps |
| Gradient Accumulation | 2 | Implicit | Stabilizes updates |
#### When to Use WebExplorer's Approach
This methodology is most effective when:
- You need to train a capable web agent from scratch on open-source base models (8B-32B parameters)
- You have access to a capable model (Claude, GPT-4) for data synthesis and trajectory collection
- Your domain involves information-seeking tasks (factual QA, research, multi-page navigation)
- You need long-horizon reasoning (8-16+ sequential tool calls, 40K+ token trajectories)
- You can tolerate synthetic training data (Wikipedia-derived knowledge) with minor domain distribution mismatch
- You want transparent, reproducible training pipelines with clear ablations
#### When NOT to Use
Avoid this approach when:
- You need to handle real-time, continuously updated web content (Wikipedia ages; evolved queries may become outdated)
- Your domain is highly specialized with little Wikipedia coverage (medical device manuals, proprietary databases, niche communities)
- You require sub-second tool call latency (RL-trained models average 16 tool calls per query, extending inference time substantially)
- Your evaluation involves adversarial or malicious websites (synthesis with benign models won't prepare agents for evasive content)
- You lack computational resources for full RL training (requires GPU clusters and capable judge models)
- Your task requires real-time learning or adaptation (data synthesis is one-time; doesn't update with new information)
- You need agents to handle non-English queries at comparable quality (training focused on BrowseComp-en/zh; generalization to other languages untested)
#### Common Pitfalls
1. **Insufficient Evolution Iterations**: Stopping evolution early (before 5 iterations) produces queries that remain too easy. The paper shows 86.6% accuracy→67.1% requires full pipeline; early termination yields 75%+ accuracy, reducing multi-step reasoning emergence.
2. **Weak Judge Model**: Using simple string matching instead of semantic evaluation for accuracy rewards causes models to optimize for spurious correlations. Always use capable models (DeepSeek-V3, Claude) as judges.
3. **Too-Aggressive Context Expansion**: Jumping directly from 64K to 128K tokens increases training instability. The staged expansion (64K→96K→128K) prevents catastrophic forgetting of earlier learned behaviors.
4. **Neglecting Format Rewards**: Setting format reward weight to 0 allows models to generate malformed tool calls. Even though format rewards weight only 0.2 in the composite, they're essential for maintaining interpretability and proper tool invocation.
5. **Mixing Real and Evolved Data**: Blending WebExplorer-QA with other datasets (WebSailor, WebShaper) dilutes the difficulty progression. Train exclusively on evolved QA during RL; use pre-trained base models to avoid starting from scratch.
6. **Insufficient Trajectories**: Using fewer than 13K SFT samples or 12K RL samples leads to underfitting. The paper's numbers were carefully chosen based on data diversity requirements; smaller datasets show 2-3 percentage point performance drops.
7. **Ignoring Tool Call Scaling**: If your model plateaus below 12 average tool calls per trajectory, the RL signal is too weak. Verify that reward curves show tool call growth (Figure 5 in paper); if flat, increase group size or adjust reward scaling.
#### Integration Example
This code demonstrates how to orchestrate the full pipeline:
```python
# main.py
import anthropic
from exploration import explore_entity, generate_initial_qa
from evolution import evolve_to_target_difficulty
from sft import train_sft_model, create_sft_dataset
from grpo import GRPOTrainer
def full_webexplorer_pipeline(base_model: str = "Qwen/Qwen3-8B",
num_seeds: int = 100):
"""
Complete WebExplorer training pipeline from data synthesis to final model.
"""
client = anthropic.Anthropic()
# Stage 1: Collect Wikipedia seed entities
seed_entities = ["David Hackett Souter", "Marie Curie", "Albert Einstein"] # Simplified
# Stage 1a: Model-based exploration
print("=== Stage 1a: Model-Based Exploration ===")
initial_qas = []
for entity in seed_entities:
entity_info = explore_entity(entity, client)
question, answer = generate_initial_qa(entity_info, client)
initial_qas.append((question, answer))
# Stage 1b: Long-to-short query evolution
print("=== Stage 1b: Query Evolution (5 iterations) ===")
evolved_qas = []
for question, answer in initial_qas:
evolved_q = evolve_to_target_difficulty(question, answer, target_turns=9, client=client)
evolved_qas.append((evolved_q, answer))
# Stage 2a: Supervised Fine-Tuning
print("=== Stage 2a: SFT Training ===")
# In real implementation: collect 13K trajectories via rejection sampling
trajectories = [] # Placeholder
dataset = create_sft_dataset(trajectories, tokenizer=None)
sft_model = train_sft_model(
model_name=base_model,
trajectory_dataset=dataset,
output_dir="./webexplorer-sft"
)
# Stage 2b: Reinforcement Learning
print("=== Stage 2b: RL Training with GRPO ===")
trainer = GRPOTrainer(sft_model, tokenizer=None)
for epoch in range(3):
questions = [q for q, _ in evolved_qas]
answers = [a for _, a in evolved_qas]
trainer.train_grpo_step(questions, answers, group_size=8, batch_size=64)
# Evaluate on benchmark
# benchmark_score = evaluate_on_browsecomp(trainer.model)
# print(f"Epoch {epoch}: BrowseComp-en = {benchmark_score}%")
# Save final model
sft_model.save_pretrained("./webexplorer-8b-final")
print("=== Training Complete ===")
return sft_model
if __name__ == "__main__":
model = full_webexplorer_pipeline()
```
This pipeline demonstrates the complete workflow: data synthesis (exploration + evolution), SFT training, and RL optimization, mirroring the approach described in the paper.
## Reference
For full methodological details, original results tables, and ablation studies, consult the paper:
Liu et al. (2024). "WebExplorer: Explore and Evolve for Training Long-Horizon Web Agents." HKUST & MiniMax. [https://arxiv.org/abs/2509.06501](https://arxiv.org/abs/2509.06501)
Source code and WebExplorer-8B model are available at [https://github.com/hkust-nlp/WebExplorer](https://github.com/hkust-nlp/WebExplorer) and [https://huggingface.co/hkust-nlp/WebExplorer-8B](https://huggingface.co/hkust-nlp/WebExplorer-8B).