Fara 7b Agentic Eval

---
name: fara-7b-agentic-eval
description: This evaluation probes the agentic capabilities of computer-use models by measuring their ability to complete multi-step web browsing and task-completion tasks on live websites. It assesses both functional success rates and operational efficiency, including token usage, cost, and interaction length. Use when the user wants to benchmark on WebVoyager, Online-Mind2Web, DeepShop, WebTailBench, ScreenSpot, or asks about evaluating this task. Reports success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.19663
  bibtex_key: awadallah2025fara7b
  confidence: high
---

# fara-7b-agentic-eval

> Fara-7B: An Efficient Agentic Model for Computer Use — Ahmed Awadallah et al. (2025) (arXiv:2511.19663, 2025)

## What this evaluates

This evaluation probes the agentic capabilities of computer-use models by measuring their ability to complete multi-step web browsing and task-completion tasks on live websites. It assesses both functional success rates and operational efficiency, including token usage, cost, and interaction length.

## Datasets

- **WebVoyager** — total ?; splits: test (-1)
- **Online-Mind2Web** — total ?; splits: test (-1)
- **DeepShop** — total ?; splits: test (-1)
- **WebTailBench** — total 609; splits: test (609)
- **ScreenSpot** — total ?; splits: test (-1)

## Metrics

- `success rate` **(primary)** — range: percent
  - The percentage of tasks successfully completed out of the total number of tasks. Calculated by averaging results over three independent runs, with trajectories capped at 100 steps.
- `cost per task` — range: other
  - The average monetary cost in USD required to complete a single task, derived from token pricing and action counts.
- `actions per task` — range: other
  - The average number of low-level browser actions executed by the model to complete a task.

## Input / output format

**Input**: Screenshot of the current browser/UI state paired with the task instruction/prompt.

**Output**: Low-level browser actions (e.g., click, type, scroll) or agent step commands.

## Scoring recipe

```python
def compute_success_rate(tasks, judge_model):
    successful = 0
    for task in tasks:
        trajectory = run_agent(task.prompt, max_steps=100)
        if trajectory.failed_due_to_env_error:
            trajectory = retry_up_to_5_times(trajectory)
        if judge_model.evaluate(trajectory, task.gold_criteria):
            successful += 1
    return (successful / len(tasks)) * 100
# Average over 3 independent runs for final metric.
```

## Common pitfalls

- Live websites change daily, causing high variance; evaluations must be averaged over multiple runs and only retry on environment errors, not incorrect completions.
- Time-sensitive tasks become stale; requires filtering impossible tasks or modifying future dates to current ones to ensure solvability.
- LLM-as-a-judge success rates often overestimate human-verified accuracy due to prompt and model misalignment.

## Evidence (verbatim from paper)

> We evaluate our agent on three popular benchmarks against live websites: WebVoyager, Online Mind2Web, and DeepShop, as well as our WebTailBench. For ascertaining success rate, we retain the same prompts, llm-as-a-judge model type, and procedure published with each benchmark. Namely, we use GPT-4o along with the official respective prompts in the LLM-based judge for WebVoyager and Deepshop, and o4-mini as the LLM-based judge for Online-Mind2Web and WebTailBench.

## Citation

```bibtex
@misc{awadallah2025fara7b,
  title={Fara-7B: An Efficient Agentic Model for Computer Use},
  author={Ahmed Awadallah et al. (2025)},
  year={2025},
  note={arXiv:2511.19663}
}
```

- arXiv: 2511.19663