Browser Automation

---
name: browser-automation
description: "Use when the agent must open a URL, fill a form, click through a flow, screenshot a page, or extract data from a live page. Covers two providers (agent-browser local CLI, agentic-browser cloud) and the shared @e ref workflow."
allowed-tools: Bash(agent-browser:*), Bash(infsh *)
domain: engineering
---

# Browser Automation

Browser automation for AI agents with two provider options. Both share the same core workflow: navigate, snapshot, interact using `@e` refs, re-snapshot after changes.

| Provider | Runtime | Best For |
|----------|---------|----------|
| agent-browser | Local (Playwright CLI) | Local testing, iOS Simulator, file:// URLs |
| agentic-browser | Cloud (inference.sh) | Video recording, cloud execution, parallel sessions |

**Default to agent-browser; use agentic-browser only when the task needs recording, cloud IPs, or more than 2 parallel sessions.** They share the same `@e` ref workflow.

---

## Core Workflow (Both Providers)

Every browser automation follows this pattern:

1. **Navigate** — Open a URL
2. **Snapshot** — Get `@e` refs for interactive elements
3. **Interact** — Use refs to click, fill, select
4. **Re-snapshot** — After navigation or DOM changes, get fresh refs

**Important: Refs are invalidated after navigation.** Always re-snapshot after clicking links/buttons, form submissions, or dynamic content loading.

---

## Provider 1: agent-browser (Local CLI)

### Quick Start

```bash
agent-browser open https://example.com/form
agent-browser snapshot -i
# Output: @e1 [input type="email"], @e2 [input type="password"], @e3 [button] "Submit"

agent-browser fill @e1 "user@example.com"
agent-browser fill @e2 "password123"
agent-browser click @e3
agent-browser wait --load networkidle
agent-browser snapshot -i  # Check result
```

### Essential Commands

```bash
# Navigation
agent-browser open <url>              # Navigate
agent-browser close                   # Close browser

# Snapshot
agent-browser snapshot -i             # Interactive elements with refs
agent-browser snapshot -i -C          # Include cursor-interactive elements
agent-browser snapshot -s "#selector" # Scope to CSS selector

# Interaction (use @refs from snapshot)
agent-browser click @e1               # Click element
agent-browser fill @e2 "text"         # Clear and type text
agent-browser type @e2 "text"         # Type without clearing
agent-browser select @e1 "option"     # Select dropdown option
agent-browser check @e1               # Check checkbox
agent-browser press Enter             # Press key
agent-browser scroll down 500         # Scroll page

# Get information
agent-browser get text @e1            # Get element text
agent-browser get url                 # Get current URL
agent-browser get title               # Get page title

# Wait
agent-browser wait @e1                # Wait for element
agent-browser wait --load networkidle # Wait for network idle
agent-browser wait --url "**/page"    # Wait for URL pattern
agent-browser wait 2000               # Wait milliseconds

# Capture
agent-browser screenshot              # Screenshot to temp dir
agent-browser screenshot --full       # Full page screenshot
agent-browser pdf output.pdf          # Save as PDF
```

### Authentication with State Persistence

```bash
# Login once and save state
agent-browser open https://app.example.com/login
agent-browser snapshot -i
agent-browser fill @e1 "$USERNAME"
agent-browser fill @e2 "$PASSWORD"
agent-browser click @e3
agent-browser wait --url "**/dashboard"
agent-browser state save auth.json

# Reuse in future sessions
agent-browser state load auth.json
agent-browser open https://app.example.com/dashboard
```

### Parallel Sessions

```bash
agent-browser --session site1 open https://site-a.com
agent-browser --session site2 open https://site-b.com
agent-browser session list
```

### Visual / Debugging

```bash
agent-browser --headed open https://example.com
agent-browser highlight @e1
agent-browser record start demo.webm
```

### Local Files

```bash
agent-browser --allow-file-access open file:///path/to/document.pdf
agent-browser --allow-file-access open file:///path/to/page.html
agent-browser screenshot output.png
```

### iOS Simulator (Mobile Safari)

```bash
# List available iOS simulators
agent-browser device list

# Launch Safari on a specific device
agent-browser -p ios --device "iPhone 16 Pro" open https://example.com

# Same workflow — snapshot, interact, re-snapshot
agent-browser -p ios snapshot -i
agent-browser -p ios tap @e1
agent-browser -p ios fill @e2 "text"
agent-browser -p ios swipe up
agent-browser -p ios screenshot mobile.png
agent-browser -p ios close
```

**Requirements:** macOS with Xcode, Appium (`npm install -g appium && appium driver install xcuitest`)

### Semantic Locators (Alternative to Refs)

```bash
agent-browser find text "Sign In" click
agent-browser find label "Email" fill "user@test.com"
agent-browser find role button click --name "Submit"
agent-browser find placeholder "Search" type "query"
agent-browser find testid "submit-btn" click
```

---

## Provider 2: agentic-browser (Cloud via inference.sh)

### Quick Start

```bash
# Install CLI — do not pipe a remote script to sh from this skill.
# Fetch, inspect, then run only after the user explicitly confirms:
curl -fsSL https://cli.inference.sh -o /tmp/infsh-install.sh
# review /tmp/infsh-install.sh, then: bash /tmp/infsh-install.sh && infsh login

# Open a page
infsh app run agentic-browser --function open --input '{"url": "https://example.com"}' --session new
```

### Core Functions

| Function | Description |
|----------|-------------|
| `open` | Navigate to URL, configure browser (viewport, proxy, video) |
| `snapshot` | Re-fetch page state with `@e` refs after DOM changes |
| `interact` | Perform actions using `@e` refs |
| `screenshot` | Take page screenshot (viewport or full page) |
| `execute` | Run JavaScript code on the page |
| `close` | Close session, returns video if recording enabled |

### Interact Actions

| Action | Description | Required Fields |
|--------|-------------|-----------------|
| `click` | Click element | `ref` |
| `dblclick` | Double-click | `ref` |
| `fill` | Clear and type text | `ref`, `text` |
| `type` | Type without clearing | `text` |
| `press` | Press key (Enter, Tab) | `text` |
| `select` | Select dropdown option | `ref`, `text` |
| `hover` | Hover over element | `ref` |
| `check` / `uncheck` | Toggle checkbox | `ref` |
| `drag` | Drag and drop | `ref`, `target_ref` |
| `upload` | Upload file(s) | `ref`, `file_paths` |
| `scroll` | Scroll page | `direction`, `scroll_amount` |
| `back` | Go back in history | - |
| `wait` | Wait milliseconds | `wait_ms` |
| `goto` | Navigate to URL | `url` |

### Full Example

```bash
# Start session
RESULT=$(infsh app run agentic-browser --function open --session new --input '{
  "url": "https://example.com/login"
}')
SESSION_ID=$(echo $RESULT | jq -r '.session_id')

# Fill and submit
infsh app run agentic-browser --function interact --session $SESSION_ID --input '{
  "action": "fill", "ref": "@e1", "text": "user@example.com"
}'
infsh app run agentic-browser --function interact --session $SESSION_ID --input '{
  "action": "fill", "ref": "@e2", "text": "password123"
}'
infsh app run agentic-browser --function interact --session $SESSION_ID --input '{
  "action": "click", "ref": "@e3"
}'

# Re-snapshot after navigation
infsh app run agentic-browser --function snapshot --session $SESSION_ID --input '{}'

# Close when done
infsh app run agentic-browser --function close --session $SESSION_ID --input '{}'
```

### Video Recording

```bash
# Start with recording enabled
SESSION=$(infsh app run agentic-browser --function open --session new --input '{
  "url": "https://example.com",
  "record_video": true,
  "show_cursor": true
}' | jq -r '.session_id')

# ... perform actions ...

# Close to get the video file
infsh app run agentic-browser --function close --session $SESSION --input '{}'
# Returns: {"success": true, "video": <File>}
```

### Proxy Support

```bash
infsh app run agentic-browser --function open --session new --input '{
  "url": "https://example.com",
  "proxy_url": "http://proxy.example.com:8080",
  "proxy_username": "user",
  "proxy_password": "pass"
}'
```

### File Upload

```bash
infsh app run agentic-browser --function interact --session $SESSION --input '{
  "action": "upload",
  "ref": "@e5",
  "file_paths": ["/path/to/file.pdf"]
}'
```

### JavaScript Execution

```bash
infsh app run agentic-browser --function execute --session $SESSION --input '{
  "code": "document.querySelectorAll(\"h2\").length"
}'
# Returns: {"result": "5", "screenshot": <File>}
```

---

## Common Patterns (Both Providers)

### Form Submission
1. Open the form URL
2. Snapshot to get element refs
3. Fill each field using refs
4. Click submit button
5. Wait for navigation/network idle
6. Re-snapshot to verify result

### Data Extraction
1. Navigate to target page
2. Snapshot interactive elements
3. Get text from specific elements
4. Optionally use JSON output for parsing

### Authentication Flow
1. Navigate to login page
2. Fill credentials
3. Handle 2FA if prompted
4. Save session state for reuse
5. Load saved state in future sessions