Drive Web And App Uis With Vision Grounded Steps When Selectors Are Brittle Or Unavailable

---
name: "Drive web and app UIs with vision-grounded steps when selectors are brittle or unavailable"
slug: "drive-web-and-app-uis-with-vision-grounded-steps-when-selectors-are-brittle-or-unavailable"
description: "Use Midscene.js when an agent needs screenshot-grounded UI actions and assertions across web, mobile, or desktop surfaces where DOM selectors are fragile, unavailable, or not the right abstraction."
github_stars: 12613
verification: "security_reviewed"
source: "https://github.com/web-infra-dev/midscene"
author: "web-infra-dev"
publisher_type: "organization"
category: "Browser Automation"
framework: "Multi-Framework"
tool_ecosystem:
  github_repo: "web-infra-dev/midscene"
  github_stars: 12613
  npm_package: "@midscene/core"
  npm_weekly_downloads: 83670
---

# Drive web and app UIs with vision-grounded steps when selectors are brittle or unavailable

Use Midscene.js when an agent needs screenshot-grounded UI actions and assertions across web, mobile, or desktop surfaces where DOM selectors are fragile, unavailable, or not the right abstraction.

## Prerequisites

Midscene.js, Node.js, a supported vision model, and a target automation surface such as Playwright, Puppeteer, Android adb, or iOS WebDriverAgent

## Installation

Requirements and caveats from upstream:
- [midscene-pc-docker](https://github.com/Mofangbao/midscene-pc-docker) - Docker image with Midscene-PC server pre-installed
- [Midscene-Python](https://github.com/Python51888/Midscene-Python) - Python SDK for Midscene automation

Basic usage or getting-started notes:
- Sample Projects: [https://github.com/web-infra-dev/midscene-example](https://github.com/web-infra-dev/midscene-example)

- Source: https://github.com/web-infra-dev/midscene
- Extracted from upstream docs: https://raw.githubusercontent.com/web-infra-dev/midscene/HEAD/README.md

## Documentation

- https://midscenejs.com

## Source

- [Agent Skill Exchange](https://agentskillexchange.com/skills/drive-web-and-app-uis-with-vision-grounded-steps-when-selectors-are-brittle-or-unavailable/)