# Midscene — AI-Powered Vision-Driven UI Automation

> Midscene uses multimodal AI to automate UI testing and interaction across web, mobile, and desktop platforms without brittle selectors.

## Install

Save as a script file and run:

# Midscene — AI-Powered Vision-Driven UI Automation

## Quick Use
```bash
npm install @midscene/web
npx midscene init
```

## Introduction
Midscene is an AI-powered UI automation framework that uses vision models to understand and interact with user interfaces. Instead of relying on fragile CSS selectors or XPath, Midscene interprets screens the way a human would, making tests more resilient to UI changes.

## What Midscene Does
- Uses multimodal AI to visually locate and interact with UI elements
- Supports web browsers, Android, and iOS platforms from a single API
- Generates natural-language action plans from high-level instructions
- Provides assertion capabilities based on visual page understanding
- Integrates with Playwright, Puppeteer, and other automation drivers

## Architecture Overview
Midscene sits as a middleware layer between your test code and the browser or device driver. When you issue a natural-language command, the framework captures a screenshot, sends it to a multimodal model for element recognition and planning, then translates the plan into concrete driver actions. A caching layer avoids redundant AI calls for repeated patterns.

## Self-Hosting & Configuration
- Install via npm as a project dependency or use the CLI scaffolding tool
- Configure the AI provider endpoint and model in a midscene config file
- Supports OpenAI-compatible endpoints including local models
- Set screenshot resolution and device emulation per test suite
- CI integration works through standard Node.js test runners

## Key Features
- Selector-free automation that survives UI redesigns
- Natural-language test authoring for non-technical team members
- Cross-platform support from a unified TypeScript API
- Built-in visual assertions without separate screenshot diffing tools
- Parallel execution support for faster test suites

## Comparison with Similar Tools
- **Playwright** — provides robust browser automation but requires explicit selectors; Midscene adds an AI layer on top
- **Selenium** — mature cross-browser tool with selector-based approach; Midscene avoids selector maintenance
- **Appium** — mobile-focused automation; Midscene covers mobile and web with vision-based interaction
- **TestCafe** — simplified web testing without WebDriver; Midscene goes further by removing selector dependency
- **Cypress** — developer-friendly browser testing; Midscene adds AI-driven element recognition

## FAQ
**Q: Does Midscene require an external AI API?**
A: Yes, it sends screenshots to a multimodal model. You can use OpenAI, Azure, or any compatible endpoint including self-hosted models.

**Q: How does it handle dynamic content?**
A: The vision model re-analyzes each screenshot in real time, so it naturally adapts to dynamic or animated content.

**Q: Can I mix Midscene with traditional selectors?**
A: Yes, Midscene integrates with Playwright and Puppeteer, so you can combine AI-driven actions with selector-based ones.

**Q: What languages are supported for test authoring?**
A: TypeScript and JavaScript via the npm package.

## Sources
- https://github.com/web-infra-dev/midscene
- https://midscenejs.com

---
Source: https://tokrepo.com/en/workflows/asset-a6760d78
Author: Script Depot