# UFO — Multi-Agent Windows UI Automation Framework by Microsoft

> An open-source multi-agent framework from Microsoft Research that automates Windows application interactions using natural language commands and vision-based UI understanding.

## Install

Save as a script file and run:

# UFO — Multi-Agent Windows UI Automation Framework by Microsoft

## Quick Use
```bash
git clone https://github.com/microsoft/UFO.git
cd UFO
pip install -r requirements.txt
python -m ufo --task "Open Notepad and type hello world"
```

## Introduction
UFO (UI-Focused Agent) is an open-source framework from Microsoft Research that uses LLM-powered agents to interact with Windows desktop applications. It interprets natural language instructions, takes screenshots to understand the current UI state, and executes actions through the Windows UI Automation API.

## What UFO Does
- Automates Windows application workflows via natural language commands
- Uses vision models to understand on-screen UI elements and layouts
- Coordinates multiple specialized agents for complex multi-app tasks
- Interacts with applications through the Windows UI Automation accessibility API
- Supports cross-application workflows spanning multiple desktop programs

## Architecture Overview
UFO employs a dual-agent architecture. A HostAgent decomposes high-level user requests into sub-tasks and dispatches them to AppAgents, each controlling a specific Windows application. AppAgents use screenshots and UI Automation tree data to locate controls, plan actions, and execute them. A central memory module tracks completed steps and application state across the workflow.

## Self-Hosting & Configuration
- Requires Windows 10/11 with Python 3.10+
- Configure an LLM backend (GPT-4V, Claude, or compatible multimodal model)
- Set API keys for the chosen LLM provider in the config file
- Install accessibility tools and enable UI Automation on the system
- Optionally configure application-specific prompts for better accuracy

## Key Features
- Natural language control of any Windows desktop application
- Screenshot-based UI understanding with accessibility tree integration
- Multi-agent coordination for workflows across multiple applications
- Action grounding that maps natural language to specific UI control operations
- MIT-licensed and actively maintained by Microsoft Research

## Comparison with Similar Tools
- **Browser Use** — Focused on web browser automation; UFO targets native Windows desktop applications
- **Open Interpreter** — General-purpose code execution agent; UFO specializes in GUI-based application control
- **AutoHotkey** — Scripted macro automation; UFO uses natural language and vision for dynamic interaction
- **Power Automate** — Microsoft's commercial RPA tool; UFO is open source and LLM-driven

## FAQ
**Q: Does UFO work on macOS or Linux?**
A: No. UFO uses the Windows UI Automation API and is designed exclusively for Windows.

**Q: Which LLMs work with UFO?**
A: Any multimodal LLM that accepts images. GPT-4V and similar vision-capable models work best.

**Q: Can UFO handle complex multi-step workflows?**
A: Yes. The HostAgent breaks tasks into sub-tasks and coordinates AppAgents across multiple applications.

**Q: Is UFO safe to run on production systems?**
A: UFO executes real UI actions. Test in a sandboxed environment before running on systems with sensitive data.

## Sources
- https://github.com/microsoft/UFO

---
Source: https://tokrepo.com/en/workflows/asset-a392b702
Author: Script Depot