UFO — Multi-Agent Windows UI Automation Framework by Microsoft

Introduction

UFO (UI-Focused Agent) is an open-source framework from Microsoft Research that uses LLM-powered agents to interact with Windows desktop applications. It interprets natural language instructions, takes screenshots to understand the current UI state, and executes actions through the Windows UI Automation API.

What UFO Does

• Automates Windows application workflows via natural language commands
• Uses vision models to understand on-screen UI elements and layouts
• Coordinates multiple specialized agents for complex multi-app tasks
• Interacts with applications through the Windows UI Automation accessibility API
• Supports cross-application workflows spanning multiple desktop programs

Architecture Overview

UFO employs a dual-agent architecture. A HostAgent decomposes high-level user requests into sub-tasks and dispatches them to AppAgents, each controlling a specific Windows application. AppAgents use screenshots and UI Automation tree data to locate controls, plan actions, and execute them. A central memory module tracks completed steps and application state across the workflow.

Self-Hosting & Configuration

• Requires Windows 10/11 with Python 3.10+
• Configure an LLM backend (GPT-4V, Claude, or compatible multimodal model)
• Set API keys for the chosen LLM provider in the config file
• Install accessibility tools and enable UI Automation on the system
• Optionally configure application-specific prompts for better accuracy

Key Features

• Natural language control of any Windows desktop application
• Screenshot-based UI understanding with accessibility tree integration
• Multi-agent coordination for workflows across multiple applications
• Action grounding that maps natural language to specific UI control operations
• MIT-licensed and actively maintained by Microsoft Research

Comparison with Similar Tools

• Browser Use — Focused on web browser automation; UFO targets native Windows desktop applications
• Open Interpreter — General-purpose code execution agent; UFO specializes in GUI-based application control
• AutoHotkey — Scripted macro automation; UFO uses natural language and vision for dynamic interaction
• Power Automate — Microsoft's commercial RPA tool; UFO is open source and LLM-driven

FAQ

Q: Does UFO work on macOS or Linux? A: No. UFO uses the Windows UI Automation API and is designed exclusively for Windows.

Q: Which LLMs work with UFO? A: Any multimodal LLM that accepts images. GPT-4V and similar vision-capable models work best.

Q: Can UFO handle complex multi-step workflows? A: Yes. The HostAgent breaks tasks into sub-tasks and coordinates AppAgents across multiple applications.

Q: Is UFO safe to run on production systems? A: UFO executes real UI actions. Test in a sandboxed environment before running on systems with sensitive data.

Sources

• https://github.com/microsoft/UFO