OmniParser — Screen Parsing Toolkit for AI Agents

Introduction

OmniParser is a screen parsing library from Microsoft Research that extracts structured information from UI screenshots. It identifies interactive elements, labels, and layout structure so that vision-language models can plan and execute actions on any graphical interface.

What OmniParser Does

• Detects clickable UI elements (buttons, links, inputs) in screenshots
• Extracts text labels and associates them with detected regions
• Outputs structured bounding-box data for downstream agent use
• Works across web pages, desktop apps, and mobile UIs
• Integrates with vision-language models for end-to-end automation

Architecture Overview

OmniParser combines a fine-tuned object detection model for UI element localization with an OCR module for text extraction. The detection model identifies interactable regions while the OCR module reads labels and content. Results are merged into a unified structured output that maps each element to its screen coordinates and semantic label.

Self-Hosting & Configuration

• Clone the repo and install Python dependencies
• Download pre-trained model weights via the provided script
• Run the demo server for interactive testing
• Configure detection thresholds via command-line arguments
• GPU recommended for real-time parsing; CPU mode available

Key Features

• Generalizes across different UI frameworks and platforms
• Provides pixel-accurate bounding boxes for each element
• Combines detection and OCR in a single pipeline
• Open weights released under permissive license
• Benchmarked on ScreenSpot and other UI grounding datasets

Comparison with Similar Tools

• SeeClick — click prediction model; OmniParser provides full element parsing
• CogAgent — end-to-end GUI agent; OmniParser is a modular parsing component
• Ferret-UI — Apple multimodal model; OmniParser focuses on structured output
• Set-of-Mark — visual prompting overlay; OmniParser directly detects elements

FAQ

Q: What image formats does OmniParser accept? A: Standard formats including PNG, JPEG, and BMP are supported.

Q: Does it require a GPU? A: GPU is recommended for speed but CPU inference is supported at lower throughput.

Q: Can OmniParser handle dynamic web pages? A: It operates on static screenshots, so you capture the current state and parse it. For dynamic content, take sequential screenshots.

Q: How accurate is element detection? A: It achieves strong results on standard UI grounding benchmarks. Accuracy varies by UI complexity.

Sources

• https://github.com/microsoft/OmniParser
• https://www.microsoft.com/en-us/research/project/omniparser/