Windows-Use: Enabling LLMs to Control the Windows GUI Without Vision Models

Windows-Use acts as a bridge between large language models (LLMs) and the Windows operating system. It operates directly at the GUI layer, allowing an agent to open applications, click buttons, type text, execute commands, and retrieve the current UI state. While most automation tools rely on computer vision to "see" and interpret the screen, Windows-Use connects the AI agent directly to the Windows backend, bypassing the need for visual processing.

Installation

To use Windows-Use, ensure your system meets the following requirements:

• Python 3.12 or newer
• UV (recommended) or pip
• Windows 7, 8, 10, or 11

Install Windows-Use

Using uv:

uv pip install windows-use

Using pip:

pip install windows-use

Basic Usage

The following example from main.py demonstrates how to initialize the agent:

from langchain_google_genai import ChatGoogleGenerativeAI
from windows_use.agent import Agent
from dotenv import load_dotenv

load_dotenv()

llm = ChatGoogleGenerativeAI(model='gemini-2.0-flash')
agent = Agent(llm=llm, browser='chrome', use_vision=True)
query = input("Enter your query: ")
agent_result = agent.invoke(query=query)
print(agent_result.content)

Running the Agent

To execute the agent as a script:

1. Run the command: python main.py
2. When prompted, enter your specific task: Enter your query: <your task>

Capabilities and Examples

Windows-Use can handle a variety of desktop automation tasks, such as:

• Task: "Write a short note about LLMs and save it to the desktop." (Demonstrated in: Word.mov)
• Task: "Switch from dark mode to light mode." (Demonstrated in: Dark.to.Light.Mode.mov)

A Word of Caution

Because Windows-Use interacts with the operating system at the GUI level, it possesses the ability to modify system settings and files. While the tool is designed to be intelligent and precise, errors can occur, and a misinterpretation of a prompt could lead to unintended system changes. For maximum security, it is highly recommended to run the agent within a sandbox or a virtualized environment.