MMOOZZGG.com

Glossary

What is computer use in AI?

In short

Computer use is the ability of an AI agent to work with programs like a person: look at the screen, move the cursor, press buttons and type text.

AI agents · computer use

MMOOZZGG.com editorsChecked: 1 min read

For a child

Usually, to help you with a computer, you need an adult nearby: they look at the screen, show you where to click and move the mouse themselves. Now there are programs that can do the same: they see the picture on the screen and press the right buttons. But they are still slow and sometimes make mistakes, so a person keeps an eye on them, as on a beginner.

In plain words

Many programs cannot exchange data with others directly: they have only windows, buttons and fields. Before, AI could not connect to such programs. In the new mode, the model gets a screenshot, understands what is on it and gives a command: click a point, type text, press a key. Then it gets a new screenshot and decides what to do next. This way it can fill in a form on a supplier’s website or move data from an email into an old accounting program. It works slower and costs more than a direct connection.

For business

This mode helps where a system has no API and rebuilding it is expensive: old accounting programs, procurement portals, supplier accounts. In effect, it replaces robots with rigid scripts (RPA): the agent does not break if a button moves. The downsides are serious. Each step needs a screenshot and a call to the model, so a task costs more and takes longer than through a direct connection. Errors happen too: at the launch in October 2024, Anthropic openly called the feature experimental and error-prone. Also, the agent sees everything on the screen, including other people’s data. Run it in a separate virtual machine with minimal rights, and test it first in a pilot on one operation.

For a specialist

Computer use is a class of tools in which a multimodal model acts in a graphical interface: it receives screenshots and returns mouse and keyboard actions. Anthropic released this feature in public beta on 22 October 2024; on the OSWorld test, the Claude 3.5 Sonnet model then scored 14.9% in the screenshot-only category. OSWorld itself (Xie et al., 2024) contains 369 tasks in real operating systems: people solve more than 72.36% of them, while models at the time of publication solved no more than 12.24%. The work cycle is screenshot, reasoning, action, new screenshot. Close variants are browser agents that read the page structure and agents with command-line access. Typical problems are coordinate accuracy, long chains of steps and prompt injections through screen content. Runs are isolated in a container or a virtual machine.

Example

A procurement manager moves orders from a supplier portal into an accounting program without an API every day. The agent opens the portal, copies new orders and fills in the records, and in the morning the manager checks the list and confirms it.

Related terms

Questions and answers

How is computer use different from RPA? RPA repeats a recorded script and breaks when the interface changes. An agent looks at the screen anew each time and finds the right button itself, but it works slower and costs more.
Is it safe to let AI control a computer? Only in an isolated environment: a separate virtual machine, its own user without extra rights and no access to personal data. A person confirms important actions, such as sending a payment.

Sources

Where companies use it

Not sure whether your company needs this?

AI solutions map
Contact us