MMOOZZGG.com

How to build an AI agent

Local LLM for an AI agent: how to choose and test one

6 min read10 sources

A local LLM for an AI agent is an open model you can download and run on your own computer or server, without sending data out. For an agent, the model must call tools, fit your memory and have a license that allows business use: Qwen 3 came out under Apache 2.0, and DeepSeek R1 under MIT.

On this page · 5 sections

Build it yourself in a week

What an agent needs from a model

A chatbot is fine with a model that writes well. An agent needs more: it chooses its own steps, calls a CRM, a calendar or a knowledge base, and checks the result.

So you choose a model for an agent by four signs. It can call tools, it fits in your computer’s memory, its license allows commercial use, and it follows your rules on your tasks.

The Russian language is checked in the last point. According to its developer, the Qwen 3 family supports 119 languages and dialects, but only a test shows the quality of answers in Russian on your topic.

If the agent works with customer data, a model on your own server in Russia makes it easier to comply with 152-FZ, Russia’s personal data law: data of Russian citizens is collected and stored in databases on Russian territory, and a transfer abroad needs a notice to Roskomnadzor, the data protection regulator. How we set this up is described on the page about data security.

The full list of open models with sizes and release dates is on the page about open language models. Here is only what matters for an agent.

Tool calling

Tool calling (function calling) is a model’s ability to answer not with a phrase but with a command: which function to call and with which parameters. For example, “find a customer by phone number” or “create a deal”. Without this ability, the agent cannot work with your systems.

Ollama, a program for running open models on your own computer, supports tool calling: the list of functions goes in the “tools” field of the request to the model. All the examples in the Ollama documentation are built on the qwen3 model.

The model library on the Ollama website has a “Tools” filter: it keeps only models that can call functions. On 16 October 2026 it showed 99 models, and that list is the place to start.

The Qwen 3 developers say separately that they improved tool calling and support for MCP, a protocol for connecting AI to services. OpenAI’s open gpt-oss models are also designed for agent work: function calling, web search, running Python code.

You can compare models by function calling accuracy in the open Berkeley Function Calling Leaderboard. Its fourth version also rates the agent’s work as a whole, and the last update came out on 12 April 2026.

Model size and hardware

Model size is counted in parameters: the numbers a model learned during training. The more parameters, the more the model knows and the more memory it needs.

Qwen 3 has six regular models from 0.6 to 32 billion parameters and two mixture-of-experts models. In such a model only part of the parameters work on each answer: 3 billion in the 30-billion model, 22 billion in the 235-billion model.

According to OpenAI, the gpt-oss-20b model runs in 16 GB of memory, and gpt-oss-120b fits on one graphics card with 80 GB. Such a server card costs $12,820, and a 32 GB graphics card costs $8,890 (prices as of 29.09.2026).

You do not need to buy hardware before testing. In Russia you can rent a gaming graphics card for about $1 an hour and run your tasks on it.

The memory calculator shows how much memory a specific model takes. For your first agent, take a model that fits your hardware with room to spare: the agent also needs memory for a long dialogue history.

License

Open weights do not always mean freedom for business. Read the license before you build an agent on the model, not after launch.

The Qwen 3 family came out on 29 April 2025 under the Apache 2.0 license, and the gpt-oss models are under Apache 2.0 too. DeepSeek R1 came out on 20 January 2025 with open weights under the MIT license. Both licenses allow commercial use and changes.

Ollama is distributed under the MIT license, but each model in its library has its own license. Check it on the model page and keep it with the project documents.

How to test a model on your own tasks

Rankings show an average picture on other people’s tasks. A test on your own tasks decides, and you can run it in a week without a programmer.

1. Write down 20–50 real agent tasks from correspondence and logs: customer questions, CRM queries, rescheduled appointments.

2. For each task, write down the correct result: which tool the agent must call and what to answer the customer.

3. Pick 2–3 models from the library with the “Tools” filter that fit your memory.

4. Run the tasks through each model with the same instructions and the same set of tools.

5. Count the share of correct answers, the share of correct tool calls and the response time. The winner is the model with fewer errors by your rules, not the one with a loud name.

How to set up such a test and which open programs calculate the metrics is covered on the page about agent evaluation. Installing Ollama step by step is described on the page about a local agent.

Questions and answers

Is a small model on a laptop enough?

For a trial, yes. According to OpenAI, the gpt-oss-20b model runs in 16 GB of memory, and Qwen 3 has models from 0.6 to 32 billion parameters. Test the chosen model on 20–50 of your own tasks: if the share of correct tool calls is low, take a larger model or rent a server.

Can you change the model after launch?

Yes, if the agent is built correctly: the instructions, tools and knowledge base do not depend on the model. In a request to Ollama, the model is set by one field with its name. After the swap, run the same task set to make sure quality did not drop.

Sources and fact-checking

Checked by the editors on 16 October 2026.

  1. Alpine Linux packages, 2026 (pkgs.alpinelinux.org)
  2. Simon Willison, 20 January 2025 (simonwillison.net)
  3. Simon Willison, 29 April 2025 (simonwillison.net)
  4. Brace Law Firm, 9 July 2023 (brace-lf.com)
  5. Qwen Blog, 29 April 2025 (qwenlm.github.io)
  6. GitHub, openai/gpt-oss, 2025 (github.com)
  7. Ollama Docs, Tool calling, 2026 (docs.ollama.com)
  8. Ollama, model library, 16 October 2026 (ollama.com)
  9. UC Berkeley, BFCL, 12 April 2026 (gorilla.cs.berkeley.edu)
  10. Anthropic Engineering, 9 January 2026 (anthropic.com)

Found a mistake? Tell us

Publications

Contact us