Small Language Models are the Future of Agentic AI
Small Language Models are the Future of Agentic AI
We are used to thinking that in the world of artificial intelligence, size matters. Giant language models (LLMs), like the ones behind ChatGPT, capture the imagination with their ability to write poetry, generate code, and conduct philosophical conversations. Tech giants are investing billions in creating ever larger and more powerful “brains” for AI. But what if this path is a dead end?
- The problem with giants: expensive and inefficient
- The solution: a team of narrow specialists
- Advantages of the “small” approach
- What prevents the transition?
We are used to thinking that in the world of artificial intelligence, size matters. Giant language models (LLMs), like the ones behind ChatGPT, capture the imagination with their ability to write poetry, generate code, and conduct philosophical conversations. Tech giants are investing billions in creating ever larger and more powerful “brains” for AI. But what if this path is a dead end?
Recent work by researchers from NVIDIA, one of the industry’s flagships, challenges this idea. In their article “Small Language Models are the Future of Agentic AI,” they argue that the future belongs not to monstrous generalists, but to teams of fast, cheap, and effective “specialists” — small language models (SLMs).
The problem with giants: expensive and inefficient
Today, most AI agents — programs that not only answer questions but also perform tasks for us (book tickets, plan projects, manage a smart home) — run on huge LLMs. This is like using a supercomputer to work with a calculator.
Imagine you ask your AI assistant to add a meeting to your calendar. Behind the scenes, your simple request is sent to a huge data center, where it is processed by a model with hundreds of billions of parameters, capable of analyzing Shakespeare. The result — it works, but it is:
- Expensive: Every such request costs real money, which results in astronomical bills for cloud computing for developers and high subscription prices for users.
- Slow: Data transfer and processing on a giant model create delays.
- Excessive: 99% of the model’s power is not used for such a simple task.
NVIDIA researchers say we’ve fallen into a trap: we use a universal “brain” to perform highly specialized, repetitive tasks. This is economically inefficient and environmentally irrational.
The solution: a team of narrow specialists
What do they suggest instead? An idea that has long been used in engineering and business — modularity. Instead of one giant, clumsy AI genius — a whole team of small language models (SLMs), each of which is trained for its own task.
An SLM is a neural network that is compact enough to run on a regular laptop or even a smartphone. It cannot reason about the meaning of life, but it will perform its function perfectly.
How will this work in practice?
Let’s imagine the same AI agent, but built on an SLM architecture. You ask it to plan a trip.
- The main “dispatcher” (a small SLM) receives your request “Plan a trip to St. Petersburg for the weekend.”
- It recognizes subtasks and calls other “specialists”:
- SLM-flight search engine accesses airline APIs and finds the best options.
- SLM-hotelier searches for suitable hotels according to your criteria.
- SLM-guide compiles a short list of attractions.
- SLM-secretary beautifully formats all the information into a single plan.
Each of these models is small, works lightning fast, and consumes a minimum of resources. And only if you ask a complex, open-ended question, like “Tell me about the architectural features of the Baroque in St. Petersburg,” the system can turn to a large, “erudite” LLM for help.
Advantages of the “small” approach
Such a system, according to NVIDIA, surpasses the current model on all fronts:
- Cost-effectiveness: The cost of processing requests can be reduced by 10-30 times. This will make AI agents more accessible to everyone.
- Speed: Responses will be almost instantaneous, as most of the calculations can be done right on your device.
- Flexibility: It is much easier to update or “retrain” one small model (for example, for a new booking site interface) than to touch a huge LLM.
- Security and confidentiality: If the models work on your device, your personal data does not need to be sent to third-party servers.
What prevents the transition?
If everything is so good, why are we not yet living in a world of SLMs? The authors highlight three main barriers:
- Investment inertia: Companies have already invested billions of dollars in infrastructure for LLMs and are in no hurry to change course.
- Habit of benchmarks: Developers are used to measuring the “coolness” of models on general erudition tests, and not on efficiency in real tasks.
- Marketing noise: All the glory and attention of the press goes to large models, while SLMs remain in the shadows.
Nevertheless, the authors are confident that economic feasibility and common sense will eventually prevail. They even offer a specific algorithm for how developers can gradually transfer their agents from LLM to SLM, collecting data on typical requests and training specialized models on them.
Academic work: Small Language Models are the Future of Agentic AI