I Wired Tools into My Local AI — Then It Started Calling Them on Its Own

When I wrote about Miaoban in essay 012, most of the work was getting an ESP32 cat to hear me, then answer with a local model on the computer. The point then was to make it actually speak. I kept that stack going and turned it into AIHub: a voice assistant that can run on my own computer, or on a server on the LAN. Speech recognition, the chat model, and speech synthesis can all stay local. Everyday conversation does not have to go to the cloud.

It is not only for one cat anymore. Open a page on the computer and you can chat or record. Miaoban on the ESP32 can be a terminal with a microphone and a speaker. I also built a command-line client, so from a terminal I can type, talk, or start a recording. They can all connect to the same AIHub service on the LAN. Models and tools live on the server, so a terminal does not need its own AI compute. A phone app can plug in later. For now I want these few ways in to feel easy.

I also did not want the deploy tied to one Mac. It can run natively on a Mac, and there is a Docker path for macOS and x86 Linux. Docker runs the service. That does not mean all the inference is packed into the container. On a Mac, the MLX models still run on the host. An x86 machine can call ASR, LLM, and TTS services that are deployed locally and exposed over HTTP. "Local," here, means the models and the private data stay in my hands. Whether a weather or news MCP tool reaches the public internet depends on that tool.

Once listen, think, and speak was actually working, I wanted the next step. If it can chat, can it also check the weather, set a timer, look at the computer's status, or call something else through MCP?

Wiring the tools in was not the hard part. Talking to it every day was where I hit something I had not expected. Sometimes I never asked it to do anything, and it went and called a tool on its own.

I might have just asked about the weather, then said 「嗯，好的」 ("mm, okay"). It could still be holding the last weather question. Or I only wanted to talk about 「天气查询工具怎么工作」 ("how does the weather-lookup tool work"), and it treated the sentence as a query. One extra weather check mostly wastes a little time. If the tool later writes files, sends a message, or changes a calendar, a misfire stops being a small bug.

I am not trying to solve "how does an AI learn more tools." I want it to know, first, when it should do nothing.

Why a small model should judge the intent

The simple plan is to let the 4B model that does the chatting see the tool list too, and decide for itself whether to call, which tool, and what arguments to pass. That is quick to build, and a new tool is easy to add. Local voice has a tighter demand. You finish a sentence, and you want a reaction soon. If every line is also handed to the 4B for a full tool plan, speed and stability get harder to control.

So in AIHub I tried another route. I took an existing Chinese Rex base model and fine-tuned it with LoRA, for intent classification and for pulling the arguments out. It is not there to chat. It mainly decides whether this sentence is ordinary talk, or a request for weather, news, system info, a shopping list, a timer, or where an object is. Take 「查一下上海明天的天气」 ("check Shanghai's weather for tomorrow"). It should recognize a weather query, and pull out 「上海」 ("Shanghai") and 「明天」 ("tomorrow"). Code then checks that the arguments are valid.

That is different from asking the larger model to write out a long tool-call JSON. The small model makes one judgment, with a clear boundary. Only if the sentence qualifies does the task get handed off. The rest of the conversation is still answered by the larger model.

A short stretch of real use showed me that chat is not as tidy as the training samples.

The first pitfall: an ordinary sentence, taken as a command

The examples that are easiest to train are 「查天气」 ("check the weather"), 「查询内存」 ("check memory"), and 「启动计时器」 ("start a timer"). Real use is full of sentences that should trigger nothing: 「天气不错」 ("nice weather"), 「你会查天气吗」 ("can you check the weather?"), 「刚才那几个工具响应有点慢」 ("those tools were a bit slow just now"), 「不用查天气，讲个笑话」 ("don't check the weather, tell a joke"). The tool words are sitting right there. The intent is something else.

This time I went back through the training set on purpose. router-everyday-v5 in the repo has 9,206 training examples, 1,230 validation examples, and 853 test examples. The labels keep null on purpose, for "no tool needed." Later, for the spoken lines that misfire in real conversation, I added hard negatives: wake words, small talk, comments, negations, and talk about a tool that does not ask it to run. They do not teach the model to do more. They teach it to hold back a little.

The repo docs record one comparison worth keeping. Accuracy on the validation set was 98.2%, and the false-trigger rate on the null class was 0.16%. Evaluated on 378 turns of real conversation logs, overall accuracy only moved from 87.9% to 89.6%, and the false-trigger rate on turns that needed no tool fell from 7.6% to 3.6%. Those are the project's evaluation notes from then. They are not production numbers I re-ran today. The 98.2% on the validation set is not how reliable it is on an ordinary day with the microphone open.

The reminder is plain. A high score on an offline test set does not mean that, once the mic is on, it will stop calling tools at random. For a voice assistant, even if most turns are judged right, one tool call that comes out of nowhere still breaks the experience.

The second pitfall: leave the context out and it fails; bring too much and it fails too

This is messier than classifying a single sentence. First turn, I say 「查一下上海今天的天气」 ("check Shanghai's weather today"). Second turn, 「那明天呢？」 ("and tomorrow?"). That is still the weather, and it should keep Shanghai. If the second turn is 「嗯，不错啊」 ("yeah, not bad"), or 「刚才你回答得有点慢」 ("you were a bit slow just now"), the weather intent should not come along.

I used to treat "let the model see more context" as the way to understand better. After tool routing, I found that chat context and execution context should stay apart. History can help an answer. When the decision is whether to start a tool, last turn's intent can be the interference.

The code already has a limit. By default, Rex judges only the current user sentence. It attaches the previous user turn only when the new line looks like a short follow-up. Pure wake words, small talk, goodbyes, and feedback about the tool itself still have a guard in the rule layer. Context does not hand Rex the whole chat unconditionally.

I still cannot say this is solved. In Chinese, 「呢」 (a soft "and what about…?"), 「继续」 ("go on"), and 「那」 ("then" / "and what about") are fuzzy on their own. The previous tool arguments are also stored as their own state. Isolating the history text is not enough. I still have to decide when those arguments may be inherited, and when they should be cleared. Next I want a focused test of three kinds of back-to-back talk: check the weather, then chat; set a timer, then change the subject; and a real follow-up. I want the wrong calls and the missed calls for each strategy.

The third pitfall: MCP tools keep arriving, and you cannot retrain for each one

Rex fits intents that are fixed, frequent, and clearly bounded. It does not know which MCP tools a person will install later. Today it is weather and a timer. Tomorrow it might be an external calendar. The day after, a file tool or a knowledge base. A model with a fixed set of labels cannot recognize those new abilities out of nothing.

What AIHub does now is a mix. Some clear commands go through rules. Rex handles the core intents it was trained on. For an MCP extension the user has just turned on, the 4B still reads this one sentence and the tool definitions, and makes a constrained plan. Arguments for an extension tool also have to pass a schema check. Matching a name in the sentence is not enough to run it.

So I cannot say that a new MCP needs no large model at all, and I cannot say that Rex already learns a new tool on its own. That is the part I want to improve. Can the small model first decide, reliably, whether this turn has any intent to act, and then pick candidates from the tools that are available — instead of folding every new tool into the next round of fixed-label training?

One step past that is local post-training, after it is already in use. If someone sees that a sentence was judged wrong, can the device keep one confirmed correction, and train a new LoRA on a schedule? I do not want the new version to overwrite the old one directly. Compare it with a fixed test set first. Switch only after the old tools have at least not been made worse. If something goes wrong, roll back.

That is still a research direction, not a shipped feature that "evolves on its own." The hard part is not only how fast you can train. It is how you get a label you can trust. You cannot take the tool choice the model just made, treat that as the right answer, and train it on itself.

The further I go, the more "doing nothing" looks like a skill

This round changed how I judge a local AI assistant. Speech recognition can be fast, and the larger model can answer smoothly, and if the tool judgment is unreliable it is still only a demo. For someone to keep using it, the system has to tell ordinary chat, a follow-up, and a command apart. When a request is unclear, I would rather it ask than guess and go do the thing.

I will keep working on Rex's training, on keeping context state separate, and on LoRA updates after deploy. What I most want to see is whether a small model, on a person's own device, can get a little more accurate as confirmed corrections pile up. Once those comparison runs are finished, that can be its own essay.

---

Source and references

AIHub core: github.com/zhiwehu/aihub

Web client: github.com/zhiwehu/aihub-client-web

ESP32 Miaoban firmware: github.com/zhiwehu/aihub-firmware

Command-line client: github.com/zhiwehu/aihub-client-cli

Rex routing: rex_planner.py

Tool calls: tool_chat.py

Context checks and dynamic tool routing: extension_routing.py

Rex v5 training set and test notes: router-everyday-v5

Docker deploy notes: deploy/docker/README.md

Previous essay: From an ESP32 to a Local VoiceHub on a Mac: A Pipeline That Can Speak