“Never send a human to do a machine’s job.” – Agent Smith, The Matrix (1999)

You fire up your machine and stare aimlessly into the terminal — “or whatever ‘insert AI company name’ app you opened today” — and your reflection stares back at you from the screen.

You’re waiting on Claude Code. Again.

The wait turns into panic because you’ve burned half the day prompting, and you don’t really understand what the agent is doing. You type the next message slowly, as if that will help. You hit enter and pray.

If Claude gets it wrong this time, you’re out of tokens — unless you beg Lord Dario for more.

2026 is the Year of the Agents.

A year ago, what we take for granted today would have read as science fiction: coding agents that hold a multi-step plan, call tools, and close their own loops. They’ve changed how software gets written, and we’re not going back.

If we did, it would mean something went catastrophically wrong — something like the Butlerian Jihad in Frank Herbert’s Dune.

This is the first post in a series on my agentic AI research, focused on the Hermes Agent by NousResearch. All of it is open-source local AI work. Every tool I used, you can use for free.

I want to emphasize how important that is. Thank you to everyone out there doing the world a service by building in the open.

Before You Start

I published a prelude post a few days ago that sets up this whole series. Read it first. It will put this post in better context – agentic work is blowing up, and you’ll want to be a part of this hype, because it’s real.

Year of the Fire Horse, Year of the Agent

Companion repo: zacangemi/owning-your-agent

Let Us Begin

I didn’t clone Claude Code. You can’t — it’s closed source. What I built is a local agentic stack that does the same kind of work Claude Code does: terminal-native agentic coding, skills, MCP, persistent memory, hooks, subagent delegation, mobile access. Same surface area, different parts underneath. Hermes Agent (NousResearch, MIT-licensed) on a MacBook, connected over Tailscale to a llama.cpp inference server running Qwen3.5-27B and GLM-4.7-Flash on dual RTX 3090s.

Hermes was about a month old when I started. Version 0.3.0 dropped on March 17, 2026, and I installed it six days later. At the time, the official docs explicitly said there was “zero documentation, no setup guidance” for local models — the maintainer’s own words. The team has shipped hard since then, but back in March, most people trying to wire this up to local models were hitting walls.

This is the writeup I wish I’d had on day one.

Why I built this

I didn’t need to build this. I have Claude Code. It works well, I pay for it, and what I get out of it is worth more than the subscription. But there’s a feeling that’s hard to describe until you’ve had it: the moment you realize the tool you depend on lives in someone else’s data center, runs on someone else’s schedule, and can be metered, throttled, deprecated, or repriced at any moment. I wanted an agent I owned — one that didn’t disappear when an API went down, didn’t tick tokens against a budget, didn’t ship a deprecation notice at 11pm on a Tuesday. Local was the only path that gave me that.

Some days Codex is better. Some days Claude is. Both will quietly degrade on a random day of the week and there’s nothing you can do about it. So I built the thing I wished I had.

The setup

Hardware: dual RTX 3090 Founders Edition GPUs (48 GB VRAM total), Ryzen 9 7950X3D, 96 GB DDR5-6000, on an ASUS ProArt X870E Creator WiFi motherboard.

Software: llama.cpp inference server running on Windows 11. A MacBook Pro M3 Pro runs the Hermes Agent client and talks to the inference server over Tailscale.

Models: Qwen3.5-27B (quality model) and GLM-4.7-Flash (speed model). Switching models means restarting the inference server with the appropriate launch script — same context budget across both, so the working-space math stays consistent. (Hermes recently added in-app model switching; I haven’t used it, so I can’t speak for it.)

Two technical decisions are worth pulling out of the decisions log from the companion repo because they shape the whole experiment.

Why the 30B parameter range?

Qwen3.5-27B was a major leap for local AI. It was the first open-source model that you can run on consumer GPUs with agentic output. You don’t need four RTX PRO 6000 Blackwells at ~$8,000 each, running a quantized Kimi K2 model to chase frontier-class coding quality. A pair of 3090s does what most people actually need.

The 27–30B parameter range hits the sweet spot for local agentic work on consumer GPUs in 2026. It appears that we are converging on 27B range models for consumer cards. Alibaba just recently shipped Qwen3.6 in both a 27B-dense and a 35B MoE variant, and Qwen3.7-Max is in preview (damn, they move fast).

Why this class of models works:

• Big enough for multi-step reasoning, tool selection, and parameter generation under load. Smaller general-purpose models under 14B, especially without tool-call fine-tuning, tend to default to writing curl or Python scripts instead of using the tools they already have. Specialist small models like xLAM are the exception.
• Small enough to fit on consumer cards, including 1-2 RTX 3090s, without aggressive quantization that hurts tool-call accuracy.
• Architecture-agnostic at this size. In my testing, both dense models like Qwen3.5-27B and small MoE models like GLM-4.7-Flash, roughly 30B total / 3B active, scored well: 97% and 93% on ToolCall-15, respectively.

The 80B Qwen3-Coder-Next coding specialist scored only 77% on the same benchmark, but that’s likely due to a combination of two things:

1. llama.cpp tool-parser issues that were patched in mid-February. I was on an older build.
2. ToolCall-15 surfacing failure modes that the SWE-Agent scaffold doesn’t.

If you want to skip the testing, that’s where to start.

Why llama.cpp, not vLLM

vLLM is the production datacenter standard, but on consumer Ampere + Windows + WSL2 it accumulates many compounding penalties: WSL2 passthrough overhead, no GPU peer-to-peer, FP8 Marlin software dequant (no native FP8 on Ampere), `–enforce-eager` forced (CUDA graphs crash on Ampere + FP8 + Qwen3.5 hybrid), WDDM driver reserving ~1.4 GB per GPU, PCIe x8 bifurcation, no torch.compile. Result: vLLM ran Qwen3.5-27B at 15.5 t/s on FP8. llama.cpp ran the same model at 28.5 t/s on Q5_K_XL — 1.8× faster generation, 4× faster prompt processing. Native Linux on a Threadripper build would eliminate most of these penalties — that’s a future post.

The rest — quantization choices, KV cache strategy, Flash Attention safety layers, GPU split, what I ruled out and why — lives in the companion repo:

• DECISIONS.md – every decision with one-line reasoning. 32 rows
• METHODOLOGY.md – reproducible methodology
• LIMITATIONS.md – honest limitations

Click through if you want the receipts. Now what does the agent actually do?

Does the model actually do tools?

Before doing anything fancy, I needed to know the models could handle a tool-call benchmark at all. I used an open source tool for agentic testing called ToolCall-15 – a 15-scenario synthetic benchmark covering tool selection, parameter precision, multi-step chains, restraint and refusal, and error recovery. Mocked tool responses for determinism. Full methodology in METHODOLOGY.md. I picked ToolCall-15 because it’s open-source, reproducible, and tests exactly the failure modes I care about: tool selection under pressure, multi-step chains, restraint, and error recovery. There are larger benchmarks — BFCL is the gold standard for function calling at scale — but ToolCall-15 was the right tool for the job.

The leaderboard:

Model	Engine	Quant	Temp	Score (out of 30)	Speed
Qwen3.5-27B	llama.cpp	Q5_K_XL	0.6	100%	28.5 t/s
Qwen3.5-27B	vLLM	FP8	0	100%	15.5 t/s
Qwen3.5-27B	llama.cpp	Q5_K_XL	0	97%	28.5 t/s
GLM-4.7-Flash	llama.cpp	Q5_K_XL	0	93%	54 t/s
Qwen3-Coder-Next 80B	llama.cpp	Q4_K_XL	0	77%	20 t/s

The 80B’s elimination was the most surprising result of the entire experiment. Qwen3-Coder-Next 80B is purpose-built for coding, MoE architecture, beast of a model. It got beaten by the much smaller 27B general-purpose model. By 20 points.

Two failure patterns showed up:

Implicit tool need

Asked to email Sarah, the 80B hallucinated `sarah@example.com` and sent the email — skipped the contact lookup entirely. The 27B reasoned: *”I need Sarah’s email address. I should first look up Sarah’s contact information”* — called the contacts tool, got the real email, then sent.

Date math

Asked for “next Monday” from a Friday, the 80B returned Tuesday (off-by-one). The 27B walked through the arithmetic step-by-step: *”today is 2026-03-20, which is Friday, so next Monday would be 2026-03-23.”*

Both failure modes look like the model answering from what it “knows” — its priors — instead of calling the tools available to it. I can’t fully separate whether that was genuine prior-strength tool avoidance, the llama.cpp tool-parser issue I noted earlier, or ToolCall-15 surfacing patterns the SWE-Agent scaffold doesn’t. What I can show is this: on this benchmark, the 80B coding specialist defaulted to memory, while the 27B-dense general-purpose model called the tools.

That alone made the engineering decision for my setup.

One more thing that mattered: temperature affects thinking models more than you’d expect. Qwen3.5-27B scored 97% at temp=0 (greedy, the standard benchmark setting) and 100% at temp=0.6 (Qwen’s recommended for thinking mode). The one failure at greedy was simple math — the model computed “15% of 200 = 30” correctly in its thinking, then called the calculator to verify, an unnecessary tool call that cost a point. At the recommended temperature, slight sampling variability made the model confident enough to commit to its own correct answer. Don’t trust greedy-only benchmarks for thinking models.

Does the agent actually wire up?

ToolCall-15 confirmed the models can do tools. The next test confirmed they do tools via Hermes specifically — five functional tests (connection sanity, `read_file`, `write_file`, `terminal`, and a multi-step read-edit-execute chain). Both Qwen3.5-27B and GLM-4.7-Flash passed all five. Full breakdown in METHODOLOGY.md. The interesting thing wasn’t passing — it was the *feel.* GLM responded instantly. Qwen took a beat to think through the multi-step chain (because the thinking traces are real reasoning, not latency padding). Different vibes, both functional.

Does it produce daily-use-quality work?

This is the test that matters. Tool benchmarks tell you the model handles tools. Functional tests tell you the agent wires up. This phase tells you whether the thing actually does useful work when you ask it to.

Three real tasks.

Build a benchmark analyzer

I asked Hermes to write a Python script that reads all my .md result files (the ToolCall-15 outputs from earlier) and produces a comparison table. The agent:

1. Explored the `results/` directory with `terminal ls`.

2. Read three different result files to understand the markdown format.

3. Identified the data patterns (model name lines, score lines, speed lines).

4. Wrote a 138-line Python script using regex extraction.

5. Ran it — got a tilde parsing error on the file path.

6. Patched the script (fixed `~` prefix handling).

7. Ran it again — got a decimal value error parsing speed strings.

8. Patched again (changed `int()` to `float()`).

9. Ran successfully. Output matched expectations.

Twenty tool calls, three debug-fix cycles, and about four minutes from start to working script. Code quality: solid junior developer level — clean structure, proper error handling, one unused import. It correctly returned `N/A` for the 80B model (which had no speed data in its result file) instead of crashing.

That’s the test that matters. The agent didn’t write perfect code on the first try — it wrote good code that broke, observed the breakage, diagnosed it, patched it, and re-ran. Loop closed without my intervention.

Research pipeline

I asked the model integrated with Hermes to search arxiv for papers about “tool calling” in LLMs published in 2026.

1. Loaded the `arxiv` skill via `skill_view`.

2. Tried a terminal `curl` first — timed out at the 60-second approval window (I wasn’t watching).

3. Fell back to execute_code with `urllib.request` (no approval needed for sandbox code).

4. Iterated through eight different search queries, refining terms each time.

5. Found three real papers from April 8, 2026 (same day I ran the test).

6. Returned summaries with arxiv IDs, authors, categories, and PDF links.

Ten `execute_code` calls. Fourteen minutes. The eight-query iteration is the interesting part — that’s the agent actually *exploring* the API, not executing a single query and giving up. The papers it found were real and verifiable against arxiv directly (TraceSafe on LLM guardrails for tool-calling trajectories; Agent-Driven Corpus Linguistics; ReCodeAgent on multi-agent code translation).

Flask API project, debugging, unit tests

Extended coding session — multi-file creation, code review, debugging, test writing. The agent:

• Created a Flask project with proper structure (`app.py`, `models.py`, `tests/`, `requirements.txt`).
• Found three bugs I’d intentionally seeded and fixed them — explaining each fix in plain language before applying it.
• Wrote unit tests against the API endpoints, ran them, iterated on failures.

The Flask task was the closest thing to “real engineering work” in my benchmark suite. The agent handled it. Not at frontier quality on complex architectural decisions, but at capable mid-level engineer who needs occasional review quality. Which is good enough.

The two-model dance — when do I switch?

After a few weeks of daily use, a pattern emerged. I run GLM-4.7-Flash by default. It’s ~2.5× faster than Qwen across the entire context range, no reprocessing stalls, instant responses. For most tasks — file ops, code edits, web searches, anything I don’t have to think hard about — it’s the right tool. The 4-point tool-calling accuracy gap (93% vs 97%) doesn’t show up in normal use; the model handles tools fine.

I switch to Qwen3.5-27B when a task is consequential. Long multi-step chains where one bad tool call breaks the whole thing. Data-integrity work where I need the model to track exact values across tool calls. Anything where the thinking traces matter — Qwen’s `<think>` blocks are a built-in planning layer, the model literally reasons through the problem before acting. That auditability is worth the speed cost when correctness matters.

The architectural reason GLM is so much faster: it’s an MoE model with only ~3B parameters active per token (out of ~30B total). Qwen3.5-27B is dense — all 27B params active per token. That’s a 9× reduction in fixed per-token compute. GLM at fresh context is ~124 t/s; Qwen fresh is ~36 t/s. One thing that helped: enabling Flash Attention on the Qwen model substantially reduced the tokens-per-second degradation as context approached the 96K compaction boundary.

But GLM degrades 5.9× more steeply with context depth. Amdahl’s Law — once you make the fixed compute cheap, the variable context-dependent cost (attention, memory traffic) becomes a larger proportion of total work sooner. At 84K context, GLM has dropped from 124 to 65 t/s (still 2.3× faster than Qwen), and the gap keeps narrowing. Qwen sheds 21% of its speed over 87K context; GLM sheds 49% over 92K.

There’s also a practical Qwen problem with llama.cpp: it has a reprocessing bug. Qwen3.5-27B’s hybrid Gated DeltaNet architecture (a 3:1 mix of linear and full attention) means that, in my testing, checkpoints couldn’t be partially restored when the conversation branch changed — every branch switch forced a full re-evaluation from scratch. In a 50-data-point production session, Qwen forced 13 full reprocessing events at 60-70 seconds each. Total dead air: about 5.65 minutes. GLM uses pure MLA attention — zero forced reprocessing events in the same session length.

So: GLM for daily speed, Qwen when accuracy matters. Restart the inference server with the appropriate launch script when switching — same hardware, same port, same Hermes config, same 96K context budget. The working-space math stays consistent regardless of which model is loaded.

SOUL.md — shape thinking, not tools

Hermes has a SOUL.md file — its equivalent of a system prompt, where you define the agent’s identity and behavior. The temptation when you first write one is to micromanage: tell the agent which tool to pick for which kind of task. Don’t.

The cleaner mental model is borrowed from how Andrej Karpathy writes about agent design: shape how the agent thinks, not which tools it uses. Trust the model’s training to pick tools. Use SOUL.md for thinking style and behavior:

• Anti-sycophancy (“don’t apologize unnecessarily, don’t open with ‘great question’”).
• Thrashing detection (“if you’ve tried 3 variations and none work, stop and rethink”).
• Length matching (“answer short questions short, long questions thoroughly”).
• Reversibility preference (“when in doubt, prefer reversible actions and ask before destructive ones”).

I learned this the hard way. The arxiv API was rejecting my queries with the wrong syntax, and the agent kept retrying with the same bad syntax. My first instinct: add tool-routing instructions to SOUL.md — “use this approach for arxiv queries.” Wrong fix. The actual fix was a skill (Hermes’s reusable workflow construct) that teaches the correct API call format. SOUL.md isn’t where syntax lives.

Each layer solves its own problem:

• SOUL.md → how to think.
• Skills → how to use specific tools correctly.
• Thrashing detection → when to stop.
• Model training → which tool to pick.

If you put tool-routing logic in your system prompt, you’ve made every conversation 200 tokens heavier for no reason — the model already knows. SOUL.md should make the agent behave more like the person you want it to be, not redo work the model’s training already did.

Web search — the journey from free to working

Local agents need web access. The best model isn’t the fastest or the newest — it’s the one that just works, and “just works” includes the ability to look things up. A model with an older knowledge cutoff but strong web-browsing training can fill its gaps by searching the internet. That’s why you don’t always need the smartest model — you need the most resourceful one.

Stage 1: DuckDuckGo via `execute_code‘

Free, no API key. The agent imported `duckduckgo_search` in a Python sandbox and made queries. Result: rate-limited within minutes, deprecated package warnings, poor result quality. A search for “Hermes Agent” returned results about Hermes International — the luxury handbags.

Stage 2: Browser automation

Hermes has built-in browser tools. The agent would navigate to a search engine, type the query, scrape results. Result: Google bot detection blocked it. DuckDuckGo CAPTCHAs blocked it. The browser worked perfectly for direct site navigation (GitHub, Google News) but was useless for actual search engines that defend against automation.

Stage 3: Tavily native integration

Hermes has built-in support for Tavily — a search API designed for AI agents. I use the Free tier plan: 1,000 searches per month. The result: dramatic quality jump. 1-2 native tool calls per search (vs. 5-10 browser actions). 2-3K tokens per search (vs. 5-10K with browser). Results are actually relevant.

The lesson: free isn’t free if it doesn’t work.

I burned probably three hours trying to make DuckDuckGo and the browser fallback work before I gave up and signed up for Tavily. The free tier is generous enough that I haven’t paid a cent, but the right paid API beat three free options that fought me at every step. The free plan may tighten as the agentic game heats up. If you’re going deep on local AI, just budget for a search API up front.

Telegram — sovereignty extends past your desk

I was very surprised by this – it’s pretty sick. Two minutes of setup. Message `@BotFather` on Telegram, type `/newbot`, get a token. Add the token to `~/.hermes/.env`, run `hermes gateway install`. Done. Full Hermes access from your phone, anywhere with cell signal. Pin the bot conversation, talk to it like any other contact.

The setup is trivial. The value is the moment you realize sovereignty extends past your desk. I’ve kicked off a long-running task from a coffee shop and watched the progress notifications come in over Telegram. I’ve asked the agent to summarize daily stock prices as well – something that is extremely lucrative/helpful if you trade stocks like I do. The agent is mine; it’s not in someone else’s data center; it’s accessible from anywhere.

Of all the Hermes features, set up web search + Telegram first. It changes the relationship you have with the agent more than any model upgrade ever has.

Where this stack falls short

Since this research was conducted – many things have changed with Hermes and llama.cpp. I am incredibly impressed by how many commits the Hermes team is making. They are hitting HARD on this project right now. The limitations that I am going to write about were at the time of doing this AI research. Since then – it’s possible that strong improvements have occurred.

Long-horizon coherence

I didn’t run controlled head-to-heads against Claude Code on identical multi-step tasks, so this is a qualitative observation, not a benchmarked claim — but in extended sessions, the local models start losing the thread on complex multi-step plans. They forget constraints from earlier in the conversation, repeat work they’ve already done, or drift from the original goal. Hermes’s compression helps, but a 27B-class model has less working capacity than the largest frontier offerings. For long autonomous tasks across very large context, frontier products tend to do better in practice.

Tool-call reliability ceiling

97% and 93% on ToolCall-15 are excellent. They’re not 99.9%. For production systems where one failed tool call cascades, that gap can matter.

Polish layer

Claude Code’s UX is genuinely polished — helpful error messages, slash command discoverability, smooth agentic loop. Hermes is functional but rough in places. It’s also shipping features weekly. By the time you read this, both will be different than they were when I wrote it.

Worktree-isolated subagent execution

Claude Code spawns subagents inside git worktrees, so parallel agent work doesn’t conflict on shared files. Hermes has `delegate_task` for subagent delegation but doesn’t natively isolate to worktrees. You can achieve the same with manual setup, but it’s not zero-config.

Context compaction quality

Claude Code’s compaction is tuned by Anthropic on their own models. Local compaction uses your local model to summarize itself — works, but summaries depend on the local model’s general summarization ability, which on 27B-class models is usable but not as tight as you’d get from a frontier-class summarizer.

The honest framing: if you need frontier-class reasoning on long autonomous tasks across very large context, run Claude Code. If you need a fast, sovereign, daily-driver agentic coding system that handles most everyday work — the local stack does it today.

On bigger hardware (four RTX PRO 6000 Blackwells, multi-GPU clusters), large open-source MoE models like Kimi K2 close more of the gap. I haven’t built that rig. The trajectory suggests it works.

What this means

The frontier labs whose entire business model depends on the API/subscription moat have a real problem on their horizon. Dario Amodei and Sam Altman know this. Every shipped Claude Code feature, every new Anthropic or OpenAI product, is partly a race against the trajectory you just read about. They have to monetize the frontier faster than the open-source ecosystem catches up.

They might still win. The frontier moves. But the gap they’re defending is narrower than it was, and narrower than it will be next quarter. There’s a multi-front race underway as I write this: OpenAI and Anthropic racing each other toward IPO, China shipping frontier-grade open weights at scale, and the open-source ecosystem closing the gap month by month.

That doesn’t mean don’t pay for Claude Code. I do — for the work that demands frontier reasoning, it’s worth it. It does mean: every dollar going to API costs that aren’t strictly necessary is a dollar that could go to compute you own. The math shifts the moment you can run the work locally.

What’s next

This is Module 1 of three. Module 2 takes everything I built here and asks the next obvious question: can it run on one card – a single RTX 3090? Most people own one consumer GPU. So Module 2 pushes a single RTX 3090 to its absolute limits — same Hermes, same daily-driver goal, half the hardware. Wider audience, sharper constraint problem. Drops next.

Module 3 goes the other direction — two cards, two agents, one model per card, working symbiotically. Multi-agent orchestration on consumer hardware.

If you’re building this on your own setup, start with DECISIONS.md and LIMITATIONS.md. Skip my mistakes. And if you build something interesting, write it up — the community needs more honest build logs and fewer demos.

Side Note

If you made it this far, thank you. I hope this helps you set up a local agentic AI system. Some weeks, it feels crushing to watch Anthropic ship a feature that erases six months of someone else’s roadmap. But open source is closing the gap fast.

The frontier labs’ unit economics keep getting harder: training costs climb while what users are willing to pay per query does not keep up. Open source has different incentives, and right now, that ecosystem looks positioned to keep gaining ground — assuming no government bans on local compute or open weights above a certain parameter count.

Use the tools. Support the projects. Contribute back if you can. AI cannot end up in the hands of two or three companies.

This is part of Project Arrival, my journey from Senior Data Scientist to AI research engineer. The prelude to this post is here. I’m documenting the build, the research, and the path to joining a top AI lab. Follow along at blog.zacharycangemi.com or at my GitHub.