January 2026
I refactored 9 agents to 1 LLM call
Nine agents took 14 minutes on a network incident. I rebuilt it as six deterministic nodes and one LLM call — 40 seconds, auditable, and owned by the ops teams.
January 2026
Nine agents took 14 minutes on a network incident. I rebuilt it as six deterministic nodes and one LLM call — 40 seconds, auditable, and owned by the ops teams.
The first time I ran the system of the team I'd just joined, it took 14 minutes to complete.
Fourteen minutes. For a task that, done manually, took a senior ops engineer 30 minutes — logging into multiple systems, pulling data by hand, cross-referencing across tools, writing up the report. Thirty minutes that nobody could compress, because each step required a human with access to a different system. That was the baseline the AI system was supposed to beat.
I scrolled through the LangSmith trace. Nine agents. Each one spun up its own ReAct loop, called tools, reasoned about the results, passed state to the next via an in-memory store. There was a supervisor agent for the real-time checks. A supervisor agent for priority classification. Agents, agents, agents everywhere. I had already been through that multi-agent frenzy in a past project, and I knew the hell it could become.
The system was built by a smart team, under real pressure, to solve a real problem: network operations teams were manually classifying network incidents, making errors, routing them to the wrong teams. The AI approach made sense. Agents seemed like the natural fit — each step needed something to look at network data and decide what mattered.
The architecture reflected that instinct faithfully. An agent to validate the incident. An agent for real-time router checks, with a supervisor agent underneath it managing two sub-agents for reachability. A priority classification supervisor routing to three specialist agents, one per priority tier. An agent for the ticket comment. An agent for email. An agent to log results.
Each one received the full context it might need — which in practice meant dumping large payloads of network data into a prompt so the agent could extract one field, check one condition, return one boolean. The reasoning step was often two lines of logic wrapped in hundreds of tokens of context.
The irony is that agents were chosen in the name of flexibility. The thinking goes: if we hardcode a workflow, we're locked in forever. Agents can adapt. But a 9-agent system isn't flexible — it's fragile in nine places simultaneously. Nine prompt surfaces to maintain. Nine reasoning paths that can diverge. A combinatorial explosion of emergent interactions between them. When the ops team tried to adjust the classification logic, one prompt change in the priority agent would shift behavior three steps downstream in ways nobody could trace. They stopped trying.
A 6-node deterministic workflow with one LLM call is easier to change than a 9-agent system. That was the part that surprised people.
Research suggests 79% of multi-agent system failures come from specification and coordination issues, not infrastructure. The agents weren't broken. The architecture was asking them to do too much.
As I read through the code, I kept asking the same thing: does this step actually need to reason?
Incident validation: check a field value and return true/false. That's not reasoning, that's an if statement.
Real-time router status checks: call the SD-WAN management API, get structured data back. The agent was reasoning about data that came back clean and typed.
Priority classification: apply a set of rules. Rules that the ops team knew, had opinions about, wanted to own — but couldn't touch, because encoding them in agent prompts made them too unpredictable.
Ticket comment: yes. This is the one step where language actually matters. The output is read by a human. The quality of the summary affects how fast the ops team understands the incident and acts on it. This is where an LLM earns its place.
That's one step out of nine.
Six nodes. Five deterministic, one LLM call.
Topology, basic checks, and extended checks are API calls with typed, structured output — the kind of thing that should have been a function from the start. The email node sends a notification. The priority node runs Python.
That handles the agent bloat. The harder problem was priority classification — you can't just delete the agent and hardcode the rules, because the rules are business logic that ops teams need to own and change. Hardcoding them means every rule update goes through a developer. That's the same problem, different shape.
The solution was to move the LLM out of the hot path entirely. Ops teams write their classification rules in natural language — "flag as P1 if the primary and backup routers are both unreachable" — and an LLM converts those into validated Python once, at authoring time. The workflow executes the generated code at runtime. No LLM in the hot path. No unpredictability on a live incident. Code you can read, test, diff, and roll back.
This pattern has a name — Compiled AI — though I didn't know that when I was writing the refactor. LLM reasoning at authoring time, deterministic execution at runtime.
The supervisors are gone. The InMemoryStore is gone. The agent_to_tool_wrapper — which wrapped agents as tools so other agents could call them — is gone.
The uncomfortable sentence at the center of this design is: an LLM generates Python code that runs on live network data. So we spent more time on the guardrails than on the generator itself.
An ops engineer writes or updates a classification rule in natural language via a UI. The backend pulls the current prompt from LangSmith along with the previous version — the N-1 context matters, I'll get to that. It injects the actual Pydantic model source code as schema context, not JSON schemas. The real Python classes, including property methods and validators, so the LLM sees the exact same type contract the generated code will have to satisfy. The LLM generates at temperature 0.0. Then three validation stages run before anything touches the codebase.
Stage one is a Python AST parse — basic syntax. Stage two checks structure: expected function signature, docstring present, required imports. Stage three is the one that matters for security: an import allowlist enforced via AST walk. The generated code can only import from a fixed set of modules — typing, datetime, re, the workflow's own data models, and a handful of stdlib utilities. If the generated code tried to import os and call subprocess.run, stage three catches it at the AST walk before the file is ever written to disk. exec, eval, subprocess, socket — all blocked explicitly. A separate input sanitizer upstream catches the prompt injection attempt that would have produced that code in the first place, with 16 pattern detections for instruction overrides and role-switching attempts. Neither alone is sufficient.
And even if something slipped through all of that — the runtime environment runs with no outbound network access and minimal OS permissions. The attack surface for LLMs is effectively infinite; you can't close it entirely at the prompt level. So you assume breach at the LLM boundary and make the blast radius as small as possible at the execution boundary. The generated function processes typed structs and returns a priority classification. That's all it can do.
If all three stages pass, the file is written to workflow/rules/ with a header stamping the model name, temperature, and prompt commit hash — so every generated rule is fully traceable back to the exact LangSmith prompt version that produced it. If a classification starts behaving unexpectedly three weeks from now, you look at the file header and know exactly which prompt commit to audit.
The N-1 context: when an ops team edits an existing rule rather than creating one from scratch, the system sends the LLM three things — the previous prompt version, the current prompt version, and the existing generated code. The instruction is to compare the two prompts, identify what changed, and make only the targeted modifications. Not a full rewrite. Rule updates arrive as readable diffs on the GitLab review, not full file replacements. A reviewer can see exactly what business logic shifted and why.
From there, two paths to live system. The first is hot-reload on the running server: the workflow imports rule modules dynamically at invocation time, so when a file is updated on disk, the next incident picks it up immediately — no restart, no redeploy. This is for ops teams doing trial and error: write a rule, run it against a real incident, see what the classification produces, adjust, regenerate, try again. The second path is the GitLab MR: once the ops team is happy, the generated file goes through a branch, a code review, and a merge into the main codebase. This is how the rule survives deployments and becomes canonical. Ops owns the rules in natural language. We own the code that runs in production. The MR is where those two ownership boundaries meet.
The whole thing — from ops writing a rule to it being permanent — goes through LLM generation, three validation passes, live trial-and-error on the server, a human code review, and a merge. That felt like the right amount of guardrails for code running on a live network.
40 seconds end-to-end. Down from 14 minutes of broken AI, and down from 30 minutes of irreducible human work.
Cheaper, because nine multi-step ReAct loops became one LLM call per incident. Maintainable, because deterministic code has tests — you can break a change before it reaches production, which you cannot do with a prompt. And for the first time, the team running the network could actually read and edit the rules driving the classification. The rules were theirs.
Something I didn't anticipate: the old system used a self-hosted model everywhere, because client data couldn't leave the network at runtime. With the new architecture, the build-time step runs in a controlled authoring environment — so we could use a stronger frontier model there, where the data constraint doesn't apply. Better model for the part that needs it. Constrained model for the part that runs live.
We ran this on real incidents. Not a demo, not a simulation — actual data from the network operations team. It wasn't production at scale, and I'm not claiming it was. But it worked on real inputs, in 40 seconds, and ops teams could own what it did. That was the first time any of that was true.
The architecture works for one team, one customer, one set of SD-WAN rules. The harder problem is what happens when you run this across multiple operations teams, each handling different customers with their own classification logic.
The rules aren't universal. A network incident that's P1 for one enterprise customer might be P3 for another, depending on their SLA, their redundancy setup, their tolerance for disruption. The build-time approach handles this well for a single client — you generate rules once and they run deterministically. Across multiple clients, you need separate rule sets, isolation between them, deployment without cross-contamination, and per-team visibility into only their own rules.
That's the problem we're working on now. It's less of an AI problem and more of a software architecture problem — multi-tenancy, rule versioning, deployment isolation. The generated-code approach actually makes this easier than agents would have: rules are just files, files can be namespaced, namespacing is a solved problem. But "solved problem" and "solved for your specific production constraints" are different things.
The mental model is intuitive. You have a multi-step process that touches different systems, so you build multiple agents. The team that wrote the original system wasn't wrong to reach for them.
The question worth asking at every step is simpler than it sounds: does this step need to reason, or does it need to be correct? Reasoning — ambiguity, natural language, judgment under incomplete information — is what LLMs are for. Correctness — deterministic lookups, rule application, typed transformations — is what code is for. Anthropic put it plainly in Building Effective Agents: "find the simplest solution possible and only increase complexity when needed."
The most expensive thing in the old system wasn't the token cost. It was the maintenance surface — nine agents, each with their own prompt, their own failure mode, their own way of surprising you when the classification starts drifting. That's not flexibility. That's nine things to keep alive.
The system I handed back ran in 40 seconds. Ops teams could own the rules. And somewhere between those two facts is the actual lesson: the right amount of AI is less than you think, placed more carefully than you'd expect.