Nobody Is Waiting

There are two inference gateways in my house and the split between them is not the one you would guess. It is not big model here, small model there. Both boxes serve some of the same weights. The split is by whether a human is waiting.

One gateway answers browsers and chat. Somebody is sitting there watching, and time to first token is what they feel. The other serves crons, classifiers, agent loops, and batch jobs. Nothing on that side has a person attached to it. A request can take a minute or an hour and the result lands in a queue either way.

Writing that rule down changed how I think about the stack. It only became possible to write once the hardware stopped charging me by the token.

Per-Token Pricing Prices Your Attention

When inference is billed per call, every call is a decision. Not a large one. It happens hundreds of times a day anyway. Should I spend on this. Should I batch it, or use the cheaper model. Should I bother at all.

That decision has a cost that does not show up on the invoice. It is why automation that would clearly pay for itself still never gets built. The blocker is the justification, not the payback: somebody has to defend a recurring line item for a job nobody is asking for. The same instinct XKCD 1205 trained into everyone about automation effort reappears one layer down, applied to runtime.

A box you own has a fixed cost. It draws power sitting idle and it draws somewhat more working. Amortized over three years, the difference between a machine that runs nothing overnight and one that runs eight hours of jobs is small enough that I stopped modelling it. Which means the question inverts. It used to be “should I spend on this job.” Now it is “the machine is on anyway, what should it be doing.”

Those two questions produce different lists.

The Work That Does Not Need Now

Ask the question straight and a surprising share of what I run has no deadline inside the hour.

Mail is the clean example. I have a classifier triaging an inbox: read the message, decide whether it stays in the inbox, assign a label. On a hundred-message dry run it kept fourteen and archived eighty-six. The review afterward was mostly about one category being over-used as a catch-all for automated mail, which is a one-line prompt fix. The job runs on a four-billion-parameter model on the batch gateway. It could run at three in the morning, or at noon, and I would not be able to tell the difference by looking at my inbox at nine.

The same is true of most personal automation. Receipts. Calendar tidying. Anything that reconciles two systems that disagree. None of it is time sensitive in the way it feels when you are doing it by hand, because when you do it by hand the latency is you.

Then there is the long chain. Take an incoming issue report. Investigate it against the logs and the database, form a hypothesis, write a branch, run the tests, and leave a documented result behind. That is not a request. It is a shift. Supervising it interactively is unpleasant, because most of it is waiting. It suits a window where nobody is watching.

I should be precise about what runs today. The triage crons run. Overnight agent loops on my own repositories run, with a calls-per-hour ceiling and a per-invocation timeout, and they produce commits. The full chain from issue report through to a published knowledge base article does not run unattended end to end. Parts of it are wired, the distillation step runs and I describe it below, and the promotion to anything customer-facing has a human in it on purpose. The design is right. It is not finished, and I would rather say that than let a diagram imply otherwise.

Slow Is Free at 3am

Wendell at Level1Techs compares a pair of DGX Sparks with a system running four RTX Pro 6000s in this August video. It helped me think about slow local inference in terms of the work it can finish.

At 6:48, he says the smaller setup runs about seven times slower. He calls it “entirely usable for a couple of people that want to use things agentically.” Earlier, at 2:07, he puts the larger system at roughly seven times the speed and seven times the cost. Those are his results for those machines, not a benchmark for mine.

At 7:02, he distinguishes interactive chat from agentic work: a large task split into smaller tasks under a supervisor. In my setup, that distinction determines where a request runs. A slower local run can be preferable when it meets the quality requirement and finishes before I need the result. During a conversation, I pay for the delay in attention.

My ledger records the same trade-off. The local arm of my cost ledger takes roughly twice the wall clock of the cloud arm on the same case. Seven minutes against twenty seconds on one, thirty-one minutes against eighteen on another.

While I am sitting there, that gap is the experience. It is the difference between a tool and a chore.

At three in the morning it is not a number at all. Both arms finish before I wake up. Two figures still count in that comparison. The cost, which favors local by between thirty-six and ninety-six times. And the quality, which came out the same on the cases I ran.

This is why the latency split matters more than the model-size split. Sort by how big the model is and you put the good model where the money is and the weak model where the work is. Sort by whether anyone is waiting and a capable model can run on the batch lane for as long as it needs. A 321B model generating at ten tokens a second is a bad chat experience and a perfectly good overnight worker.

At 21:00, Wendell describes the capability gap between cloud and local models narrowing. At 24:56, he estimates payback in weeks under full utilization and the token prices in his comparison. That utilization assumption limits the comparison. My ledger gives a payback in months for a cheaper machine at a different workload volume.

His security-audit example at 26:17 also depends on a bounded task. He supplies steps, skills, checks, and instructions to flag anomalies. That connects to the process detail an agent needs. Before handing work off overnight, somebody has to describe it at the level of steps.

Humans Preempt Agents

Once agents run continuously on the same hardware you use interactively, they compete with you. They win, because they never stop asking.

The first version of this was maddening. I would open a chat window and wait, because a batch job had the model loaded and the queue full. The obvious fix, separate hardware for batch, is expensive and only moves the problem to whichever box is cheaper.

What I run instead is a preemption gate. The agent lane gets a header the interactive lanes do not. While a human has been active in the last three minutes, requests on that lane get a 429 and a retry-after, and the agent SDKs I use back off on their own. Three minutes after I stop, batch work resumes. Interactive requests never touch the gate at all.

Both details from building it generalize past my house.

It fails open twice. The activity signal is fire-and-forget over UDP, so if the gate is down the interactive path never notices. And the agent route lists the direct upstream as its second option, so a gate that stops answering becomes a dial failure and traffic proxies straight through. A preemption mechanism that can take down the thing it protects is worse than no preemption mechanism. That is the same reasoning as putting the model where being wrong is cheap, applied to a piece of infrastructure rather than to a model.

And presence is not activity. On day one the gate latched permanently within minutes, because an open but idle browser tab polls a version endpoint every few seconds. The gate read that as a human. The fix was a prefix ignore list, which is boring, but the lesson is not: any signal you use to mean “somebody is here” will eventually be produced by something that is not a person.

Routing Needs Its Own Configuration

The gateway split and the preemption gate started as routing decisions I made by hand, one consumer at a time. A model upgrade exposed five bindings that had drifted apart. Repointing some but not all of them made two consumers repeatedly evict each other’s weights from memory. I had to fix the routing across separate configuration files.

NVIDIA’s NeMo Switchyard helped me design a shared routing layer for the homelab. This is a separate source from the video. Switchyard routes requests to OpenAI-compatible endpoints; the models run behind those endpoints. The router itself needs no CUDA, so it can direct work to the two AMD machines I already own.

The homelab’s support-specialist setup documents two routes: one calls the specialist directly for measurement; the other can escalate to a larger model. That is a narrower deployment than migrating every consumer to a shared router. The direct gateways remain the fallback, and the design records the selected Switchyard release as experimental. Routing policy also needs its own evaluation: a classifier that costs more time than it saves can make a request slower.

The Level1Techs discussion on local AI and specialization is another source cited in that design. Repeated work supplies examples for training a smaller specialist, while evaluation determines whether it should receive traffic. My specialist experiments also produced models that improved on an offline evaluation and failed in live investigations. Training on the work is a way to test a candidate, not evidence that it should run unattended.

A shared route lets me change which model handles a job without editing every caller. The human-activity gate still has a separate job: keeping background work from delaying a person.

The Loop Needed the Idle Hours

I wrote recently that the loop is the cheap part, and that the hard problem is what survives between iterations. That argument has an unstated premise: that you can afford to run the loop enough times for the memory under it to matter.

Metered inference makes that expensive, and the pressure shows up in what you build. You write loops that terminate quickly, because each iteration is a line item. You skip the speculative pass that reads a hundred old sessions looking for a pattern nobody has named. It might return nothing, and it will definitely cost something.

Unmetered inference on hardware you own removes that pressure. The archive I keep of every agent session is a local postgres database. The distillation pass over it reads a completed investigation and extracts a structured observation. It is the kind of job that returns nothing some nights. That is fine. It ran while I was asleep on a machine that was drawing power anyway.

The pattern-finding work is the same. A long-running problem that nobody has had a free afternoon to characterize can get a hundred afternoons of machine time, spread across weeks. The marginal cost of the hundred and first is nothing. Some of those runs return a finding. Most do not. Metered, that ratio is a budget conversation. Unmetered, it is Tuesday.

The Overnight Window Needs More Discipline

Everything above argues for running more autonomous work, unattended, for longer stretches.

An eight-hour unsupervised loop is the ideal environment for a confidently wrong result to spread. The failure in a model that could not do the work produced a well-formatted answer anyway is bad enough when you read the output five minutes later. It is considerably worse when six subsequent steps built on it while you slept.

So the overnight window is not a place to relax the discipline. It is where the discipline has to already be in place: verification as a diff rather than a read, deterministic code wherever being wrong is permanent, and a restorable pre-image rather than a promise anywhere the loop can touch something it cannot put back. My knowledge base promotion step still has a person in it. The model can draft the article. Publication is the step where a wrong answer does not get reversed on its own.

Cheap compute lets you run more loops. It does not make a bad loop safe. If anything it raises the stakes on the safety layer, because the invoice used to be what limited blast radius.

The Arc So Far

The arc has been circling a single reframe: as building got cheap, the constraint moved to deciding and describing. Owning the hardware moved another constraint, and I did not see that one coming. Per-token pricing trains you to treat machine time as scarce. You keep designing for that scarcity after it stops applying to work nobody is waiting on.

The companion piece to this one is about what you own when the price of inference goes to nothing. It is the other half of the same answer. You own the clock. The box runs whether you use it or not, so the question left is what should be running at three in the morning.

Most of that answer is work I had been doing myself during the day.

If you have a queue of work that nobody is waiting on and it is still running during business hours, reach out at [email protected].