Cheap Is Not the Same as Yours

For one week in August, one of the better models available to anyone was free, with rate limits described as near-unlimited, on infrastructure sized for a hundred trillion tokens a day. Cheap would have been a discount. This was free.

Six days later the stealth model got its name.

A 1M-token context window. Native multimodality. An MIT license, which is about as permissive as software licensing gets. Any one of those would be news on its own. And it was trained and served on Chinese accelerators rather than Nvidia’s. That is the first credible sign in years that one company does not get to set the floor price of compute for everyone.

The API pricing landed at roughly a hundredth of what I pay for a frontier cloud model. That comparison is not fair to either side. I am going to make it anyway, because two orders of magnitude is not a margin. It is a different category of decision.

So the price of inference collapsed. That sets what a run costs. It does not say who decides what you may run.

Zero Retention Is a Policy

The offer says zero data retention, and I have no reason to think it is dishonest. It is still a promise, made by the party that holds your data, evaluated by that same party, with no mechanism you control.

I have made this argument before about agents that promise to confirm before acting. It repeats here. A control inside the thing it constrains is a preference, not a boundary. The database enforces permissions; the query does not promise to behave.

Retention is the half you can see. Whatever the policy says about storage, your private repository crossed the wire. Your customer data, your unreleased code, your internal runbooks: they were serialized, sent to infrastructure in another jurisdiction, processed, and returned. The retention promise covers what happens after that transfer. The transfer is the exposure, and it already happened.

None of this is specific to any one provider. It applies to every hosted endpoint I use, including the ones I like and pay for. Routing layers that advertise zero retention are relaying to somebody, and the somebody has their own policy and their own government.

For a large amount of what I do, that is a fine trade and I make it daily. For the rest, the question is what the alternative costs, in numbers rather than in principle.

What the Ledger Says

I keep a cost ledger. Same case, run twice: once through a frontier cloud model in an agent harness, once through a local model on a box in my office. The tools, the data access, and the task were identical. Cloud tokens are priced at list. Local runs are charged for the electricity they burned plus the share of hardware life they consumed.

Three cases from a support operation I work with, all in August:

The quality column in that ledger reads “same” on all three. On the vendor auth case, both arms independently reached the same conclusion: an expired refresh token with no headless recovery path, re-link the only fix. The cloud arm needed two rounds of human correction to get its draft right. The local arm needed none.

A table like that is easy to misread, and I would rather do the debunking myself.

The local cost basis is energy plus amortized capital, and the wattage figure in it is still an estimate rather than a measurement at the plug. It is marked as such in the ledger and it will move when I measure it. The direction of the error is unlikely to close a 36x gap. But the number is not accurate to three decimal places yet, and I am not going to pretend it is.

The cloud figures are a floor, not a ceiling. On one case the usage panel reported eleven percent more cache read than the transcript accounts for, which means the process made calls the transcript does not show. Harvested cloud cost undercounts.

Latency went the other way. The drafting case took the cloud model twenty seconds and the local model seven minutes. Those are not the same unit of work, and the ledger says so. The cloud session had been open since the previous evening, with the investigation already in its context, and ninety-one percent of its bill was cache. The local arm ran the investigation itself, from nothing, inside those seven minutes. Even correcting for that, local is slower. On the case where both ran the full job, the cloud model finished in eighteen minutes against thirty-one.

So: roughly twice the wall clock, for between a thirty-sixth and a ninety-sixth of the cost, at the same quality on these three cases. Three cases is not a study. It is enough to change how I decide.

The payback arithmetic is not subtle. The box is a two thousand dollar machine amortized over three years. A handful of cases a day at three dollars each covers it inside a few months. After that, the marginal cost of a run is a rounding error on the power bill. I am not claiming this generalizes to your workload. I am claiming that if you run a lot of agent work on a repetitive process, you should do the arithmetic once instead of assuming.

The License Was the Event

The free API week got the attention. The MIT release is what changed my situation.

A 321B-parameter model with 18B active fits, at a two-bit quantization, in about 101 GiB. A Strix Halo box with 128 GB of unified memory has roughly 125 GiB the OS can allocate, once you claw back the graphics carveout. Those two numbers are close enough that it works. Since the twenty-seventh of August it has been running in my office, serving the same endpoint my browser talks to.

Some numbers from that box, since that is where my estimates turned out wrong:

The KV cache costs about 27 KiB per token, measured. Deriving it from the attention architecture predicts twelve. If I had trusted the derivation I would have asked for a full million tokens of context and taken a global out-of-memory rather than a clean failure. Measured, the ceiling is 512k, which is what it runs at. Prefill is 134 tokens per second, generation is about ten. Cold load to first token through the gateway is forty-nine seconds.

Ten tokens per second is not a pleasant conversation. It is entirely fine for something that runs while I am asleep, which is the other half of this argument.

The quality of a two-bit quantization of a 321B mixture-of-experts model is also unvalidated on my box. Not “probably fine.” Unvalidated. It is a measurement I owe and have not taken, and anyone telling you a Q2 quant is equivalent to the hosted version is guessing.

A Gate Has an Owner

Anthropic published a position on open-weight models this year. Read it rather than the reaction to it, because the reaction is not what the document says.

It does not propose a ban. It says so directly: Anthropic has never advocated for a ban on open-weight models, and describes capable open-weight models without dangerous capabilities as a public good. It argues that protectionist bans would not address the national security concerns it lists. What it proposes instead is narrow: restrict advanced chip exports, and crack down on industrial-scale distillation that routes around those restrictions. The third item asks for safety testing of sufficiently capable models, whether they ship open or closed.

I think that is a more defensible position than the one people are arguing against, and I do not have a strong objection to most of it. The concern it raises about open weights is concrete: guardrails and usage monitoring are harder to apply to a file somebody already downloaded.

The last clause is the hard one. A testing bar gated on “sufficiently capable” is a gate, and every gate gets an owner. Nobody has to conspire for that. Regulation is a gate, and even reasonable regulation has this property: the terms of access get set somewhere other than where you are standing.

You do not need a ban for that to matter. A hosted model can be deprecated, repriced, rate-limited, restricted by jurisdiction, or retired on a schedule that has nothing to do with your project. All of those have already happened to somebody this year. The free week ends. The endpoint returns a different model under the same name. The pricing page changes.

Weights on a disk in your house have none of those failure modes. Whatever GLM-5.3-Flash is today, the copy on my machine will still be that in three years, at the same speed, for the cost of the electricity. That is what the MIT release delivered, and it outlasts a week of free API access.

This Is Not for Everyone

I want to spell out the toll, the way I did with building physical things.

Getting a 321B model onto a consumer box is not a download. It is a graphics memory carveout dropped to its floor in the BIOS, and a kernel command-line page limit raised to match. Both have to be written in two places, because a kernel upgrade reverts the one you forgot without warning. It is a forked inference engine pinned to a specific draft commit, because the quantizer is still moving and an unpinned pull will mix shards from two different runs. And it is reading architecture metadata out of a model file before the first load, because that decides whether the thing boots or takes the machine down.

I enjoy that. It should not be a requirement. For most workloads a hosted endpoint does the job, and I keep using one.

The calculation has changed. “Frontier quality in the cloud versus a toy at home” is no longer the choice on offer. On my own work, measured, it was the same answer for a fraction of the cost, with the tradeoff paid in latency rather than in quality. That is a trade you can make deliberately, per workload, instead of assuming it away.

One Layer Lower

That instinct runs through the pieces before this one: put what you depend on where you control it. Graduate a workflow off the model runtime once its structure is clear. Give the model a vote and not a veto where being wrong is permanent. Build the restorable pre-image rather than trusting a promise.

The model is the layer below all of that. I am not arguing that hosted inference is bad, and I have no plan to outrun a frontier lab from my office. The case is narrower. A capability you rent is a capability someone else can change the terms on, and the ones I have built a process around should not be in that category.

Inference getting cheaper is good news. Prices can go back up. A hosted model can be retired. A copy already on a disk in your office stays as it is.

If you are weighing local inference against hosted for a workload that runs constantly, reach out at [email protected].