Back to Insights
Insights

AI Models Only Get Six Weeks at the Top. Profit Takes Longer.

The Economics of Serving Open-Weight Models

Co-authored by Jiarong Xing and UpScaleX.
For more insights and further collaborations on this topic, contact jxing@fluxion-sys.ai and zidi@upscalex.ai
Glowing glass server blocks along an upward-trending line on a dark surface
Published on 11/06/2026

Jiarong Xing is a co-founder of Fluxion AI, a co-author of the open-source project kvcached, and an assistant professor of computer science at Rice University.

None of the ten most-used models on OpenRouter in the first week of April was still in the top ten in October. Not one.

The money keeps coming anyway. On October 14, Modal raised $750 million at a $15.75 billion valuation, saying it had passed $1 billion in annualized run rate. The most-used model on OpenRouter that week, DeepSeek V4.1 Flash, was five weeks old.

The inference business in 2026 runs on a contradiction. Capital is pouring in as if it were a utility. Demand behaves as if it were fashion.

The simplest way we have found to describe it: serving AI is a manufacturing business whose product line is replaced every few months. Every new model is a new product, and every new product needs its own assembly line.

And the line does not run for long. Hugging Face reports that developers engage with a given open model for about six weeks on average. A model’s commercial life is now measured in weeks. Most of its tokens are served early, exactly when serving it is least optimized.

If a model only gets about six weeks at the top, what should we be watching? Not the leaderboard, and not the price board. Watch how many tokens a provider gets out of each GPU-hour, which we call yield, and how many days it takes to get there after a new model ships. Call it time-to-yield.

Scale and GPU access still set the floor on cost. Retooling speed decides how much margin sits above it.

Jiarong has spent years working on how models get served. This piece looks at the economics of open-weight serving from the engine room, and at why the public price board tells only half the story.

Where the margin comes from: yield

Everyone rents GPU-hours at roughly the same price. Everyone sells tokens at roughly the same price. What differs is how many tokens come out of each GPU-hour at the speed customers expect. That is yield.

Cost per million tokens equals the hourly price of a GPU divided by the millions of tokens it produces in that hour. Small differences in yield become the difference between profit and loss.

Take one GPU rented at $5 an hour, selling output tokens at a market price of $0.60 per million. Hold the model and the per-user speed fixed. The only variable is yield:

Same GPU, same modelTokens/s per GPUCost per million output tokens
Day-0 support (hypothetical)1,000$1.39
Break-even at a $0.60 price~2,300$0.60
After a 2.7x software gain2,700$0.51
Fully tuned for the model (hypothetical)4,000$0.35

Illustrative. Throughput values are hypothetical; the 2.7x step matches the gain NVIDIA reported on GB300 within six months. Ignores input-token revenue and caching.

Nothing about the hardware changes between those rows. The same GPU goes from losing money on every token to earning a healthy margin, purely through software tuned to the model.

Gains of that size are real. NVIDIA reported that its GB300 systems delivered 2.7x more throughput and 60% lower cost per token within six months, on the same hardware. When vLLM topped a public speed leaderboard on several new models this year, the gains came from model-specific work: merging GPU operations to cut per-layer kernel launches from about 33 toward 10, and fixing a case where roughly half of text-generation time went to cross-GPU communication that had not been merged with the surrounding computation.

Here is the cruel part. Yield is lowest exactly when demand is highest. A new model launches with basic support, gets tuned over the following weeks, and by the time it runs well, traffic has moved to the next release.

Why is the window so short? Three shifts, and one caution about the numbers everyone watches.

Read the leaderboard carefully

OpenRouter, the largest public marketplace for model APIs, is where much of the industry watches demand. By its own count, weekly volume rose from 0.5 trillion tokens in mid-January 2025 to 128.4 trillion in mid-September 2026, roughly 250x in 20 months.

Volume is not revenue, though. By September, open-weight models accounted for more than 60% of OpenRouter tokens but less than 30% of spend. On Vercel’s AI Gateway, open-weight models crossed half of all tokens in August, at 56%, while capturing 14 cents of every dollar.

Some of the volume is not commercial at all. Free endpoints sit in the weekly top ten. For one week in early October, the #1 model was Space Bunny Alpha, an anonymous “stealth” model served free while its developer tested it before launch. It was later revealed as MiniMax M3.1 Flash.

The number worth planning around is margin per GPU-hour, and it appears on no leaderboard.

Shift 1: The token is a commodity with a public price board

Aggregators list every provider that serves a model side by side, with price, latency and throughput. For a buyer, that is a gift. For a seller, it means the product is identical and the price is public.

On one popular open model that most providers list at $0.30 per million input tokens and $1.20 per million output tokens, at least 11 providers already charge $0.60 or less for output. Half the going rate is now a normal price.

Costs are falling fast, and prices fall just as fast. Epoch AI estimates the cost of reaching a fixed benchmark score drops about 47% per quarter, roughly 13x a year. NVIDIA says its GB300 rack systems cut cost per token by up to 35x versus Hopper. The price board makes sure those gains flow straight to buyers.

On every model, the price is set by the most efficient provider willing to serve it.

Shift 2: The product line changes every month

Between October 2025 and mid-October 2026, at least 33 open-weight models with 100 billion parameters or more shipped from 14 labs. That is about 2.6 frontier-scale releases a month, ranging from 284 billion to 2.8 trillion parameters.

They are not variations on one design. In the same window, at least a dozen new internal designs for attention and memory reached production open models, and several labs shipped weights in new low-precision number formats. Each one needs new low-level GPU code, called kernels, before it runs fast.

Demand moves just as quickly. The #1 model on OpenRouter changed at least six times in nine months.

Period (2026)#1 model on OpenRouterLabOpen weights
Oct 5 to 14DeepSeek V4.1 FlashDeepSeekYes
Week ending Oct 4Space Bunny Alpha (free stealth period)MiniMax (revealed later)Unknown
Early AugustDeepSeek V4 FlashDeepSeekYes
JulyMiMo-V2.5XiaomiYes
Early JuneDeepSeek V4 FlashDeepSeekYes
Early AprilMiMo-V2-ProXiaomiNo
FebruaryMiniMax M2.5MiniMaxYes

Snapshots mix daily, weekly and monthly rankings from different trackers. They show turnover, not exact share.

New entrants scale within weeks. StepFun’s Step 5 Preview entered the weekly top ten at #2 less than a month after its API launch.

Shift 3: One-size-fits-all engines are paying a maintenance tax

Most providers run open-source serving engines, mainly vLLM and SGLang, that aim to serve every model on every chip. Generality was the right bet when models looked alike. Now every new design adds another code path.

vLLM’s Python codebase grew from about 343,000 lines to about 1.07 million in twelve months. Its model definitions grew only 1.5x over the same period; the rest is machinery for new attention types, chips and parallelism. In mid-October it had more than 5,000 open pull requests. Its maintainers wrote in July that testing “gets harder every day.”

SGLang shows the same pattern: its serving runtime roughly tripled in ten months and now carries code paths named after individual models. DeepSeek originally built its production engine on an early fork of vLLM, but has since open-sourced DeepInfer, a high-performance inference framework for DeepSeek models.

Generality has a cost. It is paid in days to support each new model, and in yield once support lands.

What gets eaten, what gets more valuable

Sort every advantage an inference business claims by one test: does it survive the next model release?

Eaten. Reselling popular open models at serverless prices. Announcing “we support model X,” since dozens of providers do within days. Competing on price, because the price board guarantees someone will match it.

More valuable. Speed from released weights to an efficient deployment. Serving customers’ own customized models, which no public price board can compare; Fireworks says more than 95% of its tokens come from them. Work beyond raw tokens: about $300 million of Modal’s roughly $1 billion run rate comes from sandboxes. And capacity that can follow demand from one model to the next.

Ambiguous, and probably worse for providers. Open-model licenses are adding a commercial gate for anyone who resells them. Under the Kimi K3 license, a model-as-a-service business whose revenue, together with its affiliates, exceeds $20 million over any 12 months must sign a separate agreement with Moonshot. The test counts all of the provider’s revenue, not just revenue from K3, so it catches almost every provider of meaningful size. What those agreements cost is starting to surface: Moonshot has reportedly asked the major clouds for up to 30% of K3-related revenue, and has already reached similar arrangements with smaller providers. Meta’s Muse Spark Open carries a similar clause. At commodity prices, a 30% revenue share can be larger than the entire gross margin. Many labs still ship under permissive licenses, so providers can route around the restrictive ones for now, but labs that control both the model and the license gain leverage over everyone serving it.

What a factory built for retooling looks like

The operators getting this right share four habits.

They specialize per model. DeepSeek’s in-house stack pairs a heavily customized engine with its own attention kernels, matrix-math library and expert-communication library. In early 2025 it reported about 14,800 output tokens per second per 8-GPU node and a theoretical cost-profit margin of 545% at its list prices, while noting actual revenue was substantially lower. SGLang’s later deployment of DeepSeek on GB200 racks, built on those components, reached 3.8x the prefill throughput and 4.8x the decode throughput of its earlier H100 setup.

They let AI write code, but let verification decide. In early 2025 an AI system reported a GPU kernel 150x faster than PyTorch. It was actually about 3x slower, because it had found a hole in the correctness check. A 2026 benchmark that closed such holes found the best frontier model’s kernels averaged 0.88x PyTorch’s speed. The same models do far better inside structure. Meta’s KernelFalcon, which executes and checks every candidate, produced correct kernels for all 250 KernelBench tasks. Meta’s KernelAgent, with hardware profiling in the loop, reached 89% of the H100’s theoretical limit on its tasks.

They make capacity follow demand. Instead of pinning GPUs to one model, they shift capacity as traffic moves between models and stages of serving. Mooncake, the serving platform behind Moonshot’s Kimi, splits serving into stages and pools their resources across the cluster, which let Kimi handle 75% more requests while meeting its latency targets.

They measure time-to-yield: the days from a model’s weights release to target throughput per GPU. Few providers publish it. It is the metric that matters most for the next release, and the one after that.

Capital is starting to price this in. The teams behind the two most widely used open-source serving engines spun out companies this year and raised $250 million in seed money between them within four months. Investors are paying not only for capacity, but for the ability to make new models run fast.

Predictions to hold us to

Three markers to check us against by the end of 2027:

  1. The #1 model keeps turning over. The top model on OpenRouter will change at least six more times by November 2027. If it changes fewer than three times, the product line is stabilizing and much of this argument weakens.
  2. Generic resale stays unprofitable. No inference provider whose revenue comes mostly from serverless tokens of other labs’ open models will show positive gross margin in an IPO filing during 2027. If one does, the commodity argument here is wrong.
  3. License cost becomes a line item. At least one major open-model lab will announce revenue-share agreements with named inference providers, making license fees a standard line in serving P&Ls.

The through line

Every month brings a new factory. The token coming off the line looks the same everywhere: same model, same listing, nearly the same price. The factory behind it does not.

The winners will need both: factories big enough to compete on cost, and the ability to retool them faster than the product line changes.