Self-hosting an LLM in production: hardware, cost, latency trade-offs

Why this guide exists

"Should we self-host?" is the most common question we field from engineering leaders in their first year of serious LLM deployment, and it is almost always asked without the numbers that would answer it. The question is tractable. The answer depends on four variables: your sustained request volume, your latency tolerance, your data sensitivity, and your willingness to operate GPU infrastructure. It produces different conclusions for different workloads inside the same company.

This guide is the reference material we wish existed when those conversations start. It is deliberately vendor-neutral: we name hardware and model families by their public specifications, not by commercial relationships. It is written for engineering teams who need to produce a defensible recommendation to a CFO, not a research paper.

The worked example throughout is a 70B-parameter dense model, a well-understood workhorse for production deployments that need to be materially better than a small model but not so large that the economics collapse. The framework generalises to smaller and larger dense models and, with different constants, to mixture-of-experts models; we flag where the numbers shift.

1. The decision, in one paragraph

Self-host when your sustained inference volume is high enough that amortised GPU cost beats per-token API pricing, when your latency budget is tight enough that a round-trip to a hyperscaler is material, or when your data classification genuinely prohibits external processing. Use APIs when your traffic is bursty, your volume is modest, or your team cannot commit to operating GPU infrastructure as a first-class production concern. Most organisations end up with both, routed by workload, and the interesting engineering work is in the routing layer, not in picking a side.

The rest of this guide is what you need to make that paragraph concrete for your workload.

2. What "70B in production" actually means

A 70B-parameter dense transformer (the class that includes Llama 3.3 70B and Qwen2.5-72B, both released in the second half of 2024) has a few characteristics that determine everything downstream.

We use Llama 3.3 70B as the worked example because it is dense and its architecture is published, which keeps every number in this guide checkable. Most notable open-weight releases since then use a mixture-of-experts (MoE) design instead: Meta's Llama 4 Scout and Maverick (17 billion active parameters out of 109 and 400 billion), OpenAI's gpt-oss-120b (5.1 billion active out of 117 billion), and the larger Qwen3 and Qwen3.5 models (up to 235 and 397 billion in total), while Qwen's dense models in those two generations stop at 32 and 27 billion. For an MoE model, memory follows the total parameter count and per-token decode traffic follows the active count, so the method below still works but the constants change.

At 16-bit precision (bf16 or fp16), the weights alone occupy about 141 GB of GPU memory (70.6 billion parameters × 2 bytes). At 8-bit quantisation, about 71 GB. At 4-bit, about 35 GB, plus a small overhead for scaling factors. The KV cache, the per-request memory that holds attention state, adds meaningfully on top of this, scaling linearly with context length and batch size. Llama 3.3 70B uses grouped-query attention with 80 layers, 8 key-value heads and a head dimension of 128, so each token costs 80 × 8 × 128 × 2 (keys and values) × 2 bytes = 327,680 bytes, about 0.33 MB, at 16-bit precision. For an 8K context and a batch of 16 concurrent requests, that is 16 × 8,192 × 327,680 bytes, about 43 GB of KV cache.

This memory footprint is what dictates your hardware floor. At bf16 you need more than 141 GB before the first request arrives, so a single 141 GB H200 cannot serve the model at full precision, and two 80 GB H100s leave almost nothing for the KV cache once the runtime has taken its share. The workable bf16 options are two H200s connected by NVLink, four H100s, or a single GPU with 180 GB or more (the B200 at 180 GB, the MI300X at 192 GB) running a modest KV cache. Otherwise you quantise: at fp8 the weights fit on one H200 or two H100s with room for the cache. Every production decision for this model class is downstream of that fact.

The second thing that matters is the compute profile. Inference on a dense 70B model is memory-bandwidth-bound for generation (decode) and compute-bound for prompt processing (prefill). This means single-user latency, and throughput at moderate batch sizes, are governed by how fast your GPU can stream weights and KV cache from HBM to the tensor cores, while prompt processing is governed by raw FLOPs. You will optimise for one or the other, and the choice shapes your hardware selection.

3. Hardware options, as of mid-2026

The practical GPU inventory for serious 70B inference is narrower than vendor marketing suggests. We will cover the five classes that matter.

H100 80GB SXM / PCIe

The workhorse of the last few years, still the most common starting point for organisations building their first inference cluster. At 16-bit precision a 70B model does not fit usefully on two H100s: 160 GB less 141 GB of weights leaves almost nothing for the KV cache. Two H100s with NVLink serve it well at fp8, and four serve it at 16-bit with reasonable batch sizes. The PCIe variant saves capex but has lower compute and power limits, and without an NVLink bridge the traffic between GPUs goes over PCIe: fine for lower-concurrency workloads, painful once you start batching aggressively. Rental prices fell through 2025, then rose again in 2026 as capacity tightened (see section 12).

H200 141GB

The quiet upgrade. Same compute architecture as the H100, but with HBM3e giving you 141 GB per GPU and 4.8 TB/s of memory bandwidth, against 80 GB and 3.35 TB/s on the H100 SXM. For 70B inference this is the sweet spot for 2026, with one correction to a common assumption: at bf16 the weights alone (about 141 GB) fill a single H200 and leave nothing for the KV cache. The bf16 unit is a pair of H200s with tensor parallelism, which leaves about 110 GB for KV cache after setting aside 10% of memory for the runtime; at fp8 a single H200 holds the weights with about 55 GB to spare. The bandwidth increase translates directly into single-request latency improvements. For a two-GPU server you would normally buy the PCIe H200 NVL, which pairs over a two-way NVLink bridge; the SXM version ships on four- and eight-GPU HGX boards. If you are building a new cluster today and the procurement calendar allows, this is usually where the analysis lands.

B200 180GB

The generational step. 180 GB of HBM3e per GPU on HGX and DGX systems, about 8 TB/s of memory bandwidth (2.4x the H100 SXM, and bandwidth sets the ceiling for single-stream decode), and native fp4 support that shrinks the 70B weights to about 35 GB. At bf16 a single B200 holds the weights with about 20 GB left for KV cache, enough for light concurrency; at fp8 it is comfortable. Its successor, the B300 (Blackwell Ultra), has about 260 GB per GPU on DGX B300 and has been shipping in volume since September 2025. The caveats are availability, power (NVIDIA rates a DGX B200 at about 14.3 kW for eight GPUs, a system-level thermal envelope that most existing datacentre rows cannot accommodate), and cost. For organisations with genuine scale (sustained tens of thousands of requests per minute) the Blackwell economics are compelling. For anyone else, H200 is the more pragmatic choice.

AMD MI300X 192GB

The credible alternative. 192 GB of HBM3 per GPU and 5.3 TB/s of memory bandwidth, with its successors already on the market: the MI325X (256 GB, 6 TB/s) and the MI355X (288 GB, 8 TB/s), and the MI400 series due in the second half of 2026. At bf16 a single MI300X holds the 70B weights with about 30 GB left for KV cache. The software stack (ROCm) has matured substantially: vLLM and SGLang both support the MI300 and MI350 series, and the ecosystem gap that existed in 2024 is largely closed for mainstream inference workloads. TensorRT-LLM is not an option here; it runs on NVIDIA GPUs only. Pricing and availability vary by supplier, so get competing quotes rather than assuming a fixed discount to NVIDIA. The risk remains operational: if your team has no ROCm experience, budget real time for the learning curve, and do not underestimate the cost of debugging a kernel-level issue on a less-trodden path.

Custom silicon

AWS Trainium3, Google TPU v7 (Ironwood), Groq LPU, Cerebras, SambaNova. Each of these can be the right answer for specific workloads, and each carries lock-in considerations that sit outside pure economics. Groq's LPU produces remarkable single-stream speed (Artificial Analysis measures close to 300 tokens per second on Llama 3.3 70B), useful for interactive applications where tokens-per-second-per-user dominates perception, but the architectural assumptions differ enough from GPU deployment that porting is non-trivial. In December 2025 NVIDIA licensed Groq's inference technology and hired its founder and president; Groq says GroqCloud continues to operate as an independent service, and the change is worth weighing before you build a long-term dependency on it. TPUs are excellent inside the Google Cloud ecosystem and awkward outside it; the current generation, Ironwood, is available there. Trainium3, generally available since December 2025, is worth evaluating for large-batch inference if your stack already lives in AWS. We treat these as workload-specific accelerators rather than general-purpose infrastructure, and we would not recommend any of them as a first deployment unless you have a specific latency or cost target that only they can hit.

For the rest of this guide, the reference hardware is a node of two H200s serving Llama 3.3 70B at bf16, with tensor parallelism across both GPUs. We keep bf16 because it is the precision the model was released at, so its quality is the baseline you evaluated; fp8 on the same hardware is the main cost lever, and we price it alongside. The methodology transfers; the constants will shift.

4. Sizing the deployment

The sizing question reduces to four inputs: expected concurrent requests at peak, the median and p99 context length, your tokens-per-second-per-user latency target, and your tolerable queue depth.

Most teams get this wrong in the same direction. They size for average load, discover at peak that tail latency collapses, and either over-provision reactively or start dropping requests. The correct default is to size for your p95 concurrent load with a 30% headroom, then autoscale the tail.

The reference node (two H200s, tensor parallel 2, bf16) running a well-tuned inference server (vLLM, SGLang, or TensorRT-LLM) will sustain, roughly, the figures below. They are our estimates from the bandwidth and compute arithmetic explained under the table, for a reference request of 2,000 input tokens and 400 output tokens; measure your own.

How the figures are derived. Decode: each step reads every GPU's share of the weights (70.6 GB) plus its share of the batch's KV cache (half of 327,680 bytes for every token in context) at 4.8 TB/s, of which we assume 60–75% is achieved in practice; at batch 64 with 2,200 tokens of average context that is about 28 ms per step, or about 2,300 tokens/sec across the batch. Prefill: about 141 GFLOP per token (2 × 70.6 billion parameters) against about 1,670 dense bf16 TFLOPS for two H200 NVL cards at 50% utilisation, roughly 6,000 tokens/sec. The reference request therefore occupies the node for about 2,000/6,000 + 400/2,300 = 0.51 seconds, hence about 2 per second. As a cross-check, NVIDIA's published TensorRT-LLM figures for Llama 3.3 70B at fp8 on two H200s, measured at unconstrained latency, are about 4,800 output tokens/sec for 1,000 input and 1,000 output tokens, and about 750 for 2,048 input and 128 output; the same arithmetic is consistent with both.

These numbers assume 16-bit precision. On the same node, fp8 weights halve the bytes read for every decoded token and double the tensor-core rate for prefill; our arithmetic gives about 1.8 times the requests per second for the reference mix, and measured gains are often lower. fp8 quality loss is often small on models of this size, but it is workload-dependent, so test it against your evaluation set. fp4 needs Blackwell-class GPUs (the H100 and H200 have no fp4 tensor cores) and shows measurable quality degradation on some tasks, so test even more carefully.

For sizing, the useful abstraction is requests per node-hour for your own request shape. The reference node handles about 2 reference requests per second at full load, or 7,200 per hour. Translate your expected peak request volume into this unit and divide by the headroom you need; that gives you the minimum node count. Prompt length matters as much as answer length: on this node the 2,000-token prompt costs about twice the node time of the 400-token answer.

5. Latency, measured properly

Latency is where self-hosting earns its keep or loses its case, and the conversation is usually sloppy. "Faster than the API" is not a useful claim; the question that matters is: faster on which latency percentile, at which token position, under which load?

Break the latency budget into four components.

Time to first token (TTFT)

The interval between request arrival and the first generated token landing in the client. TTFT is dominated by prefill: tokenisation, attention computation over the full input context, and the first decode step. For a 70B model on the reference node, TTFT on a warm cache with a 2K input context lands around 300–450ms; on a 16K input context it is roughly 2.5–4s. Commercial API endpoints for the same model show times to first token from about 0.65 to 2.6 seconds depending on provider, network included (Artificial Analysis, September 2026). The difference is in the tail: a self-hosted deployment with no network hop and no cross-tenant queueing can deliver predictable p99 TTFT, while shared APIs can spike to several seconds under load.

Inter-token latency (ITL)

The interval between generated tokens after the first. This is the number that governs the perceived speed of streaming responses. On the reference node at bf16, expect roughly 23–33ms per token under moderate batching (batch 32–64); even a lone request cannot go below about 15ms, the time to read 70.6 GB per GPU at 4.8 TB/s. Quantised to fp8, roughly 13–20ms. Many API endpoints for this model stream faster than a bf16 node: Artificial Analysis measures them at 16 to about 290 tokens per second, with a median of about 84 (roughly 12ms per token). Groq's LPU and Cerebras' wafer-scale inference go further (Cerebras reported 2,100 tokens per second on Llama 3.1 70B in October 2024, under 0.5ms per token), which materially changes what interactive applications feel like.

End-to-end latency

TTFT plus (ITL × output tokens). For a typical 300-token response with a 2K input context on the reference node, this is roughly 350ms + (300 × 26ms) = about 8 seconds. For the same workload on a commercial API endpoint, the medians of Artificial Analysis's measurements for this model (about 1.8 seconds to first token and 84 tokens per second) give roughly 1.8s + (300 × 12ms) = 5.4 seconds, network included, with the fastest endpoints near 2 seconds and the slowest above 20. So at the median a bf16 node is often slower than a good endpoint for a streamed answer, and fp8 closes most of that gap. What self-hosting offers is a short, predictable time to first token and a tail you control, and the tail is where real products live or die. Measure both on your own traffic.

Network and serialisation

A self-hosted deployment inside your VPC adds 2–15ms of network round-trip. A commercial API from the same cloud region adds 30–80ms typically, 200ms+ under load or across regions. If your application makes multiple LLM calls per user action (increasingly common with agentic workloads) this compounds quickly.

6. Cost, with the actual arithmetic

This is the section that matters. We will work through the total cost of operating the reference 70B deployment and compare it directly to current commercial API pricing for the same model. The methodology is what transfers; plug your own numbers in.

The cost model

A self-hosted inference deployment has five cost components: GPU capex or lease, host infrastructure (CPU, memory, networking, storage), power and cooling, personnel, and opportunity cost (the engineering time spent operating the system rather than building product). Most internal business cases ignore the last two and overstate the savings as a result.

For a meaningful comparison, we amortise hardware over three years, include a fully-loaded operational overhead, and compare against September 2026 list prices for Llama 3.3 70B from four inference providers. We compare whole requests rather than output tokens alone, because APIs charge for input tokens and prefill occupies your GPUs.

Reference deployment: 2× H200 cluster

Two H200 GPUs connected via NVLink (in a two-GPU server, the PCIe H200 NVL cards with a two-way NVLink bridge), hosted on a dual-socket server with 1 TB of system RAM, 100 GbE networking, and appropriate storage, serving Llama 3.3 70B at bf16 across both GPUs. Integrator quotes in mid-2026 put the H200 NVL at roughly $28,000–34,000 per card, and we estimate $30,000–40,000 for the host. Total capex: approximately $100,000.

Depreciation is the midpoint capex of about $97,000 over three years. Power uses Eurostat's price for Irish non-household consumers in the second half of 2025, in the 20–499 MWh band this deployment falls into (2 kW × 8,760 hours × 1.4 = about 24.5 MWh a year): €0.2915/kWh excluding VAT and recoverable taxes, converted at the ECB reference rate of $1.1367 (24 September 2026). France, at €0.1946, would bring the line to about $5,400. The colocation and personnel lines are our planning assumptions. With all ranges at their ends, the owned total runs from about $85,000 to $120,000 a year.

Cloud-hosted equivalent: H200 list prices in September 2026 run from about $4.50 per GPU-hour on demand at a smaller provider (Nebius), through about $6–7 on AWS Capacity Blocks and CoreWeave, to about $7.90 on AWS on demand. At $4.50–6.50 per GPU-hour, one GPU costs 8,760 hours × $4.50 = about $39,000 to 8,760 × $6.50 = about $57,000 a year, so the two-GPU equivalent is $79,000–114,000 before staff; add the same personnel line and the total is about $145,000. The large clouds mostly sell H200 capacity in eight-GPU instances, so a two-GPU footprint usually means a smaller provider. More expensive than owned hardware amortised, but with no capex and full elasticity.

Throughput at this scale

At the performance envelope discussed earlier, the reference node serves about 2 reference requests per second (2,000 input and 400 output tokens each) at full load, which is 2 × 31.5 million seconds = about 63 million requests a year. Assume realistic average utilisation of 50% (production traffic is not uniform), and you get about 31.5 million requests a year, around 86,000 a day, or about 76 billion tokens processed, 12.6 billion of them output.

The self-hosted rows assume 50% utilisation: $100,000 / 31.5 million requests = $3.17 per 1,000, and the same over 76 billion tokens = $1.32 per million. The fp8 row applies the 1.8x throughput estimate from section 4. The API range covers list prices from Together AI ($1.04 per million tokens, input and output alike), Cloudflare Workers AI ($0.29 input, $2.25 output), Novita ($0.135 input, $0.40 output) and DeepInfra ($0.10 input, $0.32 output, an fp8 deployment), applied to 2,000 input and 400 output tokens per request. If you prefer the per-output-token figure many business cases use, the owned node comes to $100,000 / 12.6 billion = about $7.90 per million output tokens, but that figure carries five input tokens for every output token, so compare it with an API's cost for the same request shape rather than with its output price.

The conclusion

On cost alone, a bf16 70B deployment on owned H200s does not beat per-token APIs for the same model in 2026. At 50% utilisation it costs about $3.20 per 1,000 reference requests, against $0.33–2.50 at list prices. Moving to fp8 on your own hardware roughly halves that, and the cheapest listing is still about five times cheaper. Even at fleet scale, where staff costs are spread thin, a bf16 node's hardware, power and colocation alone come to about $1.20 per 1,000 requests at 70% utilisation, more than three times the cheapest listing. The cases where self-hosting wins are the ones in sections 5 and 9 (latency tails and data constraints), plus very high, steady utilisation against a list-price API.

There is a threshold, and it is simple to compute: break-even volume = annual self-hosting cost / API cost per request. Against the highest list price above ($2.50 per 1,000 reference requests), the owned node breaks even at $100,000 / $0.0025 = 40 million requests a year, about 110,000 a day. That is 63% of the node's full-load capacity of 63 million, sustained around the clock, and equivalent to about $8,300 a month of API spend. With fp8 on the same node the same volume is about 35% of capacity. Against the cheapest listing ($0.33 per 1,000), the formula gives about 300 million requests a year, nearly five times what the bf16 node can serve at full load, so there is no break-even. Below the threshold, APIs are the correct answer on cost alone. Above it, the economics can flip, and the question becomes whether you want to build the operational capability.

7. The operational cost nobody models

The financial case above undercounts the real cost of self-hosting in one important way: it assumes the inference stack works. In production, it does not, without ongoing engineering investment. The list of things you are now responsible for includes:

• Model weight management and versioning
• Evaluation infrastructure that runs against every new model or configuration change
• Inference server upgrades (vLLM, SGLang, and TensorRT-LLM all move quickly and introduce breaking changes)
• GPU driver and CUDA stack maintenance
• Kernel-level debugging when a batch pattern tickles a rare bug
• Capacity planning that keeps pace with product growth
• Incident response at 3am when the inference tier falls over
• Security patching of the full stack
• Regulatory compliance documentation for the infrastructure
• Monitoring, alerting, and SLO definition for a fundamentally probabilistic service

None of this is exotic; it is table stakes for running any production system. But teams new to GPU operations routinely underestimate the magnitude. A fair way to sanity-check a self-hosting business case is to ask whether the team has operated a similarly demanding stateful workload before. If the answer is no, the break-even volume is higher than the pure cost model suggests, and the quality of the first six months of service will be worse than an API.

8. The quality question

Throughout this guide we have treated "70B-class" as a single capability category. It is not, and the gap between the best commercial models and the best open-weight 70B models, while narrowed substantially over the last two years, is still real on the workloads that demand the most capability: complex reasoning, long-context coherence, tool use, and the kind of nuanced instruction-following that distinguishes a product from a demo.

For a large fraction of production use cases (classification, extraction, summarisation, retrieval-augmented generation, structured output, and the bulk of agentic tool-calling) a well-tuned open 70B model is indistinguishable from a frontier API in output quality, and self-hosting is a straightforward cost conversation. For the hardest 10–20% of use cases, the quality gap still favours the frontier APIs, and no amount of GPU investment closes it. The practical architecture for most organisations is to route by workload: self-host the volume, call out to a frontier API for the hard cases, and treat the routing layer as a first-class piece of infrastructure.

9. Data, sovereignty, and the cases where cost is not the question

Some workloads are not a cost conversation. If your data classification or contractual commitments prohibit data leaving your infrastructure (common for classified defence work and for some healthcare and financial-services data), the question is not whether self-hosting is cheaper. It is whether self-hosting is possible, at what quality, and on what timeline.

Two patterns are worth knowing.

Air-gapped inference

The model runs inside infrastructure that has no route to the public internet, typically for classified or regulated workloads. Every piece of the stack (weights, inference server, monitoring) must be vendored and auditable. Open-weight models are the only viable option here; API access is definitionally out. This is a mature pattern with real reference deployments, and the trade-offs are well understood.

Sovereign cloud inference

The model runs in infrastructure located and operated within a specific jurisdiction, often by a provider that commits to keeping the data, the support staff and operational control there. Organisations choose it to reduce data-transfer and foreign-access risk. It does not by itself satisfy any particular law. GDPR, DORA and the EU AI Act do not, as a general rule, require data to stay in one country; they set rules on transfers, third-party risk and governance, and where the model runs is one input to how you meet them. Nor does the label settle who can be compelled to hand data over. A provider subject to US jurisdiction can be required under the CLOUD Act (18 U.S.C. § 2713) to disclose data in its possession, custody or control wherever the data is stored, and whether that reaches a European entity's sovereign offering depends on how control is structured, which is a question for your own legal review. This is a middle path: the operational convenience of a cloud deployment with reduced transfer and access risk, at a cost premium. All three large US clouds now market a European sovereign offering, as do European providers; GPU availability and terms vary substantially, and the contractual due diligence is as important as the technical architecture.

Both patterns materially change the economics, and both are increasingly common. If your organisation falls into one of these categories, the self-hosting question is not whether, but how, and the rest of this guide's cost analysis applies only as a lower bound: air-gapped and sovereign deployments add cost on top of it.

10. A deployment architecture that works

For teams building their first serious self-hosted 70B deployment, the architecture that reliably works in production has five components.

1. An inference server (vLLM, SGLang, or TensorRT-LLM). All three are credible in 2026; vLLM has the broadest community, SGLang has the best structured-output performance, TensorRT-LLM has the best raw NVIDIA throughput and runs on NVIDIA GPUs only. Pick one, commit to it, and do not switch without a specific reason.
2. A gateway layer: a thin service in front of the inference server that handles authentication, rate limiting, request routing, logging, and the retry logic that inevitably becomes necessary. Do not put these concerns inside the inference server. LiteLLM, Portkey, and Kong AI Gateway are common choices; an in-house FastAPI service is equally valid for most teams.
3. An evaluation harness: automated regression testing that runs on every model or configuration change, anchored to a real workload trace, not a benchmark. This is the most important piece of infrastructure and the most frequently deferred. Without it, you cannot safely upgrade anything.
4. A routing layer: the hybrid-deployment glue that decides which requests go to self-hosted infrastructure, which go to external APIs, and which go to smaller local models. Even if you start with 100% self-hosted traffic, build the routing seam now; it is the foundation of every cost optimisation you will want to make later.
5. Observability: request-level tracing, token-level accounting, p50/p95/p99 latency by model and endpoint, and cost attribution back to the business unit that generated the traffic. The last point is where most programmes fail: without cost attribution, self-hosting looks free to downstream consumers, and demand expands to fill capacity.

This architecture scales from a two-GPU pilot to a hundred-GPU production fleet without structural change. It also makes the self-host-vs-API decision reversible at the workload level, which is the correct default.

11. A checklist for the decision

If you have read this far, the decision you are making is probably one of these four.

Whichever of these applies, check the model licence before you commit, because open weights do not come with one standard set of terms. Llama 3.3 ships under the Llama 3.3 Community License Agreement, which requires a separate licence from Meta above 700 million monthly active users, "Built with Llama" attribution, "Llama" at the start of the name of any derived model, compliance with Meta's Acceptable Use Policy, and California law. Llama 4's Acceptable Use Policy withholds the licence rights for its multimodal models from individuals domiciled in the EU and companies with their principal place of business there (end users of products built on them are not affected), which matters to any EU organisation considering Llama 4 Scout or Maverick. Qwen2.5-72B uses the Qwen licence, which requires a separate licence above 100 million monthly active users; Qwen3, Qwen3.5 and gpt-oss use Apache 2.0. Have someone read the actual terms for the model you choose; this summary is a starting point, not legal advice.

12. What will change in the next twelve months

Any reference document on LLM infrastructure has a shelf life measured in quarters, not years. The shifts we are watching:

• GPU supply is tight, not loosening. SemiAnalysis found H100 one-year rental contracts up almost 40% between October 2025 and March 2026, with on-demand capacity sold out across GPU types, and rental rates were still rising in September 2026. Blackwell (B200, B300) is shipping, and NVIDIA's Rubin and AMD's MI400 series are due from the second half of 2026. Whether that new supply brings H100 and H200 prices down within the year is the open question; until it does, the hardware side of the equation is not getting cheaper, and the break-even volume stays high.
• Quantisation continues to mature. fp4 inference on 70B models at production quality is now borderline viable and likely to become routine. It needs Blackwell-class GPUs, which support fp4 natively (the H100 and H200 do not), so its gains arrive with new hardware rather than on every GPU you already own.
• Commercial API pricing has fallen fast, though not evenly: Epoch AI measured the price of reaching a given capability level falling by between 9x and 900x a year depending on the task, and noted that the fastest drops were the most recent and the least certain to persist. Rising GPU costs may slow the trend. The break-even volume threshold moves up as API pricing falls, and down as hardware pricing falls; the two forces are partially offsetting.
• The open-weight quality gap continues to narrow on most workloads and persists on the hardest. This is the most consequential trend for the self-host decision, and it is impossible to predict where it settles.

Appendix: Quick reference numbers