In short (TL;DR)

Data as of Nov 7, 2026: inference overtakes training in AI-optimized IaaS spending (55% in 2026), a comparison of vLLM v0.33.0 / SGLang v0.5.23 / llm-d 1.0, a guide to open-weight models for self-hosting from Qwen3.5 to Kimi K3.1 Pro, and an APAC data center picture where power is now the binding constraint.

Data as of 7 November 2026, focusing on evidence from 2025-2027. This report summarizes the current state of AI inference, serving software, open-weight models for self-hosting, and the APAC data center market - including the evidence limitations you should know before making decisions.

Executive summary

  1. 2026 can be called the "Year of Inference" in the economics of cloud infrastructure - not proof that inference now consumes more compute than all training worldwide. Gartner expects AI-optimized IaaS spending on inference to reach $23.3B in 2026 versus $19B for training (55% of the market), rising to 59% in 2027, per the JLL Asia Pacific Data Centre report
  2. The 47% quarterly cost decline (roughly 13x per year) is supported by September 2026 research, but refers to the cheapest cost of reaching a fixed capability score (quality-adjusted cost) - not that any given model’s token price or a self-hosted cluster’s power bill drops 13x every year
  3. For self-hosting, the 2026 inflection is model- and hardware-specific software optimization - speculative decoding, low-precision KV cache, prefix caching, paged experts, and prefill/decode disaggregation - more than whether vLLM or SGLang is "always faster"
  4. The frameworks to evaluate now are vLLM v0.33.0, SGLang v0.5.23, and llm-d 1.0. vLLM/SGLang are model-serving engines, while llm-d is a distributed serving/control plane on Kubernetes - not a like-for-like competitor
  5. Open-weight models split into two clear markets: 4B-35B / mid-size MoE that genuinely fit in-organization servers (Qwen3.5, Gemma 4), and frontier open-weight models in the hundreds of billions to trillions of parameters (DeepSeek V4.1 Flash, GLM-5.3, Kimi K3.1 Pro) that require serious multi-GPU/multi-node infrastructure
  6. APAC is in power-constrained execution: the H1 2026 data center pipeline stands at 26,455 MW - 4,764 MW under construction and 21,691 MW planned - with power and grid connections now the binding constraint

1. The facts about the "Year of Inference"

QuestionLatest conclusion
Has inference overtaken training?Yes in AI-optimized IaaS spending tracked by Gartner: $23.3B vs $19B in 2026 - do not overgeneralize to all compute worldwide
2027 shareGartner expects inference at 59% of AI-optimized IaaS spending in 2027
Total marketAI-optimized IaaS is projected to grow from about $42.3B in 2026 to $66.1B in 2027 per BMI analysis
47% quarterly price declineCorrect per recent research, but it is quality-adjusted cost per task - picking the cheapest model that passes a target score - not the raw token price of an existing deployment
Jevons paradoxA reasonable hypothesis; current data only proves cost-per-capability is falling fast while total inference spending rises - not token-consumption growth for every organization

2026 is the point where cloud AI infrastructure spending on serving inference overtakes training per Gartner projections, while cost per capability level falls rapidly - but agents, long context, and multi-step workflows keep total compute and infrastructure demand climbing.

2. Current state of inference software

Framework / latest versionLatest performance evidenceBest forInterpretation caveats
vLLM v0.33.0 (Nov 2, 2026)DeepSeek-V4.1-Flash + DSpark on GB300 rises from 1,373 to 3,390 output tok/s; FlashInfer DCP prefill cuts mean TTFT 39-48% at 100K input on 8xB200; FP8 KV cache for GLM-5.x halves KV memory per rank but adds 7-11% TTFTOpenAI-compatible serving, single-GPU to multi-node, NVIDIA/AMD/Intel/CPU, and systems needing a broad ecosystemMost figures are per-pull-request benchmarks, not controlled apples-to-apples framework comparisons
SGLang v0.5.23 (Nov 2, 2026)DeepSeek-V4.1 on GB300 reaches up to 1.5x prefill throughput and ~28% lower TTFT for 32K-64K prompts; low-latency kernels speed single-user decode up to 44% on GB300 and 75% on MI350X; Paged Experts beats CPU offload by up to 30xAgentic/long-context workloads, MoE, Radix/prefix caching, speculative decoding, rapid optimization experimentsMany features are opt-in, and numbers are tied to specific models, GPUs, batch/concurrency, and kernels
llm-d 1.0 (Oct 20, 2026)On an agentic coding workload on TPU 7x, prefix-aware routing and host-memory offload cut median TTFT by more than 85% and lifted throughput 2.4x versus Kubernetes Service; supports both vLLM and SGLang per the Google Cloud launch postKubernetes clusters, mixed accelerators, multi-tenancy, cache-aware routing, KV offload, autoscaling, prefill/decode disaggregationNot a kernel engine; benefits show up with multiple replicas, prefix reuse, or bursty traffic
llm-d v0.7 heterogeneousIn a 20-GPU pool from three vendors: 14.2K output tok/s vs 9.6K peak baseline; at 85 QPS it beats Kubernetes round-robin by 91% with TTFT of 6.8s vs 36.4s per the llm-d heterogeneous benchmarkOrganizations with mixed GPU generations/vendors wanting one inference poolThe workload had ~7.2K-token shared prefixes, strongly favoring cache-aware routing; accelerators were anonymized

Engineering conclusion: do not just ask "is vLLM or SGLang faster" - choose by workload:

  • Start with vLLM for API compatibility, broad hardware coverage, many integrations, and a more conservative production path (see the vLLM V1 guide)
  • Benchmark SGLang alongside for new MoE models, agentic long context, speculative decoding, or high prefix-reuse workloads
  • Add llm-d once you grow beyond one replica/node and hit cache locality, queue fairness, mixed hardware, or prefill/decode contention issues
  • Never compare headline numbers from different releases directly - models, precision, prompt lengths, concurrency, and GPUs differ

3. Latest open-weight models for self-hosting

Use case tierModels to testSize/profileSelf-hosting fit
Single GPU / small serverQwen3.5-9BDense 9B; MMLU-Pro 82.5A sensible default; weights ~18 GB at BF16 or ~4.5 GB theoretical at 4-bit before runtime/KV overhead
Large single GPU or 2 GPUsQwen3.5-35B-A3BMoE 35B total / ~3B active; MMLU-Pro 85.3Lower compute per token than dense 35B but still stores all 35B weights; ~70 GB BF16 or 17.5 GB at 4-bit
Local multimodal / edge serverGemma 4 12B or 26B-A4BE2B, E4B, 12B, 26B MoE and 31B Dense; Apache 2.0; vLLM, SGLang, llama.cpp, Ollama supportGreat for in-organization multimodal and resource-constrained devices
Enterprise multi-GPUGLM-5.3-Flash320B total / 18B active; official GGUF ~328 GBNeeds multi-GPU or aggressive quantization/offload; low active parameters do not mean you can store only 18B
Enterprise clusterDeepSeek-V4.1-Flash552B total / 16B active, 1M context, MIT licenseSuited to multi-node inference; gets more dedicated optimizations in recent vLLM/SGLang than general models
Frontier on-premiseGLM-5.3753B total / ~40B active, 1M context; repository ~756 GBRequires a cluster and distributed serving; not a single-GPU model
Maximum frontierKimi K3.1 Pro~2.8T total / 104B active; official weights ~1.56 TB downloadOpen-weight but demands large infrastructure; check the custom license before commercial use

Note: memory figures estimate weights only (2 bytes/parameter BF16, 0.5 byte 4-bit). Real deployments must also budget quantization metadata, CUDA graphs, activations, runtime workspace, and KV cache - especially for long context and high concurrency.

On quality vs speed, Artificial Analysis as of 7 November 2026 ranks Kimi K3.1 Pro as the top open-weight model by Intelligence Index, but its measured API output speed is around 33 tok/s at $2/M input and $10/M output, while DeepSeek V4.1 Flash runs about 205-221 tok/s at $0.30/M input and $1.20/M output. These compare hosted endpoints, not the throughput an on-prem cluster would deliver.

4. Self-hosting approach for 2026-2027

Recommended optimization order

  1. Define SLOs before choosing hardware: P50/P95 TTFT, TPOT or inter-token latency, requests/s and output tok/s, input/output length distribution, concurrency and prefix reuse, plus availability and data-residency requirements
  2. Pick the smallest model that passes task evaluation: much retrieval, classification, extraction, and internal-assistant work never needs 300B+; route only hard tasks to frontier models
  3. Quantize weights and KV cache separately: weight quantization fits models into GPUs; FP8/FP4 KV cache helps concurrency and context capacity - but test accuracy, TTFT, and throughput together
  4. Enable prefix caching when prompts genuinely repeat: great for system prompts, RAG templates, coding agents, and multi-turn sessions; benefits collapse with highly scattered prefixes
  5. Use speculative decoding when decode-bound: fits interactive generation and low-to-medium batch/concurrency; measure acceptance rate - a fast draft model that guesses wrong often may not help
  6. Disaggregate prefill/decode for long-prompt workloads: prefill is compute-heavy, decode is bandwidth-heavy; separate pools reduce interference but add network/KV-transfer complexity
  7. Add llm-d or a similar control plane with multiple replicas: cache-aware routing, load/saturation-aware scheduling, priority lanes, KV offload, and rolling upgrades that preserve session state

Benchmarks organizations should run themselves

Use at least three real trace sets: interactive (1-4K input, 200-1K output, low concurrency), RAG/enterprise (8-32K input, multi-level shared prefixes), and agentic/long-context (32-256K+ input, multi-turn, bursty concurrency). Report at minimum P50/P95/P99 TTFT, P50/P95 TPOT, output tok/s, request throughput, GPU utilization, HBM utilization, cache-hit rate, power draw, and cost per successful task, following the llm-d validation benchmark methodology

5. Data centers and power in APAC

Region/marketLatest picture and projectionsImpact on AI inference
APAC overallJLL expects 4.8 GW of new supply by 2027, 78% already pre-leased, and 24 GW of capacity added between 2025-2030Truly vacant capacity may tighten faster than construction figures suggest
APAC pipelineH1 2026 pipeline of 26,455 MW: 4,764 MW under construction, 21,691 MW plannedBeing "in the pipeline" does not mean powered and ready on schedule
Rack densityPwC expects average AI racks to rise from 15-20 kW to 40-50 kW by end-2027Design liquid-ready even if the first rollout is below 40 kW
ChinaIEA estimates data centers used over 100 TWh in 2024, possibly doubling by 2027One of the region’s largest incremental loads
MalaysiaPwC projects capacity from 522 MW in Q1 2025 to 2,055 MW by end-2026Johor is a key market, but grid, water, and renewables are constraints
ThailandCapacity expected to grow from 242 MW in 2025 to 550 MW in 2026, with supply pressure possibly easing in 2027 as new generation comes onlineVerify grid-connection dates rather than building-completion dates
Indonesia/BatamBatam has roughly 70 MW live and a 747 MW pipelineMain risks are transmission, substations, and interconnection wait times
IndiaH1 2026: 1,789 MW operational with a 3,860 MW pipeline; data centers may grow from 0.8% of electricity in FY2025 to 2.6% in FY2030High-growth market requiring early power procurement and transmission planning
SingaporeAbout 1.4 GW existing capacity, with plans for at least 300 MW more plus 200 MW tied to green-energy optionsStrong connectivity, but new capacity depends on efficiency and clean energy

New-generation rack power: separate "market average" from "frontier rack"

Average APAC racks may move to 40-50 kW by 2027, but frontier racks are far beyond that. As Kearney analysis argues, power will decide the next data center winners:

  • NVIDIA lists Vera Rubin NVL72 with provisioned rack power around 136 kW, reducible to 101 kW with power smoothing in tested workloads
  • NVIDIA is pushing the 800 VDC architecture to support 1 MW-class IT racks from 2027
  • OCP estimates H100-class racks at ~40 kW, GB200 at ~120 kW, and Rubin-generation racks in 2027 potentially at 600 kW to 1 MW

So "600 kW to 1 MW per rack" is a roadmap for the next generation of frontier racks - not the average data center density of 2026.

6. Synthesis for APAC organizations

  • Start production with Qwen3.5-9B/35B-A3B or Gemma 4 for most workloads
  • Use large DeepSeek/GLM/Kimi models via API or dedicated hosted endpoints until utilization justifies an in-house cluster
  • Benchmark both vLLM and SGLang on real hardware, pinning model revision, framework/container version, driver/CUDA/ROCm, quantization, context distribution, and concurrency
  • Choose data centers by available commissioned power and grid-connection SLAs, not pipeline announcements
  • For new procurement, require at minimum: direct-to-chip liquid-cooling readiness, 80-150 kW/rack capability for near-term AI clusters, an upgrade path to 800 VDC, and rack/GPU-level telemetry to measure tokens per kWh

Evidence limitations

  • vLLM and SGLang release-note benchmarks are first-party and very fresh, but mostly component/PR benchmarks rather than controlled framework comparisons
  • Model-card benchmarks usually come from model developers - validate with internal datasets and independent benchmarks
  • Data center "capacity", "IT load", "utility capacity", and "pipeline" are not consistently defined across reports - never sum MW figures across sources
  • Power projections are highly uncertain: projects slip due to transformers, transmission, water, permits, or power-purchase agreements

Recommended next steps

  1. Build one benchmark harness for both vLLM v0.33.0 and SGLang v0.5.23
  2. Test Qwen3.5-9B, Qwen3.5-35B-A3B, and one frontier model with real traces
  3. Compute the API vs self-hosting break-even on cost per successful task - not cost per token alone
  4. Plan capacity separately for HBM, KV cache, network bandwidth, rack power, and cooling
  5. Add llm-d only when tests show losses come from routing, cache locality, or prefill/decode contention - not the model’s own kernels

Sources

  1. JLL Asia Pacific Data Centre report www.jll.com
  2. llm-d llm-d.ai
  3. BMI analysis asian-power.com
  4. Google Cloud launch post cloud.google.com
  5. llm-d heterogeneous benchmark llm-d.ai
  6. vLLM github.com
  7. vLLM V1 guide docs.vllm.ai
  8. SGLang www.sglang.io
  9. Artificial Analysis artificialanalysis.ai
  10. llm-d validation benchmark methodology llm-d.ai
  11. PwC www.pwc.com
  12. IEA www.iea.org
  13. Kearney analysis www.kearney.com