GPU vs TPU vs custom accelerators, InfiniBand vs RoCE vs proprietary interconnects, VM vs bare metal - a comprehensive analysis of AI cloud infrastructure architecture with real cost comparisons.
When you click "deploy" on an AI cloud instance, you are renting a slice of an enormously complex hardware stack. Understanding what is underneath - the chips, the network, the storage - helps you make better decisions about where to run AI workloads and what you should actually pay.
1. Evolution and foundation of AI Cloud VMs
AI cloud computing evolved in three waves:
- Wave 1 (2016-2020): Repurposed infrastructure - standard cloud VMs with GPUs bolted on. Networking and storage were not designed for AI training patterns.
- Wave 2 (2020-2024): Purpose-built clusters - dedicated AI infrastructure with high-bandwidth interconnects (InfiniBand, NVLink fabric) and parallel storage systems.
- Wave 3 (2024-present): Full-stack AI factories - rack-scale systems (NVIDIA NVL72, Google TPU v5p pods) where compute, network and storage are designed as one unit.
2. Hardware architecture comparison: GPU, TPU and custom accelerators
| Aspect | NVIDIA GPU (H200/B200) | Google TPU (v5p) | AWS Trainium | Custom (Cerebras, etc.) |
|---|---|---|---|---|
| Architecture | General-purpose SIMT | Systolic array | Systolic array | Wafer-scale / dataflow |
| Best for | Training + inference | Large-batch training | Cost-effective training | Specialized workloads |
| Framework | CUDA (universal) | JAX/XLA | AWS Neuron | Proprietary |
| Ecosystem | Extensive | Google-locked | AWS-locked | Emerging |
Key insight: NVIDIA’s dominance is not just hardware performance - it is the CUDA software ecosystem. Every ML framework, research paper and open-source model works on CUDA first. Choosing TPU or Trainium means accepting ecosystem friction for potentially better price-performance.
3. High-speed networking and storage architecture
Interconnect options
- NVLink/NVSwitch: 900 GB/s per GPU (NVL72). Proprietary to NVIDIA. Best for intra-rack communication during training.
- InfiniBand: 400 Gbps per node. Industry standard for HPC. Low latency (sub-microsecond). Expensive but proven at scale.
- RoCE (RDMA over Converged Ethernet): 400 Gbps. Uses standard Ethernet - cheaper than InfiniBand with slightly higher latency.
- Proprietary fabric (Google ICI, AWS EFA): Cloud-provider-specific. Optimized for their infrastructure but locks you in.
Storage for AI workloads
AI training reads massive datasets sequentially; checkpoints are written in bursts. Storage requirements are fundamentally different from web applications:
- Parallel filesystems (Lustre, GPFS, WekaIO): aggregate throughput of 100+ GB/s for reading training data
- Object storage (S3, GCS): cheap for cold data, too slow for hot training loops
- Local NVMe: fastest for temporary data but not shared across nodes
4. AI Cloud VM vs Bare Metal
| Dimension | AI Cloud VM | Bare Metal |
|---|---|---|
| Time to provision | Minutes | Days to weeks |
| Elasticity | Scale up/down on demand | Fixed capacity |
| Cost model | Pay per hour (expensive at sustained load) | CapEx + maintenance (cheaper at high utilization) |
| Multi-tenancy overhead | 5-15% performance loss | Zero - full hardware access |
| Best for | Spiky workloads, experimentation, burst training | Sustained inference, predictable training pipelines |
Rule of thumb: if your GPU utilization exceeds 70% for more than 6 months, bare metal is likely cheaper.
5. Provider landscape and cost economics
- AWS (P5 instances): 8x H100 per instance. Most mature ecosystem. Premium pricing.
- Google Cloud (TPU): best price-performance for large-batch training if your code works with JAX.
- Azure (NDv5): strong enterprise integration, InfiniBand-connected.
- CoreWeave, Lambda Labs: GPU-specialist clouds. 30-50% cheaper than hyperscalers for raw GPU time.
- Regional providers (Thai clouds): lower latency for Thailand-based inference, but limited GPU availability.
6. Strategic analysis and future trends
- Inference shifts to edge - training stays centralized, but inference moves closer to users (on-premise or regional edge)
- Sovereign AI clouds - nations building domestic AI infrastructure for data sovereignty (Thailand is investing here)
- Interconnect is the new moat - raw FLOPS are commoditizing; the network connecting chips is the differentiator
- Energy becomes the constraint - data center power availability, not chip supply, increasingly limits AI capacity
7. Conclusion and procurement recommendations
- Start with cloud VMs for experimentation and spiky workloads
- Measure utilization after 3 months; if sustained >70%, evaluate bare metal or reserved capacity
- Design for portability - use containers and standard frameworks so you can move between providers
- Negotiate reserved pricing - 1-year commitments save 30%+, 3-year save 50%+
- Consider hybrid - cloud for training, on-premise for inference, with shared data infrastructure
Devsign designs and deploys AI infrastructure - from cloud architecture to on-premise GPU clusters, optimized for your actual workloads.
