GPU vs Unified Memory Tradeoffs

The hardware you choose for LLM inference determines your cost floor, latency ceiling, and operational complexity. The two main contenders for local inference are NVIDIA discrete GPUs and Apple Silicon unified memory. Each has distinct advantages.

Discrete GPUs (NVIDIA)

NVIDIA GPUs dominate production inference for good reason. The A100 and H100 offer massive memory bandwidth (2.0 TB/s and 3.35 TB/s respectively), which is the primary bottleneck for LLM inference. Combined with mature software ecosystems (CUDA, vLLM, TensorRT-LLM), discrete GPUs deliver the highest throughput per dollar at scale.

Best for: Production serving with high concurrency, large model sizes (70B+), and environments where throughput matters more than cost-per-unit.

Drawbacks: High upfront cost, power and cooling requirements, and the need for specialised DevOps knowledge. An H100 costs $25,000+ to purchase or $2–3/hour to rent.

Unified Memory (Apple Silicon)

Apple's M-series chips (M2 Ultra, M4 Max, M4 Ultra) offer unified memory architectures where CPU and GPU share the same memory pool. This means a Mac Studio with 192GB unified memory can load a 70B parameter model that would require multiple discrete GPUs.

Best for: Development and testing, small-scale inference (1–10 concurrent users), experimentation with large models, and environments where power efficiency and silence matter.

Drawbacks: Lower memory bandwidth than enterprise GPUs (800 GB/s on M4 Ultra vs 3.35 TB/s on H100), no CUDA ecosystem, and limited batch processing capability.

The Practical Decision

For production inference serving hundreds or thousands of concurrent requests, discrete GPUs win on throughput and cost efficiency. For local development, prototyping, and small-scale deployment, Apple Silicon offers a compelling combination of model capacity, power efficiency, and simplicity. Many teams use both: Apple Silicon for development and testing, NVIDIA GPUs for production.