Local LLM hardware cost efficiency

Local Large Language Model (LLM) inference performance is primarily determined by the relationship between memory capacity and memory bandwidth Verified Answer #2. For single-user workloads, the token generation stage is memory-bandwidth bound rather than compute-bound Verified Answer #1. Token generation speed (TPS) scales linearly with the effective memory bandwidth of the hardware, provided the model weights fit entirely within the available VRAM or unified memory Verified Answer #2.

Performance Metrics and Calculation

The maximum achievable TPS for a model can be estimated by dividing the hardware's memory bandwidth (GB/s) by the size of the quantized model (GB) Verified Answer #1. Hardware cost efficiency is measured by the TPS per dollar (TPS/$), which is optimized by selecting the lowest-cost hardware that meets VRAM requirements while maximizing bandwidth Verified Answer #1.

Efficiency by Model Size

8B Parameter Models

Models with 8 billion parameters quantized to 4-bit (Q4) require approximately 4 to 5 GB of VRAM Verified Answer #2Verified Answer #1. To account for system overhead and the growth of the KV cache during inference, hardware with 12 GB to 16 GB of VRAM is considered the practical standard to avoid performance degradation Verified Answer #1.

• Intel Arc B580 (Used): This configuration is a leading value option, costing approximately $150–$180 and providing 45–55 TPS, resulting in an efficiency rating of roughly 0.27 to 0.37 TPS/$ Verified Answer #1.
• Intel Arc B580 (New): At an MSRP of $249, this card offers 456 GB/s of bandwidth and an efficiency of approximately 0.16 to 0.22 TPS/$ Verified Answer #1.
• RTX 5060 Ti 16GB (New): Costing approximately $450, this hardware achieves 80–110 TPS with an efficiency rating of approximately 0.22 TPS/$ Verified Answer #2.

The Intel Arc B580 utilizes Xe2 Battlemage architecture with dedicated XMX matrix engines, though it may require more technical effort to configure driver environments like SYCL or OpenVINO on Linux compared to other platforms Verified Answer #1.

70B Parameter Models

Models with 70 billion parameters quantized to 4-bit require between 40 and 45 GB of VRAM Verified Answer #2. These models face a "VRAM capacity cliff" because no single consumer GPU provides enough capacity with sufficient headroom for the KV cache Verified Answer #2Verified Answer #1.

• Dual Used RTX 3090: This setup costs approximately $2,600 and delivers 15–20 TPS, with a low efficiency rating of approximately 0.007 TPS/$ Verified Answer #2.
• Mac Studio M4 Max: This system costs approximately $3,500 and delivers 8–15 TPS, resulting in an efficiency rating of approximately 0.003 TPS/$ Verified Answer #2.

The Value Gap

A significant efficiency gap exists between small and large models Verified Answer #2. 8B models are highly cost-effective because they can run on a single mid-range GPU, avoiding the need for expensive inter-GPU communication or slow offloading to system RAM Verified Answer #2. In contrast, 70B models require high capital expenditure for multi-GPU builds or premium high-memory systems, which drastically reduces the TPS generated per dollar spent Verified Answer #2.