Local LLM hardware cost efficiency

Local Large Language Model (LLM) hardware efficiency is primarily determined by the relationship between memory bandwidth and the cost of the hardware required to host a specific model size Verified Answer #4. Because the processor must retrieve the entire model's weights from memory to generate each individual token, autoregressive decoding is heavily memory-bandwidth bound rather than compute-bound Verified Answer #2. The maximum achievable tokens-per-second (TPS) can be estimated by dividing the sustained memory bandwidth by the resident model size in bytes Verified Answer #3.

Efficiency by Model Class

8B Parameter Models

For 8B-class models, the highest value is typically found in single mid-range discrete GPUs with 12–16 GB of VRAM Verified Answer #1. An 8B model quantized to 4-bit (Q4) requires approximately 4–5 GB of VRAM, allowing it to run entirely on a single GPU and avoid performance penalties from PCIe-based inter-GPU communication or system RAM offloading Verified Answer #4Verified Answer #2.

• Value Winner (Used): The Intel Arc B580 12GB is cited as a top value performer, offering approximately 0.27 to 0.37 TPS per dollar when purchased used Verified Answer #2.
• Value Winner (New): The RTX 5060 Ti 16GB provides high throughput of 80–110 TPS, resulting in a rating of approximately 0.22 TPS per dollar Verified Answer #4.
• Alternative: Used RTX 3060 12GB or RTX 2080 Ti-class builds are also competitive for pure cost efficiency Verified Answer #1.

70B Parameter Models

Efficiency drops significantly for 70B models due to a "VRAM capacity cliff" Verified Answer #2. A 70B model at 4-bit quantization requires 40–45 GB of VRAM, which exceeds the capacity of any single consumer GPU, including the RTX 5090 Verified Answer #4.

• Consumer Value Winner: A dual-RTX 3090 24GB build (used) is the most cost-effective consumer configuration, providing 48 GB of total VRAM and high aggregate bandwidth Verified Answer #1. This setup typically achieves 15–20 TPS at a rating of roughly 0.007 TPS per dollar Verified Answer #4.
• New Hardware Alternative: Dual RX 7900 XTX 24GB cards are the primary mainstream alternative for new hardware, though they may face less straightforward software support than NVIDIA options Verified Answer #1.
• Unified Memory Systems: The Mac Studio M4 Max can run 70B models but offers lower efficiency at approximately 0.003 TPS per dollar due to its higher upfront capital expenditure Verified Answer #4.

Performance Considerations

Achieving high speeds of 60–120+ tokens per second is relatively easy for 8B models on most modern hardware tiers Verified Answer #3. For 30B to 32B models, a single RTX 5090 is considered a price-to-performance sweet spot that can approach these speeds Verified Answer #3. However, for 70B dense models, a dual RTX 5090 setup typically reaches only 40–65 tokens per second without the use of speculative decoding or specialized enterprise hardware Verified Answer #3. Large Mixture-of-Experts (MoE) models, such as DeepSeek-V4-Flash, require significantly more resources, needing either 192 GB of distributed GPU memory or a large unified-memory system to accommodate their 160 GB footprint Verified Answer #3.