Local LLM Hardware Requirements: GPU, VRAM and RAM Sizing

Why Local LLM Hardware Requirements Are Different From Cloud AI

When a business calls a hosted API, hardware sizing is someone else’s problem, hidden inside a per-token price. Running models on your own machine puts the sizing decision on you, and getting it wrong either means overspending on a GPU you did not need or underspending and hitting out-of-memory errors the first time a real workload arrives. The good news is that LLM hardware requirements are predictable once you know the model size, the quantization level, and the context length you need.

Three components matter, in order of importance: GPU VRAM (by far the most common bottleneck), system RAM (matters more when you offload layers to CPU or run larger models than your GPU can fully hold), and storage speed (matters mainly for model load times and swapping between multiple models).

VRAM Requirements by Model Size and Quantization

VRAM need scales with parameter count and precision. As a rule of thumb, each billion parameters needs about 2 GB of VRAM at 16-bit precision, about 1 GB at 8-bit, and roughly 0.5-0.6 GB at 4-bit, plus overhead for the context window and KV cache. In practice, that works out to:

Add a further 10-20% headroom on top of the base weight size for the KV cache, which grows with context length and concurrent requests. A model that barely fits at a 4K context window can run out of memory the moment you push it to 32K.

GPU Options: Consumer Cards vs Data-Center Cards

Consumer and prosumer GPUs with 16-24 GB of VRAM cover the 7B-14B range comfortably and, at 4-bit quantization, stretch into the 30B range. Cards with 32-48 GB of VRAM open up 30B-70B models at 4-bit without needing a second card. Data-center GPUs with 80 GB or more of VRAM, along with high-bandwidth interconnects like NVLink, are what production deployments reach for when serving multiple concurrent users or running 70B-class models at higher precision.

The right choice depends on the workload, not on buying the biggest card available. A team running an internal assistant for 20 employees on an 8B model has different needs than one serving a customer-facing chatbot to thousands of daily users on a 70B model, and sizing the GPU to the actual use case is where a lot of budget gets wasted in either direction.

CPU, System RAM, and Storage

CPU choice matters far less than GPU choice for inference, since the heavy computation happens on the GPU. Where CPU and system RAM matter is CPU offload: tools like llama.cpp can run part of a model’s layers on CPU when it does not fully fit in VRAM, and that path needs system RAM roughly equal to the portion of the model being offloaded, plus the usual overhead for the operating system and any other services. As a practical floor, 32 GB of system RAM is comfortable for 7B-13B work, and 64-128 GB gives real headroom for larger models or CPU-offload scenarios.

Storage matters more than people expect. Model files range from a few gigabytes to well over 100 GB for larger checkpoints, and loading from a slow drive adds real time to every cold start or model switch. An NVMe SSD with several hundred gigabytes to a few terabytes of free space is the practical baseline for a team that expects to keep more than one or two models on hand.

Single GPU vs Multi-GPU Setups

A single GPU is the right starting point for any model that fits comfortably in its VRAM with room for context growth. Multi-GPU setups become necessary in two situations: the model itself is too large for one card even at 4-bit (most 70B and larger models), or the business needs to serve enough concurrent users that a single GPU’s throughput is not enough, regardless of whether the model fits. Multi-GPU inference uses tensor parallelism to split each layer’s computation across cards, which requires a serving engine built for it and, ideally, a fast interconnect between GPUs rather than relying on the PCIe bus alone.

The Cloud-GPU Alternative

Renting GPU instances from a cloud provider is a reasonable way to prototype before committing to hardware, and it removes the upfront capital cost. But for a business planning to run models continuously, the ongoing hourly cost, the variability of GPU availability, and the fact that data may transit through third-party infrastructure are all real drawbacks. Companies with steady, predictable inference workloads and any sensitivity around data residency tend to land on owned hardware once they get past the pilot stage, since the cost curve favors ownership at sustained utilization and it keeps every prompt and document on infrastructure the company controls.

Right-Sizing Hardware for Your Business

The most common mistake is sizing hardware for the biggest model a team might theoretically want someday, rather than the model that fits the workload today. An 8B or 14B model, correctly quantized, handles the large majority of internal business use cases, including support drafting, document Q&A, summarization, and structured extraction, and it does so on hardware that costs a fraction of what a 70B-capable rig requires. Scaling up should be driven by a measured gap in output quality on your own tasks, not by defaulting to the largest model available.

Getting this sizing decision right the first time, and building a deployment that can grow without a full hardware replacement, is what private, local Ollama deployment for business is built for. It matches the model, quantization, and GPU to the workload rather than guessing.