Summary
This video guides users on selecting appropriate local AI models based on VRAM capacity, emphasizing practical performance over raw size. It details model recommendations for various memory tiers (4GB to 128GB+), explaining that download size is misleading due to overheads like context and inference. Key advice includes prioritizing models that run comfortably with sufficient room for context, testing real-world performance, and ensuring compatibility with chosen runtimes.
Sections
Introduction: The Misleading Simplicity of Choosing Local AI Models
Choosing local AI models is complex; file size doesn't reflect real memory needs.
Users often believe selecting a local AI model is as simple as checking VRAM and downloading a file, but this leads to crashes, slow performance, or worse results than smaller models. The actual memory requirement is much higher than the download size due to additional needs for context, temporary calculations, and the inference engine.
Real model memory use exceeds download size due to context and overhead.
A model like the 1-bit Bonzai 27B downloads at ~3.9 GB, but with a 4,000 token context, its usage rises to ~5.2 GB, and over 11 GB at 100,000 tokens. This increase is due to the growing working memory required for longer conversations, not an actual increase in model size.
Shift from 'largest possible' to 'strongest comfortable' model for actual workload.
Instead of asking what the largest model can be squeezed onto a GPU, the better question is to determine the strongest model that can run comfortably for the user's typical workload and context length.
Models for 4-8 GB VRAM
Gemma 4 E4B is a reliable choice for 4-8 GB VRAM, supporting multimodal tasks.
For 4-8 GB machines, Gemma 4 E4B is the safest recommendation. It supports text, images, and audio, includes reasoning and function calling, and boasts a 128,000 token context window. While the full context isn't practical on small GPUs, it offers flexibility for chat, summaries, light coding, and image tasks.
Wider model family support is crucial for practical, reliable local AI deployment.
Gemma's broad support across common runtimes like LM Studio, Ollama, or llama.cpp is a significant advantage. A slightly weaker model that functions correctly within these established ecosystems is often more useful than a technically superior model requiring custom setups.
Nanobage 4.2-3B offers experimental potential for high performance on small models.
Nanobage 4.2-3B (approx. 4B parameters) is an experimental option that reuses transformer layers for more computation without proportional size increases. Its creators report strong coding and reasoning results, but runtime support is less mature, potentially requiring custom code.
Bonsai 27B's extreme compression offers fascination but isn't an ideal everyday 8GB model.
Bonsai 27B, a 1-bit compressed version of Qwen 3.6-27B, reduces the model size to ~3.9 GB from ~54 GB. While an impressive compression feat retaining ~89.5% performance, it's not the best everyday model for an 8GB GPU, especially for long-horizon tasks, making Gemma a safer recommendation for most users.
Models for 12-16 GB VRAM
Gemma 4 12B offers a balanced, strong performance for 12-16 GB VRAM.
At 12-16 GB, Gemma 4 12B is the balanced choice, supporting multimodal tasks, function calling, and offering stronger reasoning/coding than the smaller E4B. A 4-bit version should provide enough room for reasonable context and other applications.
Prioritize headroom for context and applications over maximum model size.
Having remaining memory headroom is crucial. A slightly smaller model that runs without crashing, especially with longer contexts, is generally more useful than a larger model that fails under load.
Ternary Bonsai 27B provides high capability per gigabyte for adventurous users.
The ternary version of Bonsai 27B uses three states per weight, preserving more performance (~94.6%) than the 1-bit version. With a packed size of ~5.9 GB (deployed ~7.2 GB), it uses ~8.4 GB at short contexts. It offers much of a 27B model's behavior in a smaller footprint but requires specialized runtime support.
Models for 24 GB VRAM
Qwen 3.6-27B (Q4) is the consumer sweet spot for 24 GB VRAM.
24 GB VRAM is ideal for running strong 27B parameter models at 4-bit quantization. Qwen 3.6-27B in a Q4 build uses ~17-18 GB, leaving room for context. It's suitable for serious coding, reasoning, and document analysis, with broad inference framework support.
Thinking Cap offers efficiency gains by reducing token usage, but with variable quality.
Thinking Cap, a fine-tuned Qwen 3.6-27B, aims to reduce reasoning tokens (~50% fewer on average), potentially lowering latency and power usage. However, it may not preserve quality equally across all tasks, making it an efficiency trade-off that should be tested against the standard Qwen model for specific use cases.
Models for 32-64 GB VRAM
At 32-64 GB, focus shifts from quantization necessity to model choice based on workload.
With ample VRAM, the primary concern is less about fitting heavily quantized models and more about selecting the best model for the task. Higher precision becomes possible, but specialized models might still outperform larger general ones.
Gemma 4T 31B offers a strong general-purpose multimodal assistant for 32 GB+ systems.
Gemma 4T 31B is practical around 32 GB VRAM, providing a mature ecosystem and capacity for demanding tasks. It includes text, image, and audio support.
Mixture-of-experts (MoE) models balance large knowledge capacity with lower active parameter computation.
Gemma's 26B A4B MoE model stores ~25B parameters but activates only ~3.8B per token. This allows large knowledge storage with computational efficiency per token, though all experts still require storage.
Choose models in the 32-64 GB tier based on workload priorities: coding, general multimodal, or speed.
For 32-64 GB, select Qwen 3.6-27B for coding/reasoning, Gemma 4T 31B for a general multimodal assistant, or the Gemma MoE model when generation speed is prioritized over maximum quality extraction.
Models for 128 GB VRAM and Beyond
Poolside's Laguna S 2.1 (MoE) is a specialized 128 GB option for agentic coding.
Laguna S 2.1, an 118B parameter MoE model activating ~8B per token, is designed for agentic coding. It boasts a context window over 1 million tokens and strong reported performance on coding benchmarks, though vendor benchmarks require real-world verification.
128 GB systems require practical quantization; FP8 weights leave insufficient room for overhead.
Laguna's FP8 weights (~121 GB) may appear to fit a 128 GB machine, but this leaves minimal space for the OS and inference engine. A lower-bit quantization is crucial for practical operation, even on high-capacity systems.
Distinguish between memory capacity (loading model) and bandwidth/compute (generation speed).
Systems like Nvidia's DGX Spark offer large unified memory (e.g., 128 GB), but lower memory bandwidth compared to discrete GPUs means slower token generation. Capacity determines if a model can load; bandwidth and compute determine how fast it runs.
Server-class models like Motif 3 are generally not practical recommendations for desktop users.
Beyond 192 GB, server-class models like Motif 3 (314B MoE) exist but are typically beta releases, require data center GPUs, and have licensing restrictions (e.g., non-commercial use). Practicality for most developers lies in stable 24, 32, or 128 GB setups.
How to Choose Your Model: Practical Guidelines
Select the strongest model that fits comfortably, leaving space for context and runtime.
The best local model isn't the largest one that barely fits, but the strongest one that runs comfortably, ensuring sufficient memory for the actual context length and inference engine overhead.
Match models to tasks: Gemma for general/multimodal, Qwen for coding, Bonsai for compression experiments.
Align model choice with the primary task: use mature Gemma builds for lightweight chat/multimodal work, Qwen 3.6-27B for serious coding (~24 GB), Bonsai for extreme compression exploration (with tradeoffs), and Nanobind for experimental coding agents.
Context length is critical; a model's advertised max context may not be usable on your hardware.
Verify how much context your hardware can actually support. A model advertised with a massive context window might only be usable at much shorter lengths depending on available memory.
Confirm runtime compatibility before downloading to avoid architecture or feature mismatches.
Before downloading, check if your chosen runtime (Ollama, LM Studio, etc.) supports the specific model's architecture, tool calls, and reasoning format. Model availability doesn't guarantee application compatibility.
Always test models with real work, not just benchmarks, for reliable daily performance.
Benchmarks provide a starting list, but only testing with your actual workload—debugging code, analyzing long documents, using tool calls, images, and reasoning tasks—can determine the best model for reliable daily use.
Adhere to three key rules: count runtime memory, leave room for context, and prioritize daily reliability.
Remember these three rules: 1. Account for runtime memory usage (not just download size). 2. Ensure sufficient space for your expected context length. 3. Choose the model that works reliably every day, rather than the one with the largest parameter count.
Ask a Question
*Uses 1 Wisdom coin from your coin balance
