A 284 billion parameter model now runs on 64 gigabytes of RAM, on a desktop CPU, with no graphics card. A 4 billion parameter model runs on a phone. Between those two facts sits the question every buyer actually asks: what AI can I run on the hardware I already own. This scoreboard answers it with eight models, five RAM tiers, and a source for every number.
The ladder
Read the table bottom to top, or top to bottom: the point is the range. The smallest model here fits in about 3 gigabytes. The largest holds 284 billion parameters in 56.56 GiB, about 61 gigabytes. Every row carries the file size as shipped, the RAM the machine needs, the reported decode speed with its source, and the context window. Speeds come from different hardware and different measurers, so they are labeled per row and never averaged. Two rows have no published local speed at all, and the table says so rather than inventing one.
| Model | Total / active params | File size | RAM needed | Reported decode speed | Context | License |
| Qwen3.5-4B | 4B / 4B | 3.3 GB (Ollama) | No vendor minimum stated; 3.3 GB build | Not published | 262,144 tokens | Apache 2.0 |
| Gemma 3 4B | 4B / 4B | 2.2 GB (Q4) | 8 GB RAM | About 26 tok/s (Mac M4, Q4) | 128,000 tokens | Gemma License |
| Nemotron 3.5 Bolt E4B | 8B / 4B effective | 6.18 GB (GGUF) | 4 GB (RTX 5070 3.5 GB) | 133.4 tok/s (RTX 5070, NVIDIA) | 64,000 tokens | OpenMDW-1.1 |
| Mistral Small 3.1 24B | 24B / 24B | 12.8 GB (Q4_K_M) | 24 GB card or 32 GB Mac | 150 tok/s (vendor claim) | 128,000 tokens | Apache 2.0 |
| Qwen3-30B-A3B | 30.5B / 3.3B | 13.70-17.28 GiB | 16-32 GB RAM | 34.80 tok/s (0 GPU, repo) | Not stated in the report | Apache 2.0 |
| Nemotron 3.5 30B A3B | 30.2B / 3.0B | 14.35 GiB (Fractal-64) | 16-32 GB RAM | 25.0-60.4 tok/s (CPU, repo) | 256,000 tokens | OpenMDW-1.1 |
| Gemma 3 27B | 27B / 27B | 17-18 GB (Q4) | 24 GB VRAM | Not published | 128,000 tokens | Gemma License |
| DeepSeek V4.1 Flash | 284.33B / 13.17B | 56.56 GiB (1.71 bpw) | 64 GB RAM, CPU only | 9.06-9.95 tok/s (lossless, repo) | 1,000,000 tokens | DeepSeek License |
What the 284B result actually proves
The headline row comes from a patched llama.cpp build, llama-modchip version 2.4.0, published on GitHub on October 29, 2026. The project transcodes DeepSeek V4.1 Flash from Unsloth's 76.87 GiB UD-IQ1_S build down to 56.56 GiB at 1.71 bits per weight, then runs it on a 16-core Zen 5 desktop CPU, a Ryzen 9 9955HX, with zero GPUs. Peak memory use is 57.61 GiB at a 256,000-token context, which fits inside the roughly 62.5 GiB a 64 GB Linux machine can actually use. Prefill runs at 44.36 tokens a second, and decode at 9.06 to 9.95 tokens a second with all six routed experts evaluated, rising to 11.5 to 14.7 with speculative drafting. Stock llama.cpp rejects the custom tensor type, so the patched binary is required, and the project's "lossless" claim means runtime fidelity against the same checkpoint on unpatched llama.cpp, a 1.00000 logit cosine similarity, not quality against the full-precision model. At 1.71 bits per weight, the quality-versus-BF16 question is unanswered by the project, and this scoreboard does not answer it either.
Independent builders report slower numbers on other setups, which is exactly what a buyer should expect. A community gist running the stock approach with SSD expert-streaming measured about 1.9 tokens a second on a 62 GiB Strix Halo machine. A computing guide measured 5.61 tokens a second for a 3-bit build on a 16-thread Xeon with 123 GiB of RAM. A DGX Spark setup measured about 37 to 38 tokens a second with unified memory. The spread is the point: the speed depends on the quant, the CPU, and the memory bandwidth, not on the parameter count alone. At 9 tokens a second, a 500-token reply takes about 55 seconds, so the 284B row is batch and offline work, not chat. Our report on the modchip build carries the full speed table across all four setups, and the same project's 30B row, Nemotron 3.5 30B A3B at 25.0 to 60.4 tokens a second on a CPU, is the more practical frontier for most machines.
The 4GB tier is the real market
The bottom of the ladder is where most people should start, and it is also where the newest models are. NVIDIA's Nemotron 3.5 Bolt E4B, announced October 23, 2026, is an 8 billion parameter model with 4 billion effective, post-trained from Google's Gemma 4 E4B, in a 6.18 gigabyte file that runs at 133.4 tokens a second on an RTX 5070 and 15.5 on a Jetson Orin Nano. Google's Gemma 3 4B, from March 2025, fits in about 2.2 gigabytes at Q4 quantization, runs on 8 gigabytes of RAM, and handles images as well as text with a 128,000-token context. Alibaba's Qwen3.5-4B, released March 2, 2026, is a 4 billion parameter vision-language model with thinking on by default, a 262,144-token native context that extends to 1,010,000 with YaRN, and 6,376,702 Hugging Face downloads in the 30 days to November 2, 2026. The 4GB tier is the one with the most choice, the most downloads, and the least hardware risk, which is why our deep dive on the 4GB tier spent a whole post on a single model, the Bolt E4B.
The middle tiers
The middle of the ladder is where the trade-offs get interesting. Mistral Small 3.1 24B, from March 2025, is a dense 24 billion parameter model with a 128,000-token context under Apache 2.0, running on a single RTX 4090 or a 32 gigabyte Mac at a vendor-claimed 150 tokens a second, and it outperforms Gemma 3 27B on most benchmarks according to Mistral's own comparisons. Gemma 3 27B, Google's most capable consumer variant, needs about 17 to 18 gigabytes at Q4 and a 24 gigabyte card, with no published local speed to report. The two 30 billion parameter mixture-of-experts models, Nemotron 3.5 30B A3B and Qwen3-30B-A3B, both fit a 16 to 32 gigabyte machine because only about 3 billion parameters fire per token, and both report CPU decode speeds in the 25 to 60 token a second range from the same project. The lesson of the middle tiers is that total parameters stopped being the number that matters. Active parameters, bits per weight, and memory bandwidth decide what runs, which is why a 30 billion parameter model can outrun a 24 billion parameter dense model on the same machine. Our open-weight benchmark wars pillar compares the same models on capability rather than hardware fit, and the two tables answer different questions.
Three tests before you buy RAM
First, buy RAM for the model, not the model for the RAM. Pick the job, find the smallest model that does it well, then buy the RAM that model needs. Buying a 64 gigabyte machine to run a 4 gigabyte model wastes money, and buying a 4 gigabyte model because it fits wastes the machine. Second, budget for the key-value cache at full context, because the weights are not the whole bill. Mistral Small 3.1 24B needs about 20 gigabytes of FP16 key-value cache alone at its full 128,000-token context, on top of about 13 gigabytes of weights. Gemma 3 4B needs about 4 to 5 gigabytes extra for a 32,000-token context. A model that fits at 8,000 tokens may not fit at 128,000. Third, ask for the quality-versus-BF16 number before trusting a 1 to 2 bit-per-weight build. The 284B result is honest about what it proves, runtime fidelity against the same checkpoint, and silent about what it does not, quality against full precision. Any build that does not publish that comparison is asking you to trust the quant, and the only honest answer is to test it on your own prompts before you commit.
By ByteHerald Staff. The llama-modchip project, version 2.4.0, published on GitHub on October 29, 2026, reports that DeepSeek V4.1 Flash, at 284.33 billion total parameters and 13.17 billion active, runs in 56.56 GiB at 1.71 bits per weight with decode speeds of 9.06 to 9.95 tokens a second on a 16-core Zen 5 CPU, and that Nemotron 3.5 30B A3B runs at 25.0 to 60.4 tokens a second in 14.35 GiB. NVIDIA announced Nemotron 3.5 Bolt E4B on October 23, 2026, as an 8 billion parameter model with 4 billion effective, a 6.18 gigabyte file, and 133.4 tokens a second on an RTX 5070, under the OpenMDW-1.1 license. Google released Gemma 3 in March 2025, with the 4B model at about 2.2 gigabytes in Q4 quantization and the 27B model at about 17 to 18 gigabytes, both with a 128,000-token context window. Alibaba released Qwen3.5-4B on March 2, 2026, as a 4 billion parameter model with a 262,144-token context window under Apache 2.0, with 6,376,702 Hugging Face downloads in the 30 days to November 2, 2026. Mistral AI announced Mistral Small 3.1 in March 2025 as a 24 billion parameter model with a 128,000-token context window under Apache 2.0, running on a single RTX 4090 or a 32 gigabyte Mac at a vendor-claimed 150 tokens a second.
Comments
Post a Comment