The trick is selective movement.
Janas-LLM is an experimental C inference engine for mixture-of-experts models. Its repository says model weights stay on disk, needed experts stream from an NVMe SSD, and frequently reused experts remain in a memory cache. The project reports running a 48.4 GiB Qwen3-Next-80B-A3B file on one 32 GB laptop.[1]
That is not the same as loading an ordinary dense model from a slower memory tier. A mixture-of-experts model has a learned router. Each token activates a subset of expert feed-forward networks while shared parts of the model still run for every token. Hugging Face's MoE guide explains the useful split: active compute can resemble a smaller dense model even though the complete parameter set remains much larger.[2]
The useful question is not "Does the file fit?" It is "Which bytes move for this workload, and how often?"
A cache miss is part of token latency.
The Janas source at revision 98c8300 stores each routed expert in one aligned slot. Its change log describes one direct read per slot, parallel cache-miss reads, least-recently-used replacement, and protection against evicting experts used by the current request. The README says the engine uses O_DIRECT and can read the disk at full speed.[1]
That design makes the storage path visible. Reused experts can become cache hits. New routing choices can trigger reads. The model file size alone cannot predict either rate. Prompt content, expert popularity, cache capacity, drive throughput, CPU work, GPU work, and transfer overlap all matter.
Prompt speed and reply speed are different tests.
Janas publishes prompt-processing and reply-generation figures separately. Its README also says the reported numbers come from one development laptop, and that the project has not been tested across broad hardware. The page warns that the cache can pressure the desktop into swap and that one GPU run caused a driver reset.[1]
The Hacker News thread found the seam immediately. One comment questioned whether the displayed rate was enough for prompt preprocessing and agent work. That comment is not benchmark evidence, but it points to the right test. An agent repeatedly feeds tool output and expanding context into the model. A chat demo with a short prompt does not cover that workload.[4]
Offload has a workload shape.
Microsoft's ZeRO-Inference work gives a useful comparison. It streams model layers from CPU memory or NVMe into GPU memory. Its published design targets throughput work with long sequences or large batches, where compute can hide part of the transfer time. The authors say latency-sensitive, small-batch work is better served by approaches that keep the full model in GPU memory when possible.[3]
Janas uses sparse expert selection and a hot cache rather than moving every layer on every token. The mechanisms differ, but the operating rule survives. Offload trades capacity for movement. Benchmark the prompt length, concurrency, output length, and neighboring applications you will use.
Write the road test before the download.
MODEL_REVISION=[REQUIRED]
MODEL_FILE_GIB=[REQUIRED]
CACHE_CAP_GIB=[REQUIRED]
MEASURED_NVME_READ_GIB_S=[REQUIRED]
PROMPT_TOKENS=[REQUIRED]
OUTPUT_TOKENS=[REQUIRED]
PROMPT_TOK_S=[MEASURE]
GENERATION_TOK_S=[MEASURE]
EDITOR_AND_BROWSER_OPEN=[YES/NO]
SWAP_OR_DRIVER_EVENT=[RECORD]
REPEAT_ROUNDS=[REQUIRED]
RESULT=[NO VERDICT YET]Run the first test on an idle machine so the engine can reveal its best local choice. Run the acceptance test again with the real editor, language server, browser, and agent loop open. A local model that starves the rest of the cockpit did not solve the local workflow.