Magnitude launches open-source inference engine that tunes itself to local hardware for AI agents
The YC-backed startup says its Rust engine decodes up to 2x faster than llama.cpp by compiling kernels for the exact device it runs on, and it's free under Apache 2.0.
Magnitude, a startup out of Y Combinator's Summer 2025 batch, has released an open-source inference engine that compiles and tunes GPU kernels on the user's own hardware before a model ever runs. The company claims up to twice the decode speed of llama.cpp, with measured gains of 92% on an M4 Pro Mac and roughly 18% on Nvidia's DGX Spark. The engine is written in Rust, ships under Apache 2.0, and runs on Mac, Linux, and Windows.
The founders are two engineers who previously built an open-source browser agent, and their pitch targets a gap they say existing engines leave open. Datacenter systems such as vLLM and SGLang are optimized for batched throughput and give up single-session latency, while llama.cpp and Ollama chase broad compatibility rather than tuning for one machine. Magnitude's third path keeps kernels parameterized, then compiles and autotunes them on the actual device, so it supports a wide range of hardware without giving up the performance ceiling of hand-written, hardware-specific code.
The team writes kernels for the most popular open-weight model families instead of trying to cover everything. That focus is deliberate: kernel coverage, not raw architecture, is what currently separates Magnitude from engines built to handle any model you throw at them. The company credits FlashAttention, FlashInfer, and TurboQuant as influences, along with engine-level ideas like radix attention from SGLang.
Memory handling is the other half of the design. Magnitude reserves only enough memory upfront to hold model weights, then grows its heap as agent sessions expand and releases it when agents stop, so the machine stays usable for other work. A hybrid paged attention scheme borrows prefix-cache sharing from SGLang but arranges placement for memory adjacency, meaning a single session does not pay for concurrency it isn't using.
The benchmark numbers, run against llama.cpp with Qwen 3.6 35B A3B at 4-bit precision and 64k context without speculative decoding, vary by platform. On an M4 Pro Mac with 48 GB using Metal, Magnitude reported decode rising from 30 to 57 tokens per second, prefill up 9%, and per-agent memory down 28%. On an Nvidia DGX Spark over CUDA, decode went from 49 to 58 tokens per second and prefill from 2,033 to 2,507 tokens per second, with memory use 27% lower per agent.
Magnitude ships as a desktop app that connects to agents users already run, including Pi, OpenCode, Hermes, and Codex. It starts models on demand when an agent needs one and shuts them down after inactivity. The roadmap lists expert streaming to run models larger than GPU memory by loading experts just in time, a fuller kernel compiler that picks fusions and implementations automatically, and multi-device scheduling that spreads a model across CPU, GPUs, RAM, and disk.
The open question is whether on-device autotuning holds its edge as model architectures multiply. Kernel coverage for popular families is what gives Magnitude its current speed advantage, and every new architecture that lands outside that set erodes the lead unless the autotuner can generalize. Watching how quickly the team adds kernel support will say more about the engine's long-term position than any single benchmark.