Mini-AGI trains a continual-learning MoE on 8GB of VRAM, one 32K-character stream at a time

A developer posting as volotat has released Mini-AGI, an experimental language model that trains on a single 8GB consumer GPU by rethinking two assumptions that usually force large runs onto datacenter hardware. The project combines a mixture-of-experts architecture whose pool of experts expands and contracts during training with batch-size-one updates drawn from one unbroken stream of text. The author is upfront that the name oversells what is still a work in progress, and no weights have been published yet.

The memory trick is the part worth watching. Conventional training keeps a large randomized batch of sequences and their gradients resident in VRAM at once, which is why a 1B-parameter run typically demands server-grade accelerators. Mini-AGI instead reads passages of roughly 32,000 characters in order, one at a time, the way a person moves through a book, and holds only a small subset of experts on the GPU at any moment. The remaining experts sit on disk and are swapped in or evicted as needed, so the practical ceiling on parameter count becomes available storage rather than VRAM.

That second choice cuts against years of accepted practice. Continual learning over a non-shuffled stream is the classic recipe for catastrophic forgetting, and the author does not claim to have solved the problem in general. He frames the design as a bet: if single-stream training holds up at scale, the memory cost of optimization itself falls sharply, and the model's exposure to data stays under the operator's control instead of a lab's. So far the main evidence offered is a scaling-law plot published alongside the code.

Training is still working through the first of seven corpora that total 7.8 billion characters, so there is no checkpoint to benchmark and no independent replication of the scaling curve. The author is also candid that Anthropic's Claude was used heavily during development, writing that the work would have been impossible for him alone, a disclosure that lands in a field where tooling assistance is routine but seldom described in a release post. He states a motive that is political as much as technical: frustration that individuals can fine-tune and run inference on large models but cannot afford to train one from scratch, which leaves alignment aligned with corporate rather than personal interests.

The code is public and simple enough to clone and run, which is the real invitation. The current corpus is expected to finish in roughly two weeks at the present reading rate. The open question is whether the scaling graph holds as the stream lengthens and the expert pool grows, or whether single-pass training starts shedding earlier knowledge faster than it accumulates new material.