Autoregressive Language Model on the 6502 Processor

A developer trained a tiny Mamba-based autoregressive language model and wrote a custom inference engine so it could run on the MOS 6502, the 8-bit microprocessor from 1975 that powered machines such as the BBC Micro and the Apple II. The demo ran on a BBC Model B from the 1980s with 32KB of RAM. The generated sample reads like a broken children’s story: once upon a time tom and lily saw things lily were sad her house he heartd them ilily and tom said yes she saw a little girl smiled tom was so excited her mom said yes. The result is not polished generation; it is proof that a modern-style language model can produce text at all on hardware that predates personal computing as we know it.

Fitting that stack into the machine is the hard constraint. User-space memory is limited to about 25KB for both code and weights. The final layout used 9KB for the inference engine and 13KB for the model weights. The 6502 only works with 8-bit integers and has no multiply instruction, so every multiply in the model has to be synthesized from the instructions that exist. That forces the implementer to rethink kernels, quantize aggressively, and keep the graph small enough that a Mamba-style autoregressive model still fits and runs.

For engineers and builders, the project is a concrete exercise in extreme resource limits rather than a novelty about old chips. It shows how far inference can be stripped when RAM is measured in kilobytes and arithmetic is primitive: weight layout, integer-only math, and a hand-written engine become first-class design decisions. Anyone shipping models to microcontrollers, firmware, or edge devices faces a softer version of the same problem—budget the binary, remove luxuries, and make every byte and cycle count.

In market terms, the work sits far from cloud-scale transformers and GPU clusters. Commercial language models assume abundant multiply-accumulate throughput, large memory, and floating-point units. This demo sits at the opposite pole: 1970s silicon, no multiply, tens of kilobytes. That contrast makes the BBC Micro run a useful reference point for how much quality and capacity vanish when you refuse modern hardware, and for how much of modern ML stacks are luxuries rather than necessities once the task is only “generate some text.”

The practical takeaway is the budget and the architecture choice: 9KB of inference code plus 13KB of weights for a Mamba-based autoregressive model on pure 8-bit integer hardware without multiply. Watch whether similar ports appear on other constrained platforms, whether the generated text quality improves under the same memory ceiling, and how much of the pipeline can stay integer-only without breaking training or inference. The next interesting step is not a bigger model on a GPU, but a clearer answer to how strong a language model can get inside that 25KB envelope.