Quantize

A wrapper plus web UI around llama.cpp tuned for single board computers like the Raspberry Pi. It streams tokens over websockets and stays under the Pi's tiny memory budget using Q4 quantization and mmap, so a 3B model actually fits in 2GB. Tokens per second are slow, but for a home assistant that mostly answers short questions it is real-time enough to feel responsive. I wanted something that ran entirely on hardware I own: no API keys, no usage caps, and no sending my house's voice commands off to someone's cloud. Setup is a single script that auto-detects the board. Local Whisper for voice input is the next milestone.