Own Your AI
Practical guides to running AI on your own hardware.
Project: Adaptive Quantization with Self-Healing
The goal is to shrink powerful open-source models so they run well on common consumer hardware — laptops, desktops, and small workstations — without making them noticeably worse.
Adaptive quantization means the model picks the right precision for each layer on the fly. Important layers stay at higher precision; less sensitive layers drop to lower precision. That cuts memory and speeds up inference where it matters least.
Self-healing means the system watches its own output quality. When it detects that a cheaper precision is hurting answers, it bumps those layers back up, repairs the degraded weights, and keeps going. The model learns to stay small and fast while still giving trustworthy results.
Adaptive precision
Layers are quantized selectively instead of applying one blunt setting to the whole model.
Self-healing quality
Automatic rollback and repair when compression starts to hurt performance.
Consumer hardware target
Tuned for the GPUs, APUs, and unified-memory machines people already own.
Roadmap
- Proof of concept: Demonstrate adaptive quantization on a 7B–13B model with measurable quality recovery.
- Open tooling: Publish scripts and guides so others can reproduce the pipeline on their own hardware.
- Hardware targets: Tune recipes for NVIDIA, AMD/Strix Halo, and Apple Silicon machines.
- Public benchmarks: Share honest speed, memory, and quality measurements.
Why own your AI?
Privacy by default
Your documents, photos, and conversations stay on your device. They are not mined, logged, or used to train someone else's model.
No vendor lock-in
Use open models, swap them when you want, and keep working even if an online service changes its rules or prices.
Works offline
Local AI runs without an internet connection, making it more reliable and resilient for everyday tasks.
Local AI stack
A running snapshot of hardware and software for local inference. Prices, speeds, and features are estimates — update the data files as you test new gear or tools.
| Hardware | Memory | Price (USD) | Speed (8B Q4) | Speed (70B Q4) | Max model @ 4-bit, 30K ctx |
|---|---|---|---|---|---|
| NVIDIA RTX 4080 (16 GB) | 16 GB | $1,200 | ~106 tok/s | N/A | 8B |
| NVIDIA RTX 2080 Ti 22 GB (mod) | 22 GB | $500 | ~45 tok/s | ~2 tok/s | 14B |
| NVIDIA RTX 3090 (24 GB) | 24 GB | $800 | ~70 tok/s | ~8 tok/s | 32B |
| NVIDIA RTX 4090 (24 GB) | 24 GB | $2,000 | ~130 tok/s | ~10 tok/s | 32B |
| AMD Ryzen AI Max+ 395 (Strix Halo, 128 GB) | 128 GB | $2,500 | ~48 tok/s | ~32 tok/s | 70B |
| NVIDIA DGX Spark (128 GB unified) | 128 GB | $4,000 | ~39 tok/s | ~3 tok/s | 70B–100B |
| Apple Mac Studio M4 Max (128 GB) | 128 GB | $2,500 | ~90 tok/s | ~12 tok/s | 70B |
| Apple Mac Studio M3 Ultra (512 GB) | 512 GB | $7,000 | ~135 tok/s | ~12 tok/s | 400B+ |
How to read this table
Speeds are single-user decode estimates for a 7–8B model (small) and a 70B model (large), both at 4-bit quantization, with a 30K-token context window. Actual numbers depend on the runtime (llama.cpp, Ollama, MLX, vLLM, etc.), cooling, quantization format, and prompt length. Max model size assumes ~0.5 bytes per parameter for 4-bit weights plus a 30K-token KV cache and a few gigabytes of system overhead. It is the largest model that can fit entirely in memory without offloading to slower storage.
Update src/data/inferenceHardware.js as you test new hardware or run fresh benchmarks.
Want to follow along?
This project will be documented on the blog as it progresses. Expect deep dives, failures, lessons learned, and working code.
Visit the Blog