The economics of “right-sized” models
API spend from always-on assistants adds up quickly. SLMs in the 1B–8B parameter range, quantized for CPU or NPU inference, handle classification, extraction, and templated generation at a fraction of the cost—with sub-second latency.
Where edge AI shines
- Manufacturing: visual inspection plus natural-language work instructions offline
- Healthcare: clinical note drafting on secure workstations without external calls
- Retail: inventory queries and staff copilots on store networks
Deployment toolkit
Teams combine ONNX Runtime, llama.cpp, and vendor NPUs with orchestration that routes hard questions to larger cloud models only when needed—a pattern often called cascade or router architectures.
Trade-offs to plan for
Narrower reasoning depth and more frequent fine-tuning cycles are expected. Success depends on tight task scope, high-quality domain datasets, and continuous eval on production traffic samples.