Explore the strengths and optimal use cases of `llama.cpp` and `vLLM` for local LLM inference. `llama.cpp` excels on consumer hardware with optimizations like quantization and CPU support, while `vLLM` is designed for production-scale efficiency with features like continuous batching and speculative decoding on high-performance accelerators.
Dr. Lech Kalinowski presents Vault, a unique AI-native handheld terminal built on an ESP32-S3, featuring a dual-display (OLED for live interaction, e-paper for static content) and powered by a single battery. It operates in four modes—shell, assist, control, and an LLM-native RPG—and controls autonomous OpenClaw agents against a local, self-hosted 120B parameter GPT model served by NVIDIA TensorRT-LLM. The talk delves into the AI-native architecture, why inference remains on the backend, real-time agent thinking surfacing on e-paper, engineering challenges, and LLM-native game design, highlighting the potential for calm, text-first AI interaction.