Tokenless

Local inference

What's New in Inference Engineering — Philip Kiely, Baseten

What's New in Inference Engineering — Philip Kiely, Baseten

Philip Kiely provides an update on recent advancements in inference engineering, covering new techniques in quantization (TurboQuant's specific application), KV cache compaction (introducing 'Still' for learned, compressed memory), and speculative decoding (from Eagle 3 to Dlash and DSpark, including continuous retraining). He emphasizes the blurring lines between training and inference and differentiates between local and data center optimization strategies.

Llama.cpp vs vLLM: Which Local LLM Engine Actually Scales?

Llama.cpp vs vLLM: Which Local LLM Engine Actually Scales?

Explore the strengths and optimal use cases of `llama.cpp` and `vLLM` for local LLM inference. `llama.cpp` excels on consumer hardware with optimizations like quantization and CPU support, while `vLLM` is designed for production-scale efficiency with features like continuous batching and speculative decoding on high-performance accelerators.

OpenClaw in Your Hand: Building a Physical AI Terminal - Lech Kalinowski, Callstack

OpenClaw in Your Hand: Building a Physical AI Terminal - Lech Kalinowski, Callstack

Dr. Lech Kalinowski presents Vault, a unique AI-native handheld terminal built on an ESP32-S3, featuring a dual-display (OLED for live interaction, e-paper for static content) and powered by a single battery. It operates in four modes—shell, assist, control, and an LLM-native RPG—and controls autonomous OpenClaw agents against a local, self-hosted 120B parameter GPT model served by NVIDIA TensorRT-LLM. The talk delves into the AI-native architecture, why inference remains on the backend, real-time agent thinking surfacing on e-paper, engineering challenges, and LLM-native game design, highlighting the potential for calm, text-first AI interaction.