Tokenless

Cuda

Weight Folding, CUDA Streams, and the Bug That Made My Model Speak Backwards — Filip Makraduli

Weight Folding, CUDA Streams, and the Bug That Made My Model Speak Backwards — Filip Makraduli

This talk introduces FlashNorm, an optimization for transformer RMS norm layers. It details how algebraic tricks like weight folding, deferred division, and dropped pre-normalization can significantly speed up inference by mitigating GPU inefficiencies. The speaker also shares critical insights from implementing parallel CUDA streams, including debugging a subtle race condition caused by implicit stream joins, and advocates for open inference engines for deploying kernel-level research.

Can LLMs Write Fast Multi-GPU Kernels? — Simran Arora, Together AI

Can LLMs Write Fast Multi-GPU Kernels? — Simran Arora, Together AI

Simran Arora discusses the critical bottleneck shift in large AI workloads from GPU compute to inter-GPU communication. Her team's solution, ParallelKittens, offers a set of primitives to optimize multi-GPU kernels by leveraging fundamental transfer mechanisms and compute-communication overlapping. They introduce ParallelKernelBench, a benchmark to evaluate AI models' ability to generate such kernels, revealing that while models can handle syntax, they struggle with deeper reasoning about communication patterns and hardware trade-offs.

Multi-GPU Kernels, Intelligence per Watt, Heterogeneous Inference, and More | YC Paper Club

Multi-GPU Kernels, Intelligence per Watt, Heterogeneous Inference, and More | YC Paper Club

This YC Paper Club explored the growing trend of specialization in AI hardware and software, covering multi-GPU kernel optimization, intelligence per watt metrics for local AI inference, the implications of AI writing systems code, heterogeneous hardware designs for inference, and GPU-accelerated game engines for reinforcement learning.

Why Hardware-Software Co-Design Is AI's Real 100x: Dylan Patel of SemiAnalysis

Why Hardware-Software Co-Design Is AI's Real 100x: Dylan Patel of SemiAnalysis

Dylan Patel, founder of SemiAnalysis, details how software-hardware co-design drives 100x gains in AI performance, citing examples like DeepSeek's optimization for Nvidia Hopper and the architectural differences between OpenAI's sparse and Anthropic's dense models. He introduces InferenceX, a living benchmark tracking a 60x annual cost reduction in AI quality, and explains why the "CUDA moat" narrative is shifting. Patel also delves into the persistent compute crunch, the strategic rise of neoclouds, and Jensen Huang's efforts to foster a multipolar AI world to prevent hyperscaler dominance.

Your Coding Agent Should Do AI System Engineering — Ben Burtenshaw, Hugging Face

Your Coding Agent Should Do AI System Engineering — Ben Burtenshaw, Hugging Face

Ben Burtenshaw from Hugging Face demonstrates how coding agents are tackling complex AI systems engineering tasks. He outlines a three-tiered approach: interactively writing CUDA kernels, autonomously fine-tuning LLMs, and deploying a multi-agent research lab (AutoLab) to parallelize experiments, all powered by file-based "skills" and open primitives on the Hugging Face Hub.

Jensen Huang – Will Nvidia’s moat persist?

Jensen Huang – Will Nvidia’s moat persist?

Nvidia CEO Jensen Huang discusses the company's core strategy, which he defines as transforming electrons into tokens by orchestrating a vast supply chain. He details how Nvidia's true moat lies in its ecosystem and its ability to manage supply bottlenecks. Huang contrasts Nvidia's versatile 'accelerated computing' platform with competitors like TPUs, arguing programmability via CUDA is key to AI innovation. He also presents a strong case against broad AI chip export controls on China, warning it could backfire by forcing the creation of a competing tech stack. Finally, he explains why Nvidia invests in the ecosystem rather than becoming a hyperscaler itself.

Next →