Weight Folding, CUDA Streams, and the Bug That Made My Model Speak Backwards — Filip Makraduli

This talk introduces FlashNorm, an optimization for transformer RMS norm layers. It details how algebraic tricks like weight folding, deferred division, and dropped pre-normalization can significantly speed up inference by mitigating GPU inefficiencies. The speaker also shares critical insights from implementing parallel CUDA streams, including debugging a subtle race condition caused by implicit stream joins, and advocates for open inference engines for deploying kernel-level research.