Philip Kiely of Baseten outlined major advances in inference engineering, highlighting diffusion-based speculative decoding through DFlash, learned cache compaction with STILL, and the growing reliance on dedicated training to optimize data center serving speeds.
FireBrief
FireBrief