Engineers from AI21 Labs uncovered two silent bugs in vLLM that corrupted outputs in hybrid Transformer-Mamba models: an out-of-order scheduler that read stale state buffers during decode, and a 32-bit pointer overflow in the state cache.
FireBrief
FireBrief