Thomas Wolf and his team released a free seventy-thousand-word handbook detailing Mixture-of-Experts model training. The guide draws on eight hundred thousand GPU hours of experiments to solve common architecture and scaling challenges.
FireBrief
FireBrief