LLMs & Models4 min read
AI Research & Development
Ma Jiaqi taught MiniMax engineers a hard lesson about forgotten tokens
MiniMax's internal investigation into why its M2 model couldn't output the name 'Ma Jiaqi' revealed a structural mismatch between pre-training vocabulary and post-training data distribution. The root cause: low-frequency tokens' lm_head vectors drift during SFT, losing generation ability while retaining understanding. A full-vocabulary coverage fix resolved the issue and also mitigated language mixing in Japanese.
2026-07-04