Configuration error 29 Soofi S β Sovereign German-English Foundation Model π 29 Soofi S 30B-A3B pretraining report project page
The Depth Delusion: Why Transformers Should Be Wider, Not Deeper Paper β’ 2601.20994 β’ Published Jan 28 β’ 1
Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models Paper β’ 2501.12370 β’ Published Jan 21, 2025 β’ 12
Strong Teacher Not Needed? On Distillation in LLM Pretraining Paper β’ 2605.23857 β’ Published May 22 β’ 1
Adam's Law: Textual Frequency Law on Large Language Models Paper β’ 2604.02176 β’ Published Apr 2 β’ 509
KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs Paper β’ 2604.13226 β’ Published Apr 14 β’ 11
Running Featured 92 Distilling 100B+ Models 40x Faster with TRL π 92 TRL distillation for 100B+ teachers, 40x faster
view article Article SmolLM3: smol, multilingual, long-context reasoner +21 eliebak, cmpatino, anton-l, edbeeching, m-ric, nouamanetazi, akseljoonas, guipenedo, hynky, clefourrier, SaylorTwift, kashif, qgallouedec, hlarcher, glutamatt, Xenova, reach-vb, ngxson, craffel, lewtun, loubnabnl, lvwerra, thomwolf β’ Jul 8, 2025 β’ 782