Shaping capabilities with token-level data filtering
Paper
ā¢
2601.21571
ā¢
Published
ā¢
22
Nanotron Research
community
AI & ML interests
Large scale distributed AI model training, model parallelisation, low-level GPU acceleration, make GPUs go brrrrr
Recent Activity
View all activity
julien-cĀ
submitted
a
paper
to Daily Papers
5 days ago
eliebakĀ
submitted
a
paper
to Daily Papers
about 2 months ago
nouamanetaziĀ
posted
an
update
3 months ago
Post
4376
After training šš¦šØš„ššš on ššš ššššš¬ for nearly a month, I've come to realize something most people overlook: š¢š§šš«šš¬šš«š®ššš®š«š š¢š¬ šš”š š¦šš¤š-šØš«-šš«ššš¤ šššššØš« š¢š§ ššš šš«šš¢š§š¢š§š . š„
Everyone talks about model architecture and data quality. And yes, those matter immensely. But here's what nobody tells you: when your training run fails at 2 AM because of mysterious šššš šš«š«šØš«š¬, or when your expensive GPU cluster is running at šš% šššš¢šš¢šš§šš², the problem isn't your model. It's most probably a š¦š¢š¬š®š¬š šØš šš”š š”šš«šš°šš«š. š ļø
Questions that seemed simple but had no clear answers: Why is ššØš šš«šš¢š§š¢š§š š¬š„šØš°šš« šš”šš§ ššš§š¬š š¦šØššš„š¬? Which šššš šš„šš š¬ should we actually set? How often should we checkpoint without killing throughput?
That's why we built šš”š šš¦šØš„ šš«šš¢š§š¢š§š šš„šš²ššØšØš¤ š: a complete guide covering everything from model architecture and data curation to the SmolLM3 training marathon, post-training techniques, and crucially, the š¢š§šš«šš¬šš«š®ššš®š«š š„šš²šš« that most teams get wrong.
We validated real vs theoretical bandwidth across the entire stack: šššš š”š¢ššš¢š§š š šš/š¬, šššš¢š§š¤ š.š š«šššš”š¢š§š ššš šš/š¬, šššš ššš§š šš šš.š šš/š¬. Then we ran collective operations across ššš šššš¬ (16 nodes, 8xH100s each) and measured how performance degrades at scale: all-reduce drops from ššš šš/š¬ on a single node to ššš-ššš šš/š¬ across 16 nodes.
If you've ever wondered why your training runs are slower than they should be, or you're planning to scale up and want to avoid expensive mistakes, this guide might save you weeks of debugging.
šš”š šš¦šØš„ šš«šš¢š§š¢š§š šš„šš²ššØšØš¤: https://lnkd.in/e5MKXUHS
Shared with ā¤ļø by the HuggingFace team
Everyone talks about model architecture and data quality. And yes, those matter immensely. But here's what nobody tells you: when your training run fails at 2 AM because of mysterious šššš šš«š«šØš«š¬, or when your expensive GPU cluster is running at šš% šššš¢šš¢šš§šš², the problem isn't your model. It's most probably a š¦š¢š¬š®š¬š šØš šš”š š”šš«šš°šš«š. š ļø
Questions that seemed simple but had no clear answers: Why is ššØš šš«šš¢š§š¢š§š š¬š„šØš°šš« šš”šš§ ššš§š¬š š¦šØššš„š¬? Which šššš šš„šš š¬ should we actually set? How often should we checkpoint without killing throughput?
That's why we built šš”š šš¦šØš„ šš«šš¢š§š¢š§š šš„šš²ššØšØš¤ š: a complete guide covering everything from model architecture and data curation to the SmolLM3 training marathon, post-training techniques, and crucially, the š¢š§šš«šš¬šš«š®ššš®š«š š„šš²šš« that most teams get wrong.
We validated real vs theoretical bandwidth across the entire stack: šššš š”š¢ššš¢š§š š šš/š¬, šššš¢š§š¤ š.š š«šššš”š¢š§š ššš šš/š¬, šššš ššš§š šš šš.š šš/š¬. Then we ran collective operations across ššš šššš¬ (16 nodes, 8xH100s each) and measured how performance degrades at scale: all-reduce drops from ššš šš/š¬ on a single node to ššš-ššš šš/š¬ across 16 nodes.
If you've ever wondered why your training runs are slower than they should be, or you're planning to scale up and want to avoid expensive mistakes, this guide might save you weeks of debugging.
šš”š šš¦šØš„ šš«šš¢š§š¢š§š šš„šš²ššØšØš¤: https://lnkd.in/e5MKXUHS
Shared with ā¤ļø by the HuggingFace team
thomwolfĀ
authored
a
paper
4 months ago
lvwerraĀ
authored
a
paper
4 months ago
Post
4034
Super excited to announce that our research team at Hugging Face will be doing an AMA on reddit r/LocalLLaMA.
Come ask any questions to the team behind SmolLM, FineWeb and more! And who knows, maybe thereāll be a shiny new release to talk about?
Thursday 4th September, 8AM-11AM PST š¤
science
Come ask any questions to the team behind SmolLM, FineWeb and more! And who knows, maybe thereāll be a shiny new release to talk about?
Thursday 4th September, 8AM-11AM PST š¤
Post
728
Motif 2.6B tech report is pretty insane, first time i see a model with differential attention and polynorm trained at scale!
> It's trained on 2.5T of token, with a "data mixture schedule" to continuously adjust the mixture over training.
> They use WSD with a "Simple moving average" averaging the last 6 ckpt every 8B token.
> They trained on Finemath, Fineweb2, DCLM, TxT360.
> Lot of details in the finetuning data they used, for instance they used EvolKit and did some "dataset fusion" to have more compressed knowledge into the data.
> They mention they also tried Normalized GPT, QK-Norm and Cross Layer Attention.
Motif-Technologies/Motif-2.6B
> It's trained on 2.5T of token, with a "data mixture schedule" to continuously adjust the mixture over training.
> They use WSD with a "Simple moving average" averaging the last 6 ckpt every 8B token.
> They trained on Finemath, Fineweb2, DCLM, TxT360.
> Lot of details in the finetuning data they used, for instance they used EvolKit and did some "dataset fusion" to have more compressed knowledge into the data.
> They mention they also tried Normalized GPT, QK-Norm and Cross Layer Attention.
Motif-Technologies/Motif-2.6B
dark-mode
š¤
1
11
#82 opened 12 months ago
by
serhany
tfrereĀ
updated
a
Space
6 months ago
thomwolfĀ
updated
a
Space
6 months ago
order button
#118 opened 6 months ago
by
lvwerra
order button
#118 opened 6 months ago
by
lvwerra
lvwerraĀ
updated
a
Space
6 months ago
julien-cĀ
updated
a
Space
6 months ago
Update README.md
#1 opened 6 months ago
by
lvwerra
Update README.md
#2 opened 6 months ago
by
lvwerra
Update README.md
#3 opened 6 months ago
by
lvwerra
julien-cĀ
updated
a
dataset
6 months ago
julien-cĀ
published
a
dataset
6 months ago