TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior
Paper
โข
2512.20757
โข
Published
โข
16
AI & ML interests
A one-year long research workshop on large language models: the Summer of Language Models 21 ๐ธ
Recent Activity
View all activity
craffelย
authored
a
paper
10 days ago
Fails to load with transformers v4.57+
4
#14 opened 27 days ago
by
qgallouedec
nihalnayakย
authored
a
paper
about 1 month ago
monsoon-nlpย
posted
an
update
about 1 month ago
Post
391
PatchDNA, a DNA foundation model based on Meta's BLT tokenization strategy https://www.biorxiv.org/content/10.1101/2025.11.28.691095v1
rabiulawalย
authored
a
paper
about 2 months ago
thomwolfย
authored
a
paper
3 months ago
nihalnayakย
authored
a
paper
3 months ago
sashaย
authored
3
papers
3 months ago
Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model
Paper
โข
2211.02001
โข
Published
Hype, Sustainability, and the Price of the Bigger-is-Better Paradigm in AI
Paper
โข
2409.14160
โข
Published
โข
3
From Efficiency Gains to Rebound Effects: The Problem of Jevons' Paradox in AI's Polarized Environmental Debate
Paper
โข
2501.16548
โข
Published
monsoon-nlpย
posted
an
update
3 months ago
Post
458
Bio LLMs train on many genomes, but can we encode differences within a species? TomatoTomato adds pangenome tokens to represent a domestic tomato and a wild tomato in one sequence ๐
๐งฌ
monsoon-nlp/tomatotomato-gLM2-150M-v0.1
monsoon-nlp/tomatotomato-gLM2-150M-v0.1
Post
2627
Tremendous quality of life upgrade on the Hugging Face Hub - we now have auto-complete emojis ๐ค ๐ฅณ ๐ ๐ ๐
Get ready for lots more very serious analysis on a whole range of topics from yours truly now that we have unlocked this full range of expression ๐ ๐ค ๐ฃ ๐
Get ready for lots more very serious analysis on a whole range of topics from yours truly now that we have unlocked this full range of expression ๐ ๐ค ๐ฃ ๐
Let's talk about the model
1
#284 opened 5 months ago
by
kalashshah19
yjerniteย
authored
3
papers
5 months ago
The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources
Paper
โข
2406.16746
โข
Published
In-House Evaluation Is Not Enough: Towards Robust Third-Party Flaw Disclosure for General-Purpose AI
Paper
โข
2503.16861
โข
Published
โข
1
A Different Approach to AI Safety: Proceedings from the Columbia Convening on Openness in Artificial Intelligence and AI Safety
Paper
โข
2506.22183
โข
Published
โข
1
w11woย
authored
a
paper
5 months ago
Post
4211
๐๐ถ๐ฟ๐๐ ๐๐ฃ๐๐ ๐ ๐ผ๐ฑ๐ฒ๐น ๐๐ถ๐๐ต ๐๐จ ๐๐ฎ๐๐ฎ ๐ง๐ฟ๐ฎ๐ป๐๐ฝ๐ฎ๐ฟ๐ฒ๐ป๐ฐ๐ ๐ง๐ฒ๐บ๐ฝ๐น๐ฎ๐๐ฒ? ๐ช๐บ
With the release of the EU data transparency template this week, we finally got to see one of the most meaningful artifacts to come out of the AI Act implementation so far (haven't you heard? AI's all about the data! ๐๐)
The impact of the template will depend on how effectively it establishes a minimum meaningful transparency standard for companies that don't otherwise offer any transparency into their handling of e.g. personal data or (anti?-)competitive practices in commercial licensing - we'll see how those play out as new models are released after August 2nd ๐
In the meantime, I wanted to see how the template works for a fully open-source + commercially viable model, so I filled it out for the SmolLM3 - which my colleagues at Hugging Face earlier this month ๐ค ICYMI, it's fully open-source with 3B parameters and performance matching the best similar-size models (I've switched all my local apps from Qwen3 to it, you should too ๐ก)
Verdict: congrats to the European Commission AI Office for making it so straightforward! Fully open and transparent models remain a cornerstone of informed regulation and governance, but the different organizational needs of their developers aren't always properly accounted for in new regulation. In this case, it took me all of two hours to fill out and publish the template (including reading the guidelines) - so kudos for making it feasible for smaller and distributed organizations ๐ Definitely a step forward for transparency ๐
To learn more have a look at:
- The SmolLM3 model: HuggingFaceTB/SmolLM3-3B
- Its filled out Public Summary of Training Content: hfmlsoc/smollm3-eu-data-transparency
- And if you're interested, some previous remarks on regulatory minimum meaningful standards for data disclosure: https://huggingface.co/blog/yjernite/naiac-data-transparency
With the release of the EU data transparency template this week, we finally got to see one of the most meaningful artifacts to come out of the AI Act implementation so far (haven't you heard? AI's all about the data! ๐๐)
The impact of the template will depend on how effectively it establishes a minimum meaningful transparency standard for companies that don't otherwise offer any transparency into their handling of e.g. personal data or (anti?-)competitive practices in commercial licensing - we'll see how those play out as new models are released after August 2nd ๐
In the meantime, I wanted to see how the template works for a fully open-source + commercially viable model, so I filled it out for the SmolLM3 - which my colleagues at Hugging Face earlier this month ๐ค ICYMI, it's fully open-source with 3B parameters and performance matching the best similar-size models (I've switched all my local apps from Qwen3 to it, you should too ๐ก)
Verdict: congrats to the European Commission AI Office for making it so straightforward! Fully open and transparent models remain a cornerstone of informed regulation and governance, but the different organizational needs of their developers aren't always properly accounted for in new regulation. In this case, it took me all of two hours to fill out and publish the template (including reading the guidelines) - so kudos for making it feasible for smaller and distributed organizations ๐ Definitely a step forward for transparency ๐
To learn more have a look at:
- The SmolLM3 model: HuggingFaceTB/SmolLM3-3B
- Its filled out Public Summary of Training Content: hfmlsoc/smollm3-eu-data-transparency
- And if you're interested, some previous remarks on regulatory minimum meaningful standards for data disclosure: https://huggingface.co/blog/yjernite/naiac-data-transparency
NohTowย
authored
a
paper
6 months ago