Hugging Face's logo

Kernels Tests

community
Activity Feed

AI & ML interests

None defined yet.

Recent Activity

danieldk  updated a model 12 days ago
kernels-test/relu-metal
danieldk  published a model 15 days ago
kernels-test/relu-metal
danieldk  updated a model about 1 month ago
kernels-test/op-without-fake-test
View all activity

kernels-test's activity

danieldk 
posted an update 3 days ago
view post
Post
1406
We have been working on a project called kernels. kernels makes it possible to load compute kernels directly from the Hub! 🚀

We plan to give kernels a more proper introduction soon. But for those who have been following along, we are happy to announce a new release:

- New layer API with torch.compile support.
- Experimental support for loading Apple Silicon Metal 🤘 Kernels.
- Generate wheels from Hub kernels for legacy deployments.

Full release notes here: https://github.com/huggingface/kernels/releases/tag/v0.6.0
danieldk 
updated a model 12 days ago
danieldk 
published a model 15 days ago
danieldk 
updated a model about 1 month ago
danieldk 
published a model about 1 month ago
danieldk 
updated 2 models about 2 months ago
danieldk 
published a model about 2 months ago
Narsil 
posted an update 6 months ago
view post
Post
1682
Performance leap: TGI v3 is out. Processes 3x more tokens, 13x faster than vLLM on long prompts. Zero config !



3x more tokens.

By reducing our memory footprint, we’re able to ingest many more tokens and more dynamically than before. A single L4 (24GB) can handle 30k tokens on llama 3.1-8B, while vLLM gets barely 10k. A lot of work went into reducing the footprint of the runtime and its effect are best seen on smaller constrained environments.
13x faster

On long prompts (200k+ tokens) conversation replies take 27.5s in vLLM, while it takes only 2s in TGI. How so ? We keep the initial conversation around, so when a new reply comes in, we can answer almost instantly. The overhead of the lookup is ~5us. Thanks @Dani ël de Kok for the beast data structure.
Zero config

That’s it. Remove all the flags your are using and you’re likely to get the best performance. By evaluating the hardware and model, TGI carefully selects automatic values to give best performance. In production, we don’t have any flags anymore in our deployments. We kept all existing flags around, they may come in handy in niche scenarios.

Read more: https://huggingface.co/docs/text-generation-inference/conceptual/chunking
danieldk 
authored a paper 9 months ago
Narsil 
posted an update about 1 year ago
Narsil 
posted an update about 1 year ago
Narsil 
authored a paper over 1 year ago
Narsil 
authored a paper about 2 years ago