Faster Inference of LLMs using FP8 on the Intel Gaudi
Paper
•
2503.09975
•
Published
•
1
Need4Speed
company
AI & ML interests
None defined yet.
Recent Activity
View all activity
need-for-speed's activity
Haihao
authored
a
paper
17 days ago
Post
2511
Check out [DeepSeek-R1 INT2 model(
OPEA/DeepSeek-R1-int2-mixed-sym-inc). This 200GB DeepSeek-R1 model shows only about a 2% drop in MMLU, though it's quite slow due to kernel issue.
| | BF16 | INT2-mixed |
| ------------- | ------ | ---------- |
| mmlu | 0.8514 | 0.8302 |
| hellaswag | 0.6935 | 0.6657 |
| winogrande | 0.7932 | 0.7940 |
| arc_challenge | 0.6212 | 0.6084 |
| | BF16 | INT2-mixed |
| ------------- | ------ | ---------- |
| mmlu | 0.8514 | 0.8302 |
| hellaswag | 0.6935 | 0.6657 |
| winogrande | 0.7932 | 0.7940 |
| arc_challenge | 0.6212 | 0.6084 |
Post
734
OPEA Space has released several quantized DeepSeek models, including INT2. Explore them here
OPEA/deepseek-6784a012d91191015587584a
OPEA/deepseek-6784a012d91191015587584a
moshew
authored
a
paper
about 2 months ago
loubnabnl
authored
a
paper
about 2 months ago
lvwerra
authored
a
paper
about 2 months ago
lvwerra
authored
a
paper
3 months ago
Post
2342
Are we the only providers of INT4 quantized models for Llama 3.2 VL?
OPEA/Llama-3.2-90B-Vision-Instruct-int4-sym-inc
OPEA/Llama-3.2-11B-Vision-Instruct-int4-sym-inc
OPEA/Llama-3.2-90B-Vision-Instruct-int4-sym-inc
OPEA/Llama-3.2-11B-Vision-Instruct-int4-sym-inc
- 3 replies
Post
1825
AutoRound has demonstrated strong results even at 2-bit precision for VLM models like QWEN2-VL-72B. Check it out here:
OPEA/Qwen2-VL-72B-Instruct-int2-sym-inc.
-
- 4 replies
Post
345
This week, OPEA Space released several new INT4 models, including:
nvidia/Llama-3.1-Nemotron-70B-Instruct-HF
allenai/OLMo-2-1124-13B-Instruct
THUDM/glm-4v-9b
AIDC-AI/Marco-o1
and several others.
Let us know which models you'd like prioritized for quantization, and we'll do our best to make it happen!
OPEA
nvidia/Llama-3.1-Nemotron-70B-Instruct-HF
allenai/OLMo-2-1124-13B-Instruct
THUDM/glm-4v-9b
AIDC-AI/Marco-o1
and several others.
Let us know which models you'd like prioritized for quantization, and we'll do our best to make it happen!
OPEA
-
-
- 3 replies
Haihao
authored
a
paper
4 months ago
Post
3281
Making SmolLM2 reproducible: open-sourcing our training & evaluation toolkit 🛠️ https://github.com/huggingface/smollm/
- Pre-training code with nanotron
- Evaluation suite with lighteval
- Synthetic data generation using distilabel (powers our new SFT dataset HuggingFaceTB/smoltalk)
- Post-training scripts with TRL & the alignment handbook
- On-device tools with llama.cpp for summarization, rewriting & agents
Apache 2.0 licensed. V2 pre-training data mix coming soon!
Which other tools should we add next?
- Pre-training code with nanotron
- Evaluation suite with lighteval
- Synthetic data generation using distilabel (powers our new SFT dataset HuggingFaceTB/smoltalk)
- Post-training scripts with TRL & the alignment handbook
- On-device tools with llama.cpp for summarization, rewriting & agents
Apache 2.0 licensed. V2 pre-training data mix coming soon!
Which other tools should we add next?
ofirzaf
authored
2
papers
4 months ago
orenpereg
authored
5
papers
5 months ago
Term Set Expansion based on Multi-Context Term Embeddings: an End-to-end Workflow
Paper
•
1807.10104
•
Published
•
1
ABSApp: A Portable Weakly-Supervised Aspect-Based Sentiment Extraction System
Paper
•
1909.05608
•
Published
Cross-Domain Aspect Extraction using Transformers Augmented with Knowledge Graphs
Paper
•
2210.10144
•
Published
Efficient Few-Shot Learning Without Prompts
Paper
•
2209.11055
•
Published
•
3
Accelerating Speculative Decoding using Dynamic Speculation Length
Paper
•
2405.04304
•
Published
•
2