Korbinian Pöppel
commited on
Commit
·
fbd0d01
1
Parent(s):
a7ad556
Make model card more informative.
Browse files- MMLUvsTrainToken.svg +1737 -0
- README.md +22 -1
- plot_tokens_per_sec.svg +1893 -0
MMLUvsTrainToken.svg
ADDED
|
|
README.md
CHANGED
|
@@ -31,4 +31,25 @@ tokenizers = AutoTokenizer.from_pretrained("NX-AI/xLSTM-7b")
|
|
| 31 |
xlstm(tokenizer("Hello xLSTM, how are you doing?"))
|
| 32 |
```
|
| 33 |
|
| 34 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
xlstm(tokenizer("Hello xLSTM, how are you doing?"))
|
| 32 |
```
|
| 33 |
|
| 34 |
+
## Speed results
|
| 35 |
+
Generation Speed using `torch.cuda.graph` and `torch.compile` optimizations:
|
| 36 |
+

|
| 37 |
+
|
| 38 |
+
## Performance
|
| 39 |
+

|
| 40 |
+
|
| 41 |
+
Using HuggingFace's `lm_eval`:
|
| 42 |
+
|
| 43 |
+
| BBH | MMLU-Pro | Math | MUSR | GPQA | IfEval |
|
| 44 |
+
|-------|----------|--------|------|------|--------|
|
| 45 |
+
| 0.381 | 0.242 | 0.036 | 0.379|0.280 | 0.244 |
|
| 46 |
+
|
| 47 |
+
Using HuggingFace's `lighteval` in the Leaderboard-v1 settings:
|
| 48 |
+
|
| 49 |
+
|Arc-Challenge (25-shot) |MMLU (5-shot) |Hellaswag (10-shot)|Winogrande (5-shot) |TruthfulQA (0-shot) |GSM8k (5-shot) |OpenbookQA (5-shot) | PiQA (5-shot)|
|
| 50 |
+
|------------------------|--------------|-------------------|--------------------|--------------------|---------------|--------------------|--------------|
|
| 51 |
+
| 0.584 |0.589 | 0.710 |0.742 | 0.420 | 0.004 | 0.443 | 0.817 |
|
| 52 |
+
|
| 53 |
+
## License
|
| 54 |
+
NXAI Community License (see `LICENSE` file)
|
| 55 |
+
|
plot_tokens_per_sec.svg
ADDED
|
|