File size: 1,249 Bytes
801c690 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
---
license: llama3
datasets:
- yuyijiong/Long-Instruction-with-Paraphrasing
language:
- zh
- en
library_name: peft
pipeline_tag: text-generation
---
# Llama3-8b-chinese-chat-32k
## 训练方式
* 使用 NTK-aware 方法扩展上下文长度至32k
* 以 [shenzhi-wang/Llama3-8B-Chinese-Chat](https://huggingface.co/shenzhi-wang/Llama3-8B-Chinese-Chat) 为基础
在 [Long-Instruction-with-Paraphrasing](https://huggingface.co/datasets/yuyijiong/Long-Instruction-with-Paraphrasing)
数据集上,使用 QLora 微调 1 epoch。
## 长上下文表现
相比原始版本,拥有更强的长上下文能力
### LongBench (en)
| model | hotpotqa | multifieldqa_en| passage_retrieval_en|qmsum| trec|
|---------------------------|-----------|--|--|--|--|
| llama3-chinese-8b | 45.88 |50.56|68.0|22.52|73.0|
| llama3-8b-chinese-chat-32k| **47.64** |49.98|**100.0**|**25.13**|**75.0**|
### LongBench (zh)
| model | dureader | multifieldqa_zh| passage_retrieval_zh|qmsum| trec|
|-----------------------------------|-----------|--|--|--|--|
| llama3-8b-chinese-chat | 29.08 |58.4|93.5|22.52|73.0|
| llama3-8b-chinese-chat-32k | **32.31** |**58.66**|82.5|**25.13**|**75.0**| |