nm-testing/Llama-3.1-8B-Instruct-QKV-Cache-FP8

The following accuracy is using lm-eval and HF:

lm_eval \
  --model hf \
  --model_args pretrained="nm-testing/Llama-3.1-8B-Instruct-QKV-Cache-FP8",dtype=auto,device_map="auto",max_length=100000 \
  --tasks "niah_single_1" \
  --write_out \
  --batch_size 1 \
  --output_path "niah_single_1.json" \
  --show_config

Replace niah_single_1 with niah_single_2,niah_single_2,niah_multikey_1, niah_multikey_2, niah_multikey_3

Accuracy with HF

Category	Task	meta-llama/Llama-3.1-8B-Instruct	nm-testing/Llama-3.1-8B-Instruct-QKV-Cache-FP8	Recovery (%)
NIAH	niah_single_1 (100K)	OOM	0.0	0
	niah_single_2 (100K)	OOM	0.0	0
	niah_single_3 (100K)	OOM	0.0	0
	niah_multikey_1 (100K)	OOM	0.0	0.
	niah_multikey_2 (100K)	OOM	0.0	0.0
	niah_multikey_3 (100K)	OOM	0.0	0.0

The following accuracy is using lm-eval and vLLM:

 lm_eval \
    --model vllm \
    --model_args pretrained="nm-testing/Llama-3.1-8B-Instruct-QKV-Cache-FP8",dtype=auto,add_bos_token=True,max_model_len=131072,tensor_parallel_size=1,gpu_memory_utilization=0.7,enable_chunked_prefill=True,trust_remote_code=True \
    --tasks "niah_single_1" \
    --write_out \
    --batch_size 1 \
    --output_path "niah_single_1.json" \
    --show_config

Replace niah_single_1 with niah_single_2,niah_single_2,niah_multikey_1, niah_multikey_2, niah_multikey_3

Accuracy with vLLM

Category	Task	meta-llama/Llama-3.1-8B-Instruct	nm-testing/Llama-3.1-8B-Instruct-QKV-Cache-FP8	Recovery (%)
LongBench V1	Gov Report	32.16	20.82	64.73
	2WikimQA	17.35	18.53	106.8
	Qasper	18.22	26.6	145
	MultifieldQA	31.11	21.95	70.55
	HotpotQA	16.23	4.86	29.9
	Musique	9.11	0.94	10.31
NIAH	niah_single_1 (4K)	100.0	100.0	100.0
	niah_single_2 (4K)	100.0	100.0	100.0
	niah_single_3 (4K)	100.0	100.0	100.0
	niah_multikey_1 (4K)	100.0	100.0	100.0
	niah_multikey_2 (4K)	100.0	100.0	100.0
	niah_multikey_3 (4K)	100.0	100.0	100.0