Hugging Face's logo

gatepoet
/
XAgentLlama-7B-preview-q8f32_1-MLC

Model card Files Community
XAgentLlama-7B-preview-q8f32_1-MLC / logs.txt
gatepoet's picture
gatepoet
Initial commit
bed93ea verified
raw
history blame contribute delete
208 kB
/home/floriadmin/miniforge3/envs/mlc/bin/python -m mlc_llm gen_config ../dist/models/XAgentLlama-7B-preview --quantization q8f32_1 --conv-template llama-2 --output /tmp/tmpyuo2_jf8
[2024-03-18 20:20:37] INFO auto_config.py:115: Found model configuration: ../dist/models/XAgentLlama-7B-preview/config.json
[2024-03-18 20:20:37] INFO auto_config.py:153: Found model type: llama. Use `--model-type` to override.
[2024-03-18 20:20:37] INFO llama_model.py:52: context_window_size not found in config.json. Falling back to max_position_embeddings (16384)
[2024-03-18 20:20:37] INFO llama_model.py:72: prefill_chunk_size defaults to context_window_size (16384)
[2024-03-18 20:20:37] INFO config.py:106: Overriding max_batch_size from 1 to 80
[2024-03-18 20:20:37] INFO gen_config.py:133: [generation_config.json] Setting bos_token_id: 1
[2024-03-18 20:20:37] INFO gen_config.py:133: [generation_config.json] Setting eos_token_id: 2
[2024-03-18 20:20:37] INFO gen_config.py:145: Found tokenizer config: ../dist/models/XAgentLlama-7B-preview/tokenizer.model. Copying to /tmp/tmpyuo2_jf8/tokenizer.model
[2024-03-18 20:20:37] INFO gen_config.py:145: Found tokenizer config: ../dist/models/XAgentLlama-7B-preview/tokenizer.json. Copying to /tmp/tmpyuo2_jf8/tokenizer.json
[2024-03-18 20:20:37] INFO gen_config.py:147: Not found tokenizer config: ../dist/models/XAgentLlama-7B-preview/vocab.json
[2024-03-18 20:20:37] INFO gen_config.py:147: Not found tokenizer config: ../dist/models/XAgentLlama-7B-preview/merges.txt
[2024-03-18 20:20:37] INFO gen_config.py:147: Not found tokenizer config: ../dist/models/XAgentLlama-7B-preview/added_tokens.json
[2024-03-18 20:20:37] INFO gen_config.py:145: Found tokenizer config: ../dist/models/XAgentLlama-7B-preview/tokenizer_config.json. Copying to /tmp/tmpyuo2_jf8/tokenizer_config.json
[2024-03-18 20:20:37] INFO gen_config.py:75: [System default] Setting pad_token_id: 0
[2024-03-18 20:20:37] INFO gen_config.py:75: [System default] Setting temperature: 0.7
[2024-03-18 20:20:37] INFO gen_config.py:75: [System default] Setting presence_penalty: 0.0
[2024-03-18 20:20:37] INFO gen_config.py:75: [System default] Setting frequency_penalty: 0.0
[2024-03-18 20:20:37] INFO gen_config.py:75: [System default] Setting repetition_penalty: 1.0
[2024-03-18 20:20:37] INFO gen_config.py:75: [System default] Setting top_p: 0.95
[2024-03-18 20:20:37] INFO gen_config.py:75: [System default] Setting mean_gen_len: 128
[2024-03-18 20:20:37] INFO gen_config.py:75: [System default] Setting max_gen_len: 512
[2024-03-18 20:20:37] INFO gen_config.py:75: [System default] Setting shift_fill_factor: 0.3
[2024-03-18 20:20:37] INFO gen_config.py:198: Dumping configuration file to: /tmp/tmpyuo2_jf8/mlc-chat-config.json
/home/floriadmin/miniforge3/envs/mlc/bin/python -m mlc_llm convert_weight ../dist/models/XAgentLlama-7B-preview --quantization q8f32_1 --source-format auto --output /tmp/tmpyuo2_jf8
[2024-03-18 20:20:38] INFO auto_config.py:115: Found model configuration: ../dist/models/XAgentLlama-7B-preview/config.json
[2024-03-18 20:20:39] INFO auto_device.py:76: Found device: cuda:0
[2024-03-18 20:20:39] INFO auto_device.py:76: Found device: cuda:1
[2024-03-18 20:20:39] INFO auto_device.py:76: Found device: cuda:2
[2024-03-18 20:20:39] INFO auto_device.py:76: Found device: cuda:3
[2024-03-18 20:20:39] INFO auto_device.py:76: Found device: cuda:4
[2024-03-18 20:20:39] INFO auto_device.py:76: Found device: cuda:5
[2024-03-18 20:20:39] INFO auto_device.py:76: Found device: cuda:6
[2024-03-18 20:20:39] INFO auto_device.py:76: Found device: cuda:7
[2024-03-18 20:20:39] INFO auto_device.py:76: Found device: cuda:8
[2024-03-18 20:20:39] INFO auto_device.py:76: Found device: cuda:9
[2024-03-18 20:20:40] INFO auto_device.py:85: Not found device: rocm:0
[2024-03-18 20:20:41] INFO auto_device.py:85: Not found device: metal:0
[2024-03-18 20:20:44] INFO auto_device.py:76: Found device: vulkan:0
[2024-03-18 20:20:44] INFO auto_device.py:76: Found device: vulkan:1
[2024-03-18 20:20:44] INFO auto_device.py:76: Found device: vulkan:2
[2024-03-18 20:20:44] INFO auto_device.py:76: Found device: vulkan:3
[2024-03-18 20:20:44] INFO auto_device.py:76: Found device: vulkan:4
[2024-03-18 20:20:44] INFO auto_device.py:76: Found device: vulkan:5
[2024-03-18 20:20:44] INFO auto_device.py:76: Found device: vulkan:6
[2024-03-18 20:20:44] INFO auto_device.py:76: Found device: vulkan:7
[2024-03-18 20:20:44] INFO auto_device.py:76: Found device: vulkan:8
[2024-03-18 20:20:44] INFO auto_device.py:76: Found device: vulkan:9
[2024-03-18 20:20:44] INFO auto_device.py:76: Found device: vulkan:10
[2024-03-18 20:20:45] INFO auto_device.py:85: Not found device: opencl:0
[2024-03-18 20:20:45] INFO auto_device.py:33: Using device: cuda:0
[2024-03-18 20:20:45] INFO auto_weight.py:70: Finding weights in: ../dist/models/XAgentLlama-7B-preview
[2024-03-18 20:20:45] INFO auto_weight.py:129: Found source weight format: huggingface-torch. Source configuration: ../dist/models/XAgentLlama-7B-preview/pytorch_model.bin
[2024-03-18 20:20:45] INFO auto_weight.py:143: Found source weight format: huggingface-safetensor. Source configuration: ../dist/models/XAgentLlama-7B-preview/model.safetensors.index.json
[2024-03-18 20:20:45] INFO auto_weight.py:106: Using source weight configuration: ../dist/models/XAgentLlama-7B-preview/pytorch_model.bin. Use `--source` to override.
[2024-03-18 20:20:45] INFO auto_weight.py:110: Using source weight format: huggingface-torch. Use `--source-format` to override.
[2024-03-18 20:20:45] INFO auto_config.py:153: Found model type: llama. Use `--model-type` to override.
[2024-03-18 20:20:45] INFO llama_model.py:52: context_window_size not found in config.json. Falling back to max_position_embeddings (16384)
[2024-03-18 20:20:45] INFO llama_model.py:72: prefill_chunk_size defaults to context_window_size (16384)
[2024-03-18 20:20:48] INFO huggingface_loader.py:182: Loading HF parameters from: ../dist/models/XAgentLlama-7B-preview/pytorch_model.bin
Weight conversion with arguments:
--config ../dist/models/XAgentLlama-7B-preview/config.json
--quantization GroupQuantize(name='q8f32_1', kind='group-quant', group_size=32, quantize_dtype='int8', storage_dtype='uint32', model_dtype='float32', linear_weight_layout='NK', quantize_embedding=True, quantize_final_fc=True, num_elem_per_storage=4, num_storage_per_group=8, max_int_value=127)
--model-type llama
--device cuda:0
--source ../dist/models/XAgentLlama-7B-preview/pytorch_model.bin
--source-format huggingface-torch
--output /tmp/tmpyuo2_jf8
Start storing to cache /tmp/tmpyuo2_jf8
0%| | 0/195 [00:00<?, ?it/s] [2024-03-18 20:21:18] INFO group_quantization.py:232: Compiling quantize function for key: ((32016, 4096), float32, cuda, axis=1, output_transpose=False)
0%| | 0/195 [00:02<?, ?it/s] [2024-03-18 20:21:19] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.embed_tokens.q_weight", shape: (32016, 1024), dtype: uint32
0%| | 0/195 [00:03<?, ?it/s] [2024-03-18 20:21:20] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.embed_tokens.q_scale", shape: (32016, 128), dtype: float32
0%| | 0/195 [00:03<?, ?it/s]/home/floriadmin/miniforge3/envs/mlc/lib/python3.11/site-packages/numpy/core/getlimits.py:549: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
setattr(self, word, getattr(machar, word).flat[0])
/home/floriadmin/miniforge3/envs/mlc/lib/python3.11/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
return self._float_to_str(self.smallest_subnormal)
1%|▍ | 1/195 [00:03<12:41, 3.93s/it] [2024-03-18 20:21:20] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.norm.weight", shape: (4096,), dtype: float32
1%|▍ | 1/195 [00:03<12:41, 3.93s/it] [2024-03-18 20:21:22] INFO huggingface_loader.py:164: [Quantized] Parameter: "lm_head.q_weight", shape: (32016, 1024), dtype: uint32
1%|▍ | 1/195 [00:05<12:41, 3.93s/it] [2024-03-18 20:21:22] INFO huggingface_loader.py:164: [Quantized] Parameter: "lm_head.q_scale", shape: (32016, 128), dtype: float32
1%|▍ | 1/195 [00:06<12:41, 3.93s/it] 2%|█▍ | 3/195 [00:06<06:30, 2.04s/it] [2024-03-18 20:21:23] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.0.input_layernorm.weight", shape: (4096,), dtype: float32
2%|█▍ | 3/195 [00:06<06:30, 2.04s/it] [2024-03-18 20:21:23] INFO group_quantization.py:232: Compiling quantize function for key: ((12288, 4096), float32, cuda, axis=1, output_transpose=False)
2%|█▍ | 3/195 [00:07<06:30, 2.04s/it] [2024-03-18 20:21:24] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.0.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
2%|█▍ | 3/195 [00:07<06:30, 2.04s/it] [2024-03-18 20:21:24] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.0.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
2%|█▍ | 3/195 [00:08<06:30, 2.04s/it] 3%|██▎ | 5/195 [00:08<04:16, 1.35s/it] [2024-03-18 20:21:24] INFO group_quantization.py:232: Compiling quantize function for key: ((4096, 4096), float32, cuda, axis=1, output_transpose=False)
3%|██▎ | 5/195 [00:08<04:16, 1.35s/it] [2024-03-18 20:21:25] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.0.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
3%|██▎ | 5/195 [00:08<04:16, 1.35s/it] [2024-03-18 20:21:25] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.0.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
3%|██▎ | 5/195 [00:08<04:16, 1.35s/it] 3%|██▊ | 6/195 [00:08<03:42, 1.18s/it] [2024-03-18 20:21:25] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.0.post_attention_layernorm.weight", shape: (4096,), dtype: float32
3%|██▊ | 6/195 [00:08<03:42, 1.18s/it] [2024-03-18 20:21:26] INFO group_quantization.py:232: Compiling quantize function for key: ((22016, 4096), float32, cuda, axis=1, output_transpose=False)
3%|██▊ | 6/195 [00:10<03:42, 1.18s/it] [2024-03-18 20:21:27] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.0.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
3%|██▊ | 6/195 [00:10<03:42, 1.18s/it] [2024-03-18 20:21:27] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.0.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
3%|██▊ | 6/195 [00:11<03:42, 1.18s/it] 4%|███▊ | 8/195 [00:11<03:40, 1.18s/it] [2024-03-18 20:21:28] INFO group_quantization.py:232: Compiling quantize function for key: ((4096, 11008), float32, cuda, axis=1, output_transpose=False)
4%|███▊ | 8/195 [00:11<03:40, 1.18s/it] [2024-03-18 20:21:28] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.0.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
4%|███▊ | 8/195 [00:12<03:40, 1.18s/it] [2024-03-18 20:21:28] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.0.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
4%|███▊ | 8/195 [00:12<03:40, 1.18s/it] 5%|████▏ | 9/195 [00:12<03:45, 1.21s/it] [2024-03-18 20:21:28] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.1.input_layernorm.weight", shape: (4096,), dtype: float32
5%|████▏ | 9/195 [00:12<03:45, 1.21s/it] [2024-03-18 20:21:29] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.1.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
5%|████▏ | 9/195 [00:13<03:45, 1.21s/it] [2024-03-18 20:21:29] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.1.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
5%|████▏ | 9/195 [00:13<03:45, 1.21s/it] 6%|█████▏ | 11/195 [00:13<02:48, 1.09it/s] [2024-03-18 20:21:30] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.1.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
6%|█████▏ | 11/195 [00:13<02:48, 1.09it/s] [2024-03-18 20:21:30] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.1.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
6%|█████▏ | 11/195 [00:13<02:48, 1.09it/s] 6%|█████▌ | 12/195 [00:13<02:22, 1.28it/s] [2024-03-18 20:21:30] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.1.post_attention_layernorm.weight", shape: (4096,), dtype: float32
6%|█████▌ | 12/195 [00:13<02:22, 1.28it/s] [2024-03-18 20:21:31] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.1.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
6%|█████▌ | 12/195 [00:15<02:22, 1.28it/s] [2024-03-18 20:21:32] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.1.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
6%|█████▌ | 12/195 [00:15<02:22, 1.28it/s] 7%|██████▌ | 14/195 [00:15<02:37, 1.15it/s] [2024-03-18 20:21:32] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.1.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
7%|██████▌ | 14/195 [00:16<02:37, 1.15it/s] [2024-03-18 20:21:33] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.1.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
7%|██████▌ | 14/195 [00:16<02:37, 1.15it/s] 8%|███████ | 15/195 [00:16<02:36, 1.15it/s] [2024-03-18 20:21:33] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.2.input_layernorm.weight", shape: (4096,), dtype: float32
8%|███████ | 15/195 [00:16<02:36, 1.15it/s] [2024-03-18 20:21:33] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.2.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
8%|███████ | 15/195 [00:17<02:36, 1.15it/s] [2024-03-18 20:21:33] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.2.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
8%|███████ | 15/195 [00:17<02:36, 1.15it/s] 9%|███████▉ | 17/195 [00:17<02:08, 1.39it/s] [2024-03-18 20:21:34] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.2.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
9%|███████▉ | 17/195 [00:17<02:08, 1.39it/s] [2024-03-18 20:21:34] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.2.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
9%|███████▉ | 17/195 [00:17<02:08, 1.39it/s] 9%|████████▍ | 18/195 [00:17<01:51, 1.59it/s] [2024-03-18 20:21:34] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.2.post_attention_layernorm.weight", shape: (4096,), dtype: float32
9%|████████▍ | 18/195 [00:17<01:51, 1.59it/s] [2024-03-18 20:21:35] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.2.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
9%|████████▍ | 18/195 [00:19<01:51, 1.59it/s] [2024-03-18 20:21:36] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.2.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
9%|████████▍ | 18/195 [00:19<01:51, 1.59it/s] 10%|█████████▎ | 20/195 [00:19<02:10, 1.34it/s] [2024-03-18 20:21:36] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.2.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
10%|█████████▎ | 20/195 [00:20<02:10, 1.34it/s] [2024-03-18 20:21:36] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.2.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
10%|█████████▎ | 20/195 [00:20<02:10, 1.34it/s] 11%|█████████▊ | 21/195 [00:20<02:13, 1.30it/s] [2024-03-18 20:21:37] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.3.input_layernorm.weight", shape: (4096,), dtype: float32
11%|█████████▊ | 21/195 [00:20<02:13, 1.30it/s] [2024-03-18 20:21:37] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.3.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
11%|█████████▊ | 21/195 [00:21<02:13, 1.30it/s] [2024-03-18 20:21:37] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.3.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
11%|█████████▊ | 21/195 [00:21<02:13, 1.30it/s] 12%|██████████▋ | 23/195 [00:21<01:52, 1.52it/s] [2024-03-18 20:21:38] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.3.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
12%|██████████▋ | 23/195 [00:21<01:52, 1.52it/s] [2024-03-18 20:21:38] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.3.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
12%|██████████▋ | 23/195 [00:21<01:52, 1.52it/s] 12%|███████████▏ | 24/195 [00:21<01:39, 1.72it/s] [2024-03-18 20:21:38] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.3.post_attention_layernorm.weight", shape: (4096,), dtype: float32
12%|███████████▏ | 24/195 [00:21<01:39, 1.72it/s] [2024-03-18 20:21:39] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.3.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
12%|███████████▏ | 24/195 [00:23<01:39, 1.72it/s] [2024-03-18 20:21:40] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.3.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
12%|███████████▏ | 24/195 [00:23<01:39, 1.72it/s] 13%|████████████▏ | 26/195 [00:24<02:10, 1.30it/s] [2024-03-18 20:21:41] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.3.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
13%|████████████▏ | 26/195 [00:24<02:10, 1.30it/s] [2024-03-18 20:21:41] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.3.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
13%|████████████▏ | 26/195 [00:24<02:10, 1.30it/s] 14%|████████████▌ | 27/195 [00:24<02:11, 1.28it/s] [2024-03-18 20:21:41] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.4.input_layernorm.weight", shape: (4096,), dtype: float32
14%|████████████▌ | 27/195 [00:24<02:11, 1.28it/s] [2024-03-18 20:21:41] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.4.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
14%|████████████▌ | 27/195 [00:25<02:11, 1.28it/s] [2024-03-18 20:21:42] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.4.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
14%|████████████▌ | 27/195 [00:25<02:11, 1.28it/s] 15%|█████████████▌ | 29/195 [00:25<01:50, 1.51it/s] [2024-03-18 20:21:42] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.4.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
15%|█████████████▌ | 29/195 [00:26<01:50, 1.51it/s] [2024-03-18 20:21:42] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.4.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
15%|█████████████▌ | 29/195 [00:26<01:50, 1.51it/s] 15%|██████████████ | 30/195 [00:26<01:36, 1.70it/s] [2024-03-18 20:21:42] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.4.post_attention_layernorm.weight", shape: (4096,), dtype: float32
15%|██████████████ | 30/195 [00:26<01:36, 1.70it/s] [2024-03-18 20:21:44] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.4.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
15%|██████████████ | 30/195 [00:28<01:36, 1.70it/s] [2024-03-18 20:21:44] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.4.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
15%|██████████████ | 30/195 [00:28<01:36, 1.70it/s] 16%|██████████████▉ | 32/195 [00:28<02:16, 1.19it/s] [2024-03-18 20:21:45] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.4.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
16%|██████████████▉ | 32/195 [00:29<02:16, 1.19it/s] [2024-03-18 20:21:45] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.4.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
16%|██████████████▉ | 32/195 [00:29<02:16, 1.19it/s] 17%|███████████████▍ | 33/195 [00:29<02:17, 1.18it/s] [2024-03-18 20:21:45] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.5.input_layernorm.weight", shape: (4096,), dtype: float32
17%|███████████████▍ | 33/195 [00:29<02:17, 1.18it/s] [2024-03-18 20:21:46] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.5.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
17%|███████████████▍ | 33/195 [00:30<02:17, 1.18it/s] [2024-03-18 20:21:46] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.5.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
17%|███████████████▍ | 33/195 [00:30<02:17, 1.18it/s] 18%|████████████████▎ | 35/195 [00:30<01:56, 1.38it/s] [2024-03-18 20:21:47] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.5.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
18%|████████████████▎ | 35/195 [00:30<01:56, 1.38it/s] [2024-03-18 20:21:47] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.5.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
18%|████████████████▎ | 35/195 [00:30<01:56, 1.38it/s] 18%|████████████████▊ | 36/195 [00:30<01:41, 1.57it/s] [2024-03-18 20:21:47] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.5.post_attention_layernorm.weight", shape: (4096,), dtype: float32
18%|████████████████▊ | 36/195 [00:30<01:41, 1.57it/s] [2024-03-18 20:21:48] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.5.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
18%|████████████████▊ | 36/195 [00:32<01:41, 1.57it/s] [2024-03-18 20:21:49] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.5.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
18%|████████████████▊ | 36/195 [00:32<01:41, 1.57it/s] 19%|█████████████████▋ | 38/195 [00:32<02:05, 1.25it/s] [2024-03-18 20:21:50] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.5.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
19%|█████████████████▋ | 38/195 [00:33<02:05, 1.25it/s] [2024-03-18 20:21:50] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.5.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
19%|█████████████████▋ | 38/195 [00:33<02:05, 1.25it/s] 20%|██████████████████▏ | 39/195 [00:33<02:08, 1.21it/s] [2024-03-18 20:21:50] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.6.input_layernorm.weight", shape: (4096,), dtype: float32
20%|██████████████████▏ | 39/195 [00:33<02:08, 1.21it/s] [2024-03-18 20:21:50] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.6.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
20%|██████████████████▏ | 39/195 [00:34<02:08, 1.21it/s] [2024-03-18 20:21:51] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.6.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
20%|██████████████████▏ | 39/195 [00:34<02:08, 1.21it/s] 21%|███████████████████▏ | 41/195 [00:34<01:46, 1.45it/s] [2024-03-18 20:21:51] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.6.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
21%|███████████████████▏ | 41/195 [00:35<01:46, 1.45it/s] [2024-03-18 20:21:51] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.6.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
21%|███████████████████▏ | 41/195 [00:35<01:46, 1.45it/s] 22%|███████████████████▌ | 42/195 [00:35<01:32, 1.65it/s] [2024-03-18 20:21:51] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.6.post_attention_layernorm.weight", shape: (4096,), dtype: float32
22%|███████████████████▌ | 42/195 [00:35<01:32, 1.65it/s] [2024-03-18 20:21:53] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.6.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
22%|███████████████████▌ | 42/195 [00:36<01:32, 1.65it/s] [2024-03-18 20:21:53] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.6.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
22%|███████████████████▌ | 42/195 [00:37<01:32, 1.65it/s] 23%|████████████████████▌ | 44/195 [00:37<02:01, 1.24it/s] [2024-03-18 20:21:54] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.6.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
23%|████████████████████▌ | 44/195 [00:37<02:01, 1.24it/s] [2024-03-18 20:21:54] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.6.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
23%|████████████████████▌ | 44/195 [00:38<02:01, 1.24it/s] 23%|█████████████████████ | 45/195 [00:38<02:02, 1.23it/s] [2024-03-18 20:21:54] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.7.input_layernorm.weight", shape: (4096,), dtype: float32
23%|█████████████████████ | 45/195 [00:38<02:02, 1.23it/s] [2024-03-18 20:21:55] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.7.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
23%|█████████████████████ | 45/195 [00:38<02:02, 1.23it/s] [2024-03-18 20:21:55] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.7.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
23%|█████████████████████ | 45/195 [00:39<02:02, 1.23it/s] 24%|█████████████████████▉ | 47/195 [00:39<01:41, 1.46it/s] [2024-03-18 20:21:55] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.7.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
24%|█████████████████████▉ | 47/195 [00:39<01:41, 1.46it/s] [2024-03-18 20:21:55] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.7.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
24%|█████████████████████▉ | 47/195 [00:39<01:41, 1.46it/s] 25%|██████████████████████▍ | 48/195 [00:39<01:29, 1.64it/s] [2024-03-18 20:21:55] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.7.post_attention_layernorm.weight", shape: (4096,), dtype: float32
25%|██████████████████████▍ | 48/195 [00:39<01:29, 1.64it/s] [2024-03-18 20:21:57] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.7.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
25%|██████████████████████▍ | 48/195 [00:40<01:29, 1.64it/s] [2024-03-18 20:21:57] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.7.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
25%|██████████████████████▍ | 48/195 [00:41<01:29, 1.64it/s] 26%|███████████████████████▎ | 50/195 [00:41<01:45, 1.37it/s] [2024-03-18 20:21:58] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.7.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
26%|███████████████████████▎ | 50/195 [00:41<01:45, 1.37it/s] [2024-03-18 20:21:58] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.7.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
26%|███████████████████████▎ | 50/195 [00:42<01:45, 1.37it/s] 26%|███████████████████████▊ | 51/195 [00:42<01:47, 1.34it/s] [2024-03-18 20:21:58] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.8.input_layernorm.weight", shape: (4096,), dtype: float32
26%|███████████████████████▊ | 51/195 [00:42<01:47, 1.34it/s] [2024-03-18 20:21:59] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.8.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
26%|███████████████████████▊ | 51/195 [00:42<01:47, 1.34it/s] [2024-03-18 20:21:59] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.8.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
26%|███████████████████████▊ | 51/195 [00:43<01:47, 1.34it/s] 27%|████████████████████████▋ | 53/195 [00:43<01:33, 1.51it/s] [2024-03-18 20:21:59] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.8.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
27%|████████████████████████▋ | 53/195 [00:43<01:33, 1.51it/s] [2024-03-18 20:21:59] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.8.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
27%|████████████████████████▋ | 53/195 [00:43<01:33, 1.51it/s] 28%|█████████████████████████▏ | 54/195 [00:43<01:23, 1.69it/s] [2024-03-18 20:21:59] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.8.post_attention_layernorm.weight", shape: (4096,), dtype: float32
28%|█████████████████████████▏ | 54/195 [00:43<01:23, 1.69it/s] [2024-03-18 20:22:01] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.8.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
28%|█████████████████████████▏ | 54/195 [00:44<01:23, 1.69it/s] [2024-03-18 20:22:01] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.8.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
28%|█████████████████████████▏ | 54/195 [00:45<01:23, 1.69it/s] 29%|██████████████████████████▏ | 56/195 [00:45<01:41, 1.37it/s] [2024-03-18 20:22:02] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.8.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
29%|██████████████████████████▏ | 56/195 [00:46<01:41, 1.37it/s] [2024-03-18 20:22:02] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.8.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
29%|██████████████████████████▏ | 56/195 [00:46<01:41, 1.37it/s] 29%|██████████████████████████▌ | 57/195 [00:46<01:44, 1.32it/s] [2024-03-18 20:22:02] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.9.input_layernorm.weight", shape: (4096,), dtype: float32
29%|██████████████████████████▌ | 57/195 [00:46<01:44, 1.32it/s] [2024-03-18 20:22:03] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.9.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
29%|██████████████████████████▌ | 57/195 [00:46<01:44, 1.32it/s] [2024-03-18 20:22:03] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.9.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
29%|██████████████████████████▌ | 57/195 [00:47<01:44, 1.32it/s] 30%|███████████████████████████▌ | 59/195 [00:47<01:28, 1.54it/s] [2024-03-18 20:22:03] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.9.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
30%|███████████████████████████▌ | 59/195 [00:47<01:28, 1.54it/s] [2024-03-18 20:22:03] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.9.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
30%|███████████████████████████▌ | 59/195 [00:47<01:28, 1.54it/s] 31%|████████████████████████████ | 60/195 [00:47<01:17, 1.74it/s] [2024-03-18 20:22:03] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.9.post_attention_layernorm.weight", shape: (4096,), dtype: float32
31%|████████████████████████████ | 60/195 [00:47<01:17, 1.74it/s] [2024-03-18 20:22:05] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.9.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
31%|████████████████████████████ | 60/195 [00:48<01:17, 1.74it/s] [2024-03-18 20:22:05] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.9.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
31%|████████████████████████████ | 60/195 [00:49<01:17, 1.74it/s] 32%|████████████████████████████▉ | 62/195 [00:49<01:36, 1.37it/s] [2024-03-18 20:22:06] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.9.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
32%|████████████████████████████▉ | 62/195 [00:50<01:36, 1.37it/s] [2024-03-18 20:22:06] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.9.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
32%|████████████████████████████▉ | 62/195 [00:50<01:36, 1.37it/s] 32%|█████████████████████████████▍ | 63/195 [00:50<01:41, 1.29it/s] [2024-03-18 20:22:06] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.10.input_layernorm.weight", shape: (4096,), dtype: float32
32%|█████████████████████████████▍ | 63/195 [00:50<01:41, 1.29it/s] [2024-03-18 20:22:07] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.10.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
32%|█████████████████████████████▍ | 63/195 [00:51<01:41, 1.29it/s] [2024-03-18 20:22:07] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.10.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
32%|█████████████████████████████▍ | 63/195 [00:51<01:41, 1.29it/s] 33%|██████████████████████████████▎ | 65/195 [00:51<01:28, 1.47it/s] [2024-03-18 20:22:08] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.10.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
33%|██████████████████████████████▎ | 65/195 [00:51<01:28, 1.47it/s] [2024-03-18 20:22:08] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.10.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
33%|██████████████████████████████▎ | 65/195 [00:51<01:28, 1.47it/s] 34%|██████████████████████████████▊ | 66/195 [00:51<01:17, 1.67it/s] [2024-03-18 20:22:08] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.10.post_attention_layernorm.weight", shape: (4096,), dtype: float32
34%|██████████████████████████████▊ | 66/195 [00:51<01:17, 1.67it/s] [2024-03-18 20:22:09] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.10.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
34%|██████████████████████████████▊ | 66/195 [00:53<01:17, 1.67it/s] [2024-03-18 20:22:10] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.10.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
34%|██████████████████████████████▊ | 66/195 [00:53<01:17, 1.67it/s] 35%|███████████████████████████████▋ | 68/195 [00:53<01:36, 1.32it/s] [2024-03-18 20:22:10] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.10.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
35%|███████████████████████████████▋ | 68/195 [00:54<01:36, 1.32it/s] [2024-03-18 20:22:11] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.10.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
35%|███████████████████████████████▋ | 68/195 [00:54<01:36, 1.32it/s] 35%|████████████████████████████████▏ | 69/195 [00:54<01:37, 1.29it/s] [2024-03-18 20:22:11] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.11.input_layernorm.weight", shape: (4096,), dtype: float32
35%|████████████████████████████████▏ | 69/195 [00:54<01:37, 1.29it/s] [2024-03-18 20:22:11] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.11.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
35%|████████████████████████████████▏ | 69/195 [00:55<01:37, 1.29it/s] [2024-03-18 20:22:12] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.11.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
35%|████████████████████████████████▏ | 69/195 [00:55<01:37, 1.29it/s] 36%|█████████████████████████████████▏ | 71/195 [00:55<01:21, 1.51it/s] [2024-03-18 20:22:12] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.11.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
36%|█████████████████████████████████▏ | 71/195 [00:55<01:21, 1.51it/s] [2024-03-18 20:22:12] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.11.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
36%|█████████████████████████████████▏ | 71/195 [00:55<01:21, 1.51it/s] 37%|█████████████████████████████████▌ | 72/195 [00:55<01:11, 1.72it/s] [2024-03-18 20:22:12] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.11.post_attention_layernorm.weight", shape: (4096,), dtype: float32
37%|█████████████████████████████████▌ | 72/195 [00:55<01:11, 1.72it/s] [2024-03-18 20:22:13] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.11.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
37%|█████████████████████████████████▌ | 72/195 [00:57<01:11, 1.72it/s] [2024-03-18 20:22:14] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.11.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
37%|█████████████████████████████████▌ | 72/195 [00:57<01:11, 1.72it/s] 38%|██████████████████████████████████▌ | 74/195 [00:57<01:29, 1.35it/s] [2024-03-18 20:22:14] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.11.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
38%|██████████████████████████████████▌ | 74/195 [00:58<01:29, 1.35it/s] [2024-03-18 20:22:15] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.11.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
38%|██████████████████████████████████▌ | 74/195 [00:58<01:29, 1.35it/s] 38%|███████████████████████████████████ | 75/195 [00:58<01:32, 1.30it/s] [2024-03-18 20:22:15] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.12.input_layernorm.weight", shape: (4096,), dtype: float32
38%|███████████████████████████████████ | 75/195 [00:58<01:32, 1.30it/s] [2024-03-18 20:22:15] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.12.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
38%|███████████████████████████████████ | 75/195 [00:59<01:32, 1.30it/s] [2024-03-18 20:22:16] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.12.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
38%|███████████████████████████████████ | 75/195 [00:59<01:32, 1.30it/s] 39%|███████████████████████████████████▉ | 77/195 [00:59<01:19, 1.48it/s] [2024-03-18 20:22:16] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.12.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
39%|███████████████████████████████████▉ | 77/195 [01:00<01:19, 1.48it/s] [2024-03-18 20:22:16] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.12.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
39%|███████████████████████████████████▉ | 77/195 [01:00<01:19, 1.48it/s] 40%|████████████████████████████████████▍ | 78/195 [01:00<01:09, 1.69it/s] [2024-03-18 20:22:16] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.12.post_attention_layernorm.weight", shape: (4096,), dtype: float32
40%|████████████████████████████████████▍ | 78/195 [01:00<01:09, 1.69it/s] [2024-03-18 20:22:17] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.12.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
40%|████████████████████████████████████▍ | 78/195 [01:01<01:09, 1.69it/s] [2024-03-18 20:22:18] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.12.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
40%|████████████████████████████████████▍ | 78/195 [01:01<01:09, 1.69it/s] 41%|█████████████████████████████████████▎ | 80/195 [01:01<01:22, 1.40it/s] [2024-03-18 20:22:18] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.12.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
41%|█████████████████████████████████████▎ | 80/195 [01:02<01:22, 1.40it/s] [2024-03-18 20:22:19] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.12.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
41%|█████████████████████████████████████▎ | 80/195 [01:02<01:22, 1.40it/s] 42%|█████████████████████████████████████▊ | 81/195 [01:02<01:25, 1.34it/s] [2024-03-18 20:22:19] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.13.input_layernorm.weight", shape: (4096,), dtype: float32
42%|█████████████████████████████████████▊ | 81/195 [01:02<01:25, 1.34it/s] [2024-03-18 20:22:20] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.13.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
42%|█████████████████████████████████████▊ | 81/195 [01:03<01:25, 1.34it/s] [2024-03-18 20:22:20] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.13.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
42%|█████████████████████████████████████▊ | 81/195 [01:03<01:25, 1.34it/s] 43%|██████████████████████████████████████▋ | 83/195 [01:03<01:15, 1.49it/s] [2024-03-18 20:22:20] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.13.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
43%|██████████████████████████████████████▋ | 83/195 [01:04<01:15, 1.49it/s] [2024-03-18 20:22:20] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.13.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
43%|██████████████████████████████████████▋ | 83/195 [01:04<01:15, 1.49it/s] 43%|███████████████████████████████████████▏ | 84/195 [01:04<01:05, 1.70it/s] [2024-03-18 20:22:20] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.13.post_attention_layernorm.weight", shape: (4096,), dtype: float32
43%|███████████████████████████████████████▏ | 84/195 [01:04<01:05, 1.70it/s] [2024-03-18 20:22:21] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.13.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
43%|███████████████████████████████████████▏ | 84/195 [01:05<01:05, 1.70it/s] [2024-03-18 20:22:22] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.13.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
43%|███████████████████████████████████████▏ | 84/195 [01:05<01:05, 1.70it/s] 44%|████████████████████████████████████████▏ | 86/195 [01:06<01:17, 1.40it/s] [2024-03-18 20:22:23] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.13.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
44%|████████████████████████████████████████▏ | 86/195 [01:06<01:17, 1.40it/s] [2024-03-18 20:22:23] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.13.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
44%|████████████████████████████████████████▏ | 86/195 [01:06<01:17, 1.40it/s] 45%|████████████████████████████████████████▌ | 87/195 [01:06<01:20, 1.35it/s] [2024-03-18 20:22:23] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.14.input_layernorm.weight", shape: (4096,), dtype: float32
45%|████████████████████████████████████████▌ | 87/195 [01:06<01:20, 1.35it/s] [2024-03-18 20:22:24] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.14.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
45%|████████████████████████████████████████▌ | 87/195 [01:07<01:20, 1.35it/s] [2024-03-18 20:22:24] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.14.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
45%|████████████████████████████████████████▌ | 87/195 [01:07<01:20, 1.35it/s] 46%|█████████████████████████████████████████▌ | 89/195 [01:07<01:10, 1.51it/s] [2024-03-18 20:22:24] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.14.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
46%|█████████████████████████████████████████▌ | 89/195 [01:08<01:10, 1.51it/s] [2024-03-18 20:22:24] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.14.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
46%|█████████████████████████████████████████▌ | 89/195 [01:08<01:10, 1.51it/s] 46%|██████████████████████████████████████████ | 90/195 [01:08<01:01, 1.72it/s] [2024-03-18 20:22:24] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.14.post_attention_layernorm.weight", shape: (4096,), dtype: float32
46%|██████████████████████████████████████████ | 90/195 [01:08<01:01, 1.72it/s] [2024-03-18 20:22:26] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.14.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
46%|██████████████████████████████████████████ | 90/195 [01:09<01:01, 1.72it/s] [2024-03-18 20:22:26] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.14.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
46%|██████████████████████████████████████████ | 90/195 [01:10<01:01, 1.72it/s] 47%|██████████████████████████████████████████▉ | 92/195 [01:10<01:20, 1.28it/s] [2024-03-18 20:22:27] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.14.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
47%|██████████████████████████████████████████▉ | 92/195 [01:11<01:20, 1.28it/s] [2024-03-18 20:22:27] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.14.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
47%|██████████████████████████████████████████▉ | 92/195 [01:11<01:20, 1.28it/s] 48%|███████████████████████████████████████████▍ | 93/195 [01:11<01:23, 1.22it/s] [2024-03-18 20:22:27] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.15.input_layernorm.weight", shape: (4096,), dtype: float32
48%|███████████████████████████████████████████▍ | 93/195 [01:11<01:23, 1.22it/s] [2024-03-18 20:22:28] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.15.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
48%|███████████████████████████████████████████▍ | 93/195 [01:12<01:23, 1.22it/s] [2024-03-18 20:22:28] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.15.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
48%|███████████████████████████████████████████▍ | 93/195 [01:12<01:23, 1.22it/s] 49%|████████████████████████████████████████████▎ | 95/195 [01:12<01:08, 1.45it/s] [2024-03-18 20:22:29] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.15.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
49%|████████████████████████████████████████████▎ | 95/195 [01:12<01:08, 1.45it/s] [2024-03-18 20:22:29] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.15.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
49%|████████████████████████████████████████████▎ | 95/195 [01:12<01:08, 1.45it/s] 49%|████████████████████████████████████████████▊ | 96/195 [01:12<01:01, 1.61it/s] [2024-03-18 20:22:29] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.15.post_attention_layernorm.weight", shape: (4096,), dtype: float32
49%|████████████████████████████████████████████▊ | 96/195 [01:12<01:01, 1.61it/s] [2024-03-18 20:22:31] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.15.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
49%|████████████████████████████████████████████▊ | 96/195 [01:15<01:01, 1.61it/s] [2024-03-18 20:22:32] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.15.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
49%|████████████████████████████████████████████▊ | 96/195 [01:15<01:01, 1.61it/s] 50%|█████████████████████████████████████████████▋ | 98/195 [01:15<01:33, 1.04it/s] [2024-03-18 20:22:32] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.15.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
50%|█████████████████████████████████████████████▋ | 98/195 [01:16<01:33, 1.04it/s] [2024-03-18 20:22:33] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.15.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
50%|█████████████████████████████████████████████▋ | 98/195 [01:16<01:33, 1.04it/s] 51%|██████████████████████████████████████████████▏ | 99/195 [01:16<01:29, 1.07it/s] [2024-03-18 20:22:33] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.16.input_layernorm.weight", shape: (4096,), dtype: float32
51%|██████████████████████████████████████████████▏ | 99/195 [01:16<01:29, 1.07it/s] [2024-03-18 20:22:33] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.16.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
51%|██████████████████████████████████████████████▏ | 99/195 [01:17<01:29, 1.07it/s] [2024-03-18 20:22:33] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.16.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
51%|██████████████████████████████████████████████▏ | 99/195 [01:17<01:29, 1.07it/s] 52%|██████████████████████████████████████████████▌ | 101/195 [01:17<01:11, 1.32it/s] [2024-03-18 20:22:34] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.16.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
52%|██████████████████████████████████████████████▌ | 101/195 [01:17<01:11, 1.32it/s] [2024-03-18 20:22:34] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.16.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
52%|██████████████████████████████████████████████▌ | 101/195 [01:17<01:11, 1.32it/s] 52%|███████████████████████████████████████████████ | 102/195 [01:17<01:02, 1.49it/s] [2024-03-18 20:22:34] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.16.post_attention_layernorm.weight", shape: (4096,), dtype: float32
52%|███████████████████████████████████████████████ | 102/195 [01:17<01:02, 1.49it/s] [2024-03-18 20:22:35] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.16.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
52%|███████████████████████████████████████████████ | 102/195 [01:19<01:02, 1.49it/s] [2024-03-18 20:22:36] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.16.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
52%|███████████████████████████████████████████████ | 102/195 [01:19<01:02, 1.49it/s] 53%|████████████████████████████████████████████████ | 104/195 [01:19<01:10, 1.29it/s] [2024-03-18 20:22:36] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.16.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
53%|████████████████████████████████████████████████ | 104/195 [01:20<01:10, 1.29it/s] [2024-03-18 20:22:37] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.16.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
53%|████████████████████████████████████████████████ | 104/195 [01:20<01:10, 1.29it/s] 54%|████████████████████████████████████████████████▍ | 105/195 [01:20<01:10, 1.27it/s] [2024-03-18 20:22:37] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.17.input_layernorm.weight", shape: (4096,), dtype: float32
54%|████████████████████████████████████████████████▍ | 105/195 [01:20<01:10, 1.27it/s] [2024-03-18 20:22:37] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.17.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
54%|████████████████████████████████████████████████▍ | 105/195 [01:21<01:10, 1.27it/s] [2024-03-18 20:22:37] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.17.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
54%|████████████████████████████████████████████████▍ | 105/195 [01:21<01:10, 1.27it/s] 55%|█████████████████████████████████████████████████▍ | 107/195 [01:21<00:58, 1.49it/s] [2024-03-18 20:22:38] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.17.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
55%|█████████████████████████████████████████████████▍ | 107/195 [01:21<00:58, 1.49it/s] [2024-03-18 20:22:38] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.17.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
55%|█████████████████████████████████████████████████▍ | 107/195 [01:21<00:58, 1.49it/s] 55%|█████████████████████████████████████████████████▊ | 108/195 [01:21<00:51, 1.69it/s] [2024-03-18 20:22:38] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.17.post_attention_layernorm.weight", shape: (4096,), dtype: float32
55%|█████████████████████████████████████████████████▊ | 108/195 [01:21<00:51, 1.69it/s] [2024-03-18 20:22:39] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.17.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
55%|█████████████████████████████████████████████████▊ | 108/195 [01:23<00:51, 1.69it/s] [2024-03-18 20:22:40] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.17.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
55%|█████████████████████████████████████████████████▊ | 108/195 [01:23<00:51, 1.69it/s] 56%|██████████████████████████████████████████████████▊ | 110/195 [01:23<01:00, 1.39it/s] [2024-03-18 20:22:40] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.17.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
56%|██████████████████████████████████████████████████▊ | 110/195 [01:24<01:00, 1.39it/s] [2024-03-18 20:22:41] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.17.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
56%|██████████████████████████████████████████████████▊ | 110/195 [01:24<01:00, 1.39it/s] 57%|███████████████████████████████████████████████████▏ | 111/195 [01:24<01:03, 1.32it/s] [2024-03-18 20:22:41] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.18.input_layernorm.weight", shape: (4096,), dtype: float32
57%|███████████████████████████████████████████████████▏ | 111/195 [01:24<01:03, 1.32it/s] [2024-03-18 20:22:41] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.18.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
57%|███████████████████████████████████████████████████▏ | 111/195 [01:25<01:03, 1.32it/s] [2024-03-18 20:22:42] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.18.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
57%|███████████████████████████████████████████████████▏ | 111/195 [01:25<01:03, 1.32it/s] 58%|████████████████████████████████████████████████████▏ | 113/195 [01:25<00:53, 1.52it/s] [2024-03-18 20:22:42] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.18.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
58%|████████████████████████████████████████████████████▏ | 113/195 [01:25<00:53, 1.52it/s] [2024-03-18 20:22:42] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.18.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
58%|████████████████████████████████████████████████████▏ | 113/195 [01:25<00:53, 1.52it/s] 58%|████████████████████████████████████████████████████▌ | 114/195 [01:25<00:47, 1.70it/s] [2024-03-18 20:22:42] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.18.post_attention_layernorm.weight", shape: (4096,), dtype: float32
58%|████████████████████████████████████████████████████▌ | 114/195 [01:25<00:47, 1.70it/s] [2024-03-18 20:22:43] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.18.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
58%|████████████████████████████████████████████████████▌ | 114/195 [01:27<00:47, 1.70it/s] [2024-03-18 20:22:44] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.18.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
58%|████████████████████████████████████████████████████▌ | 114/195 [01:27<00:47, 1.70it/s] 59%|█████████████████████████████████████████████████████▌ | 116/195 [01:27<00:58, 1.34it/s] [2024-03-18 20:22:45] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.18.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
59%|█████████████████████████████████████████████████████▌ | 116/195 [01:28<00:58, 1.34it/s] [2024-03-18 20:22:45] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.18.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
59%|█████████████████████████████████████████████████████▌ | 116/195 [01:28<00:58, 1.34it/s] 60%|██████████████████████████████████████████████████████ | 117/195 [01:28<01:00, 1.28it/s] [2024-03-18 20:22:45] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.19.input_layernorm.weight", shape: (4096,), dtype: float32
60%|██████████████████████████████████████████████████████ | 117/195 [01:28<01:00, 1.28it/s] [2024-03-18 20:22:45] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.19.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
60%|██████████████████████████████████████████████████████ | 117/195 [01:29<01:00, 1.28it/s] [2024-03-18 20:22:46] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.19.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
60%|██████████████████████████████████████████████████████ | 117/195 [01:29<01:00, 1.28it/s] 61%|██████████████████████████████████████████████████████▉ | 119/195 [01:29<00:50, 1.51it/s] [2024-03-18 20:22:46] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.19.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
61%|██████████████████████████████████████████████████████▉ | 119/195 [01:30<00:50, 1.51it/s] [2024-03-18 20:22:46] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.19.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
61%|██████████████████████████████████████████████████████▉ | 119/195 [01:30<00:50, 1.51it/s] 62%|███████████████████████████████████████████████████████▍ | 120/195 [01:30<00:43, 1.71it/s] [2024-03-18 20:22:46] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.19.post_attention_layernorm.weight", shape: (4096,), dtype: float32
62%|███████████████████████████████████████████████████████▍ | 120/195 [01:30<00:43, 1.71it/s] [2024-03-18 20:22:47] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.19.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
62%|███████████████████████████████████████████████████████▍ | 120/195 [01:31<00:43, 1.71it/s] [2024-03-18 20:22:48] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.19.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
62%|███████████████████████████████████████████████████████▍ | 120/195 [01:31<00:43, 1.71it/s] 63%|████████████████████████████████████████████████████████▎ | 122/195 [01:32<00:53, 1.38it/s] [2024-03-18 20:22:49] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.19.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
63%|████████████████████████████████████████████████████████▎ | 122/195 [01:32<00:53, 1.38it/s] [2024-03-18 20:22:49] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.19.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
63%|████████████████████████████████████████████████████████▎ | 122/195 [01:32<00:53, 1.38it/s] 63%|████████████████████████████████████████████████████████▊ | 123/195 [01:32<00:54, 1.33it/s] [2024-03-18 20:22:49] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.20.input_layernorm.weight", shape: (4096,), dtype: float32
63%|████████████████████████████████████████████████████████▊ | 123/195 [01:32<00:54, 1.33it/s] [2024-03-18 20:22:49] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.20.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
63%|████████████████████████████████████████████████████████▊ | 123/195 [01:33<00:54, 1.33it/s] [2024-03-18 20:22:50] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.20.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
63%|████████████████████████████████████████████████████████▊ | 123/195 [01:33<00:54, 1.33it/s] 64%|█████████████████████████████████████████████████████████▋ | 125/195 [01:33<00:45, 1.54it/s] [2024-03-18 20:22:50] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.20.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
64%|█████████████████████████████████████████████████████████▋ | 125/195 [01:34<00:45, 1.54it/s] [2024-03-18 20:22:50] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.20.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
64%|█████████████████████████████████████████████████████████▋ | 125/195 [01:34<00:45, 1.54it/s] 65%|██████████████████████████████████████████████████████████▏ | 126/195 [01:34<00:39, 1.74it/s] [2024-03-18 20:22:50] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.20.post_attention_layernorm.weight", shape: (4096,), dtype: float32
65%|██████████████████████████████████████████████████████████▏ | 126/195 [01:34<00:39, 1.74it/s] [2024-03-18 20:22:51] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.20.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
65%|██████████████████████████████████████████████████████████▏ | 126/195 [01:35<00:39, 1.74it/s] [2024-03-18 20:22:52] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.20.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
65%|██████████████████████████████████████████████████████████▏ | 126/195 [01:35<00:39, 1.74it/s] 66%|███████████████████████████████████████████████████████████ | 128/195 [01:35<00:47, 1.42it/s] [2024-03-18 20:22:53] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.20.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
66%|███████████████████████████████████████████████████████████ | 128/195 [01:36<00:47, 1.42it/s] [2024-03-18 20:22:53] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.20.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
66%|███████████████████████████████████████████████████████████ | 128/195 [01:36<00:47, 1.42it/s] 66%|███████████████████████████████████████████████████████████▌ | 129/195 [01:36<00:49, 1.34it/s] [2024-03-18 20:22:53] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.21.input_layernorm.weight", shape: (4096,), dtype: float32
66%|███████████████████████████████████████████████████████████▌ | 129/195 [01:36<00:49, 1.34it/s] [2024-03-18 20:22:54] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.21.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
66%|███████████████████████████████████████████████████████████▌ | 129/195 [01:37<00:49, 1.34it/s] [2024-03-18 20:22:54] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.21.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
66%|███████████████████████████████████████████████████████████▌ | 129/195 [01:37<00:49, 1.34it/s] 67%|████████████████████████████████████████████████████████████▍ | 131/195 [01:37<00:43, 1.48it/s] [2024-03-18 20:22:54] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.21.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
67%|████████████████████████████████████████████████████████████▍ | 131/195 [01:38<00:43, 1.48it/s] [2024-03-18 20:22:54] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.21.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
67%|████████████████████████████████████████████████████████████▍ | 131/195 [01:38<00:43, 1.48it/s] 68%|████████████████████████████████████████████████████████████▉ | 132/195 [01:38<00:37, 1.66it/s] [2024-03-18 20:22:54] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.21.post_attention_layernorm.weight", shape: (4096,), dtype: float32
68%|████████████████████████████████████████████████████████████▉ | 132/195 [01:38<00:37, 1.66it/s] [2024-03-18 20:22:56] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.21.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
68%|████████████████████████████████████████████████████████████▉ | 132/195 [01:39<00:37, 1.66it/s] [2024-03-18 20:22:56] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.21.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
68%|████████████████████████████████████████████████████████████▉ | 132/195 [01:40<00:37, 1.66it/s] 69%|█████████████████████████████████████████████████████████████▊ | 134/195 [01:40<00:45, 1.35it/s] [2024-03-18 20:22:57] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.21.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
69%|█████████████████████████████████████████████████████████████▊ | 134/195 [01:40<00:45, 1.35it/s] [2024-03-18 20:22:57] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.21.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
69%|█████████████████████████████████████████████████████████████▊ | 134/195 [01:41<00:45, 1.35it/s] 69%|██████████████████████████████████████████████████████████████▎ | 135/195 [01:41<00:45, 1.31it/s] [2024-03-18 20:22:57] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.22.input_layernorm.weight", shape: (4096,), dtype: float32
69%|██████████████████████████████████████████████████████████████▎ | 135/195 [01:41<00:45, 1.31it/s] [2024-03-18 20:22:58] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.22.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
69%|██████████████████████████████████████████████████████████████▎ | 135/195 [01:41<00:45, 1.31it/s] [2024-03-18 20:22:58] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.22.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
69%|██████████████████████████████████████████████████████████████▎ | 135/195 [01:42<00:45, 1.31it/s] 70%|███████████████████████████████████████████████████████████████▏ | 137/195 [01:42<00:38, 1.53it/s] [2024-03-18 20:22:58] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.22.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
70%|███████████████████████████████████████████████████████████████▏ | 137/195 [01:42<00:38, 1.53it/s] [2024-03-18 20:22:58] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.22.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
70%|███████████████████████████████████████████████████████████████▏ | 137/195 [01:42<00:38, 1.53it/s] 71%|███████████████████████████████████████████████████████████████▋ | 138/195 [01:42<00:33, 1.72it/s] [2024-03-18 20:22:58] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.22.post_attention_layernorm.weight", shape: (4096,), dtype: float32
71%|███████████████████████████████████████████████████████████████▋ | 138/195 [01:42<00:33, 1.72it/s] [2024-03-18 20:23:00] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.22.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
71%|███████████████████████████████████████████████████████████████▋ | 138/195 [01:43<00:33, 1.72it/s] [2024-03-18 20:23:00] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.22.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
71%|███████████████████████████████████████████████████████████████▋ | 138/195 [01:44<00:33, 1.72it/s] 72%|████████████████████████████████████████████████████████████████▌ | 140/195 [01:44<00:40, 1.37it/s] [2024-03-18 20:23:01] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.22.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
72%|████████████████████████████████████████████████████████████████▌ | 140/195 [01:44<00:40, 1.37it/s] [2024-03-18 20:23:01] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.22.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
72%|████████████████████████████████████████████████████████████████▌ | 140/195 [01:45<00:40, 1.37it/s] 72%|█████████████████████████████████████████████████████████████████ | 141/195 [01:45<00:41, 1.30it/s] [2024-03-18 20:23:01] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.23.input_layernorm.weight", shape: (4096,), dtype: float32
72%|█████████████████████████████████████████████████████████████████ | 141/195 [01:45<00:41, 1.30it/s] [2024-03-18 20:23:02] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.23.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
72%|█████████████████████████████████████████████████████████████████ | 141/195 [01:45<00:41, 1.30it/s] [2024-03-18 20:23:02] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.23.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
72%|█████████████████████████████████████████████████████████████████ | 141/195 [01:46<00:41, 1.30it/s] 73%|██████████████████████████████████████████████████████████████████ | 143/195 [01:46<00:34, 1.51it/s] [2024-03-18 20:23:02] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.23.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
73%|██████████████████████████████████████████████████████████████████ | 143/195 [01:46<00:34, 1.51it/s] [2024-03-18 20:23:02] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.23.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
73%|██████████████████████████████████████████████████████████████████ | 143/195 [01:46<00:34, 1.51it/s] 74%|██████████████████████████████████████████████████████████████████▍ | 144/195 [01:46<00:29, 1.71it/s] [2024-03-18 20:23:02] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.23.post_attention_layernorm.weight", shape: (4096,), dtype: float32
74%|██████████████████████████████████████████████████████████████████▍ | 144/195 [01:46<00:29, 1.71it/s] [2024-03-18 20:23:04] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.23.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
74%|██████████████████████████████████████████████████████████████████▍ | 144/195 [01:47<00:29, 1.71it/s] [2024-03-18 20:23:04] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.23.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
74%|██████████████████████████████████████████████████████████████████▍ | 144/195 [01:48<00:29, 1.71it/s] 75%|███████████████████████████████████████████████████████████████████▍ | 146/195 [01:48<00:36, 1.36it/s] [2024-03-18 20:23:05] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.23.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
75%|███████████████████████████████████████████████████████████████████▍ | 146/195 [01:49<00:36, 1.36it/s] [2024-03-18 20:23:05] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.23.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
75%|███████████████████████████████████████████████████████████████████▍ | 146/195 [01:49<00:36, 1.36it/s] 75%|███████████████████████████████████████████████████████████████████▊ | 147/195 [01:49<00:36, 1.30it/s] [2024-03-18 20:23:05] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.24.input_layernorm.weight", shape: (4096,), dtype: float32
75%|███████████████████████████████████████████████████████████████████▊ | 147/195 [01:49<00:36, 1.30it/s] [2024-03-18 20:23:06] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.24.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
75%|███████████████████████████████████████████████████████████████████▊ | 147/195 [01:50<00:36, 1.30it/s] [2024-03-18 20:23:06] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.24.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
75%|███████████████████████████████████████████████████████████████████▊ | 147/195 [01:50<00:36, 1.30it/s] 76%|████████████████████████████████████████████████████████████████████▊ | 149/195 [01:50<00:31, 1.48it/s] [2024-03-18 20:23:07] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.24.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
76%|████████████████████████████████████████████████████████████████████▊ | 149/195 [01:50<00:31, 1.48it/s] [2024-03-18 20:23:07] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.24.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
76%|████████████████████████████████████████████████████████████████████▊ | 149/195 [01:50<00:31, 1.48it/s] 77%|█████████████████████████████████████████████████████████████████████▏ | 150/195 [01:50<00:26, 1.68it/s] [2024-03-18 20:23:07] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.24.post_attention_layernorm.weight", shape: (4096,), dtype: float32
77%|█████████████████████████████████████████████████████████████████████▏ | 150/195 [01:50<00:26, 1.68it/s] [2024-03-18 20:23:09] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.24.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
77%|█████████████████████████████████████████████████████████████████████▏ | 150/195 [01:52<00:26, 1.68it/s] [2024-03-18 20:23:09] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.24.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
77%|█████████████████████████████████████████████████████████████████████▏ | 150/195 [01:53<00:26, 1.68it/s] 78%|██████████████████████████████████████████████████████████████████████▏ | 152/195 [01:53<00:36, 1.17it/s] [2024-03-18 20:23:10] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.24.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
78%|██████████████████████████████████████████████████████████████████████▏ | 152/195 [01:54<00:36, 1.17it/s] [2024-03-18 20:23:10] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.24.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
78%|██████████████████████████████████████████████████████████████████████▏ | 152/195 [01:54<00:36, 1.17it/s] 78%|██████████████████████████████████████████████████████████████████████▌ | 153/195 [01:54<00:37, 1.13it/s] [2024-03-18 20:23:10] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.25.input_layernorm.weight", shape: (4096,), dtype: float32
78%|██████████████████████████████████████████████████████████████████████▌ | 153/195 [01:54<00:37, 1.13it/s] [2024-03-18 20:23:11] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.25.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
78%|██████████████████████████████████████████████████████████████████████▌ | 153/195 [01:55<00:37, 1.13it/s] [2024-03-18 20:23:11] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.25.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
78%|██████████████████████████████████████████████████████████████████████▌ | 153/195 [01:55<00:37, 1.13it/s] 79%|███████████████████████████████████████████████████████████████████████▌ | 155/195 [01:55<00:29, 1.34it/s] [2024-03-18 20:23:11] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.25.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
79%|███████████████████████████████████████████████████████████████████████▌ | 155/195 [01:55<00:29, 1.34it/s] [2024-03-18 20:23:12] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.25.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
79%|███████████████████████████████████████████████████████████████████████▌ | 155/195 [01:55<00:29, 1.34it/s] 80%|████████████████████████████████████████████████████████████████████████ | 156/195 [01:55<00:25, 1.52it/s] [2024-03-18 20:23:12] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.25.post_attention_layernorm.weight", shape: (4096,), dtype: float32
80%|████████████████████████████████████████████████████████████████████████ | 156/195 [01:55<00:25, 1.52it/s] [2024-03-18 20:23:13] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.25.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
80%|████████████████████████████████████████████████████████████████████████ | 156/195 [01:57<00:25, 1.52it/s] [2024-03-18 20:23:14] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.25.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
80%|████████████████████████████████████████████████████████████████████████ | 156/195 [01:57<00:25, 1.52it/s] 81%|████████████████████████████████████████████████████████████████████████▉ | 158/195 [01:57<00:29, 1.24it/s] [2024-03-18 20:23:14] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.25.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
81%|████████████████████████████████████████████████████████████████████████▉ | 158/195 [01:58<00:29, 1.24it/s] [2024-03-18 20:23:15] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.25.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
81%|████████████████████████████████████████████████████████████████████████▉ | 158/195 [01:58<00:29, 1.24it/s] 82%|█████████████████████████████████████████████████████████████████████████▍ | 159/195 [01:58<00:29, 1.21it/s] [2024-03-18 20:23:15] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.26.input_layernorm.weight", shape: (4096,), dtype: float32
82%|█████████████████████████████████████████████████████████████████████████▍ | 159/195 [01:58<00:29, 1.21it/s] [2024-03-18 20:23:15] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.26.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
82%|█████████████████████████████████████████████████████████████████████████▍ | 159/195 [01:59<00:29, 1.21it/s] [2024-03-18 20:23:15] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.26.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
82%|█████████████████████████████████████████████████████████████████████████▍ | 159/195 [01:59<00:29, 1.21it/s] 83%|██████████████████████████████████████████████████████████████████████████▎ | 161/195 [01:59<00:23, 1.44it/s] [2024-03-18 20:23:16] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.26.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
83%|██████████████████████████████████████████████████████████████████████████▎ | 161/195 [01:59<00:23, 1.44it/s] [2024-03-18 20:23:16] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.26.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
83%|██████████████████████████████████████████████████████████████████████████▎ | 161/195 [01:59<00:23, 1.44it/s] 83%|██████████████████████████████████████████████████████████████████████████▊ | 162/195 [01:59<00:20, 1.63it/s] [2024-03-18 20:23:16] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.26.post_attention_layernorm.weight", shape: (4096,), dtype: float32
83%|██████████████████████████████████████████████████████████████████████████▊ | 162/195 [01:59<00:20, 1.63it/s] [2024-03-18 20:23:17] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.26.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
83%|██████████████████████████████████████████████████████████████████████████▊ | 162/195 [02:01<00:20, 1.63it/s] [2024-03-18 20:23:18] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.26.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
83%|██████████████████████████████████████████████████████████████████████████▊ | 162/195 [02:01<00:20, 1.63it/s] 84%|███████████████████████████████████████████████████████████████████████████▋ | 164/195 [02:01<00:24, 1.28it/s] [2024-03-18 20:23:19] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.26.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
84%|███████████████████████████████████████████████████████████████████████████▋ | 164/195 [02:02<00:24, 1.28it/s] [2024-03-18 20:23:19] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.26.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
84%|███████████████████████████████████████████████████████████████████████████▋ | 164/195 [02:02<00:24, 1.28it/s] 85%|████████████████████████████████████████████████████████████████████████████▏ | 165/195 [02:02<00:23, 1.26it/s] [2024-03-18 20:23:19] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.27.input_layernorm.weight", shape: (4096,), dtype: float32
85%|████████████████████████████████████████████████████████████████████████████▏ | 165/195 [02:02<00:23, 1.26it/s] [2024-03-18 20:23:19] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.27.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
85%|████████████████████████████████████████████████████████████████████████████▏ | 165/195 [02:03<00:23, 1.26it/s] [2024-03-18 20:23:20] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.27.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
85%|████████████████████████████████████████████████████████████████████████████▏ | 165/195 [02:03<00:23, 1.26it/s] 86%|█████████████████████████████████████████████████████████████████████████████ | 167/195 [02:03<00:18, 1.49it/s] [2024-03-18 20:23:20] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.27.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
86%|█████████████████████████████████████████████████████████████████████████████ | 167/195 [02:04<00:18, 1.49it/s] [2024-03-18 20:23:20] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.27.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
86%|█████████████████████████████████████████████████████████████████████████████ | 167/195 [02:04<00:18, 1.49it/s] 86%|█████████████████████████████████████████████████████████████████████████████▌ | 168/195 [02:04<00:15, 1.69it/s] [2024-03-18 20:23:20] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.27.post_attention_layernorm.weight", shape: (4096,), dtype: float32
86%|█████████████████████████████████████████████████████████████████████████████▌ | 168/195 [02:04<00:15, 1.69it/s] [2024-03-18 20:23:21] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.27.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
86%|█████████████████████████████████████████████████████████████████████████████▌ | 168/195 [02:05<00:15, 1.69it/s] [2024-03-18 20:23:22] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.27.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
86%|█████████████████████████████████████████████████████████████████████████████▌ | 168/195 [02:05<00:15, 1.69it/s] 87%|██████████████████████████████████████████████████████████████████████████████▍ | 170/195 [02:05<00:17, 1.41it/s] [2024-03-18 20:23:22] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.27.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
87%|██████████████████████████████████████████████████████████████████████████████▍ | 170/195 [02:06<00:17, 1.41it/s] [2024-03-18 20:23:23] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.27.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
87%|██████████████████████████████████████████████████████████████████████████████▍ | 170/195 [02:06<00:17, 1.41it/s] 88%|██████████████████████████████████████████████████████████████████████████████▉ | 171/195 [02:06<00:17, 1.36it/s] [2024-03-18 20:23:23] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.28.input_layernorm.weight", shape: (4096,), dtype: float32
88%|██████████████████████████████████████████████████████████████████████████████▉ | 171/195 [02:06<00:17, 1.36it/s] [2024-03-18 20:23:23] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.28.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
88%|██████████████████████████████████████████████████████████████████████████████▉ | 171/195 [02:07<00:17, 1.36it/s] [2024-03-18 20:23:24] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.28.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
88%|██████████████████████████████████████████████████████████████████████████████▉ | 171/195 [02:07<00:17, 1.36it/s] 89%|███████████████████████████████████████████████████████████████████████████████▊ | 173/195 [02:07<00:14, 1.55it/s] [2024-03-18 20:23:24] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.28.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
89%|███████████████████████████████████████████████████████████████████████████████▊ | 173/195 [02:07<00:14, 1.55it/s] [2024-03-18 20:23:24] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.28.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
89%|███████████████████████████████████████████████████████████████████████████████▊ | 173/195 [02:07<00:14, 1.55it/s] 89%|████████████████████████████████████████████████████████████████████████████████▎ | 174/195 [02:08<00:12, 1.75it/s] [2024-03-18 20:23:24] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.28.post_attention_layernorm.weight", shape: (4096,), dtype: float32
89%|████████████████████████████████████████████████████████████████████████████████▎ | 174/195 [02:08<00:12, 1.75it/s] [2024-03-18 20:23:25] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.28.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
89%|████████████████████████████████████████████████████████████████████████████████▎ | 174/195 [02:09<00:12, 1.75it/s] [2024-03-18 20:23:26] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.28.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
89%|████████████████████████████████████████████████████████████████████████████████▎ | 174/195 [02:09<00:12, 1.75it/s] 90%|█████████████████████████████████████████████████████████████████████████████████▏ | 176/195 [02:09<00:13, 1.37it/s] [2024-03-18 20:23:26] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.28.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
90%|█████████████████████████████████████████████████████████████████████████████████▏ | 176/195 [02:10<00:13, 1.37it/s] [2024-03-18 20:23:27] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.28.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
90%|█████████████████████████████████████████████████████████████████████████████████▏ | 176/195 [02:10<00:13, 1.37it/s] 91%|█████████████████████████████████████████████████████████████████████████████████▋ | 177/195 [02:10<00:13, 1.34it/s] [2024-03-18 20:23:27] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.29.input_layernorm.weight", shape: (4096,), dtype: float32
91%|█████████████████████████████████████████████████████████████████████████████████▋ | 177/195 [02:10<00:13, 1.34it/s] [2024-03-18 20:23:27] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.29.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
91%|█████████████████████████████████████████████████████████████████████████████████▋ | 177/195 [02:11<00:13, 1.34it/s] [2024-03-18 20:23:28] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.29.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
91%|█████████████████████████████████████████████████████████████████████████████████▋ | 177/195 [02:11<00:13, 1.34it/s] 92%|██████████████████████████████████████████████████████████████████████████████████▌ | 179/195 [02:11<00:10, 1.56it/s] [2024-03-18 20:23:28] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.29.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
92%|██████████████████████████████████████████████████████████████████████████████████▌ | 179/195 [02:11<00:10, 1.56it/s] [2024-03-18 20:23:28] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.29.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
92%|██████████████████████████████████████████████████████████████████████████████████▌ | 179/195 [02:12<00:10, 1.56it/s] 92%|███████████████████████████████████████████████████████████████████████████████████ | 180/195 [02:12<00:08, 1.76it/s] [2024-03-18 20:23:28] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.29.post_attention_layernorm.weight", shape: (4096,), dtype: float32
92%|███████████████████████████████████████████████████████████████████████████████████ | 180/195 [02:12<00:08, 1.76it/s] [2024-03-18 20:23:29] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.29.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
92%|███████████████████████████████████████████████████████████████████████████████████ | 180/195 [02:13<00:08, 1.76it/s] [2024-03-18 20:23:30] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.29.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
92%|███████████████████████████████████████████████████████████████████████████████████ | 180/195 [02:13<00:08, 1.76it/s] 93%|████████████████████████████████████████████████████████████████████████████████████ | 182/195 [02:13<00:09, 1.38it/s] [2024-03-18 20:23:30] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.29.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
93%|████████████████████████████████████████████████████████████████████████████████████ | 182/195 [02:14<00:09, 1.38it/s] [2024-03-18 20:23:31] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.29.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
93%|████████████████████████████████████████████████████████████████████████████████████ | 182/195 [02:14<00:09, 1.38it/s] 94%|████████████████████████████████████████████████████████████████████████████████████▍ | 183/195 [02:14<00:08, 1.34it/s] [2024-03-18 20:23:31] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.30.input_layernorm.weight", shape: (4096,), dtype: float32
94%|████████████████████████████████████████████████████████████████████████████████████▍ | 183/195 [02:14<00:08, 1.34it/s] [2024-03-18 20:23:31] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.30.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
94%|████████████████████████████████████████████████████████████████████████████████████▍ | 183/195 [02:15<00:08, 1.34it/s] [2024-03-18 20:23:32] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.30.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
94%|████████████████████████████████████████████████████████████████████████████████████▍ | 183/195 [02:15<00:08, 1.34it/s] 95%|█████████████████████████████████████████████████████████████████████████████████████▍ | 185/195 [02:15<00:06, 1.54it/s] [2024-03-18 20:23:32] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.30.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
95%|█████████████████████████████████████████████████████████████████████████████████████▍ | 185/195 [02:16<00:06, 1.54it/s] [2024-03-18 20:23:32] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.30.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
95%|█████████████████████████████████████████████████████████████████████████████████████▍ | 185/195 [02:16<00:06, 1.54it/s] 95%|█████████████████████████████████████████████████████████████████████████████████████▊ | 186/195 [02:16<00:05, 1.74it/s] [2024-03-18 20:23:32] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.30.post_attention_layernorm.weight", shape: (4096,), dtype: float32
95%|█████████████████████████████████████████████████████████████████████████████████████▊ | 186/195 [02:16<00:05, 1.74it/s] [2024-03-18 20:23:34] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.30.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
95%|█████████████████████████████████████████████████████████████████████████████████████▊ | 186/195 [02:17<00:05, 1.74it/s] [2024-03-18 20:23:34] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.30.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
95%|█████████████████████████████████████████████████████████████████████████████████████▊ | 186/195 [02:18<00:05, 1.74it/s] 96%|██████████████████████████████████████████████████████████████████████████████████████▊ | 188/195 [02:18<00:05, 1.29it/s] [2024-03-18 20:23:35] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.30.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
96%|██████████████████████████████████████████████████████████████████████████████████████▊ | 188/195 [02:18<00:05, 1.29it/s] [2024-03-18 20:23:35] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.30.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
96%|██████████████████████████████████████████████████████████████████████████████████████▊ | 188/195 [02:19<00:05, 1.29it/s] 97%|███████████████████████████████████████████████████████████████████████████████████████▏ | 189/195 [02:19<00:04, 1.26it/s] [2024-03-18 20:23:35] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.31.input_layernorm.weight", shape: (4096,), dtype: float32
97%|███████████████████████████████████████████████████████████████████████████████████████▏ | 189/195 [02:19<00:04, 1.26it/s] [2024-03-18 20:23:36] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.31.self_attn.qkv_proj.q_weight", shape: (12288, 1024), dtype: uint32
97%|███████████████████████████████████████████████████████████████████████████████████████▏ | 189/195 [02:19<00:04, 1.26it/s] [2024-03-18 20:23:36] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.31.self_attn.qkv_proj.q_scale", shape: (12288, 128), dtype: float32
97%|███████████████████████████████████████████████████████████████████████████████████████▏ | 189/195 [02:20<00:04, 1.26it/s] 98%|████████████████████████████████████████████████████████████████████████████████████████▏ | 191/195 [02:20<00:02, 1.44it/s] [2024-03-18 20:23:36] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.31.self_attn.o_proj.q_weight", shape: (4096, 1024), dtype: uint32
98%|████████████████████████████████████████████████████████████████████████████████████████▏ | 191/195 [02:20<00:02, 1.44it/s] [2024-03-18 20:23:36] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.31.self_attn.o_proj.q_scale", shape: (4096, 128), dtype: float32
98%|████████████████████████████████████████████████████████████████████████████████████████▏ | 191/195 [02:20<00:02, 1.44it/s] 98%|████████████████████████████████████████████████████████████████████████████████████████▌ | 192/195 [02:20<00:01, 1.61it/s] [2024-03-18 20:23:36] INFO huggingface_loader.py:172: [Not quantized] Parameter: "model.layers.31.post_attention_layernorm.weight", shape: (4096,), dtype: float32
98%|████████████████████████████████████████████████████████████████████████████████████████▌ | 192/195 [02:20<00:01, 1.61it/s] [2024-03-18 20:23:38] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.31.mlp.gate_up_proj.q_weight", shape: (22016, 1024), dtype: uint32
98%|████████████████████████████████████████████████████████████████████████████████████████▌ | 192/195 [02:21<00:01, 1.61it/s] [2024-03-18 20:23:38] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.31.mlp.gate_up_proj.q_scale", shape: (22016, 128), dtype: float32
98%|████████████████████████████████████████████████████████████████████████████████████████▌ | 192/195 [02:22<00:01, 1.61it/s] 99%|█████████████████████████████████████████████████████████████████████████████████████████▌| 194/195 [02:22<00:00, 1.33it/s] [2024-03-18 20:23:39] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.31.mlp.down_proj.q_weight", shape: (4096, 2752), dtype: uint32
99%|█████████████████████████████████████████████████████████████████████████████████████████▌| 194/195 [02:23<00:00, 1.33it/s] [2024-03-18 20:23:39] INFO huggingface_loader.py:164: [Quantized] Parameter: "model.layers.31.mlp.down_proj.q_scale", shape: (4096, 344), dtype: float32
99%|█████████████████████████████████████████████████████████████████████████████████████████▌| 194/195 [02:23<00:00, 1.33it/s] 100%|██████████████████████████████████████████████████████████████████████████████████████████| 195/195 [02:23<00:00, 1.30it/s] 100%|██████████████████████████████████████████████████████████████████████████████████████████| 195/195 [02:23<00:00, 1.36it/s]
[2024-03-18 20:23:39] INFO huggingface_loader.py:194: Unloading HF weight file: ../dist/models/XAgentLlama-7B-preview/pytorch_model.bin
[2024-03-18 20:23:40] INFO stats.py:76: Time usage: HF loading: 29.093 sec; Pre-quantization mapping: 29.604 sec; Quantization: 3.236 sec
[2024-03-18 20:23:40] INFO stats.py:90: RAM usage: Peak RAM: 12.552 GB. Total bytes loaded from disk: 12.552 GB
[2024-03-18 20:23:40] INFO convert_weight.py:156: Parameter size after quantization: 7.061 GB
[2024-03-18 20:23:40] INFO convert_weight.py:161: Total parameters: 6,738,546,688
[2024-03-18 20:23:40] INFO convert_weight.py:162: Bits per parameter: 9.001
[2024-03-18 20:23:40] INFO convert_weight.py:167: Saved to directory: /tmp/tmpyuo2_jf8
All finished, 132 total shards committed, record saved to /tmp/tmpyuo2_jf8/ndarray-cache.json
Also saved a bf16 record to /tmp/tmpyuo2_jf8/ndarray-cache-b16.json