Usage indications
Hey,
Thanks for releasing the models. I'd like to test them, but they lacks documentation ! I have a few questions:
I'm concerned because the models seem to use LlamaForCausalLM but they should also have bidirectional attention right ? How is this handled ?
When loading the model I get:
lion = AutoModel.from_pretrained('hzeng/Lion-SP-1B-llama3-marco-mntp')
config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 883/883 [00:00<00:00, 7.88MB/s]
adapter_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 816/816 [00:00<00:00, 7.85MB/s]
adapter_model.bin: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 45.2M/45.2M [00:05<00:00, 7.59MB/s]
Loading adapter weights from hzeng/Lion-SP-1B-llama3-marco-mntp led to unexpected keys not found in the model: model.layers.0.self_attn.q_proj.lora_A.default.weight, model.layers.0.self_attn.q_proj.lora_B.default.weight, model.layers.0.self_attn.k_proj.lora_A.default.weight, model.layers.0.self_attn.k_proj.lora_B.default.weight, model.layers.0.self_attn.v_proj.lora_A.default.weight, model.layers.0.self_attn.v_proj.lora_B.default.weight, model.layers.0.self_attn.o_proj.lora_A.default.weight, model.layers.0.self_attn.o_proj.lora_B.default.weight, model.layers.0.mlp.gate_proj.lora_A.default.weight, model.layers.0.mlp.gate_proj.lora_B.default.weight, model.layers.0.mlp.up_proj.lora_A.default.weight, model.layers.0.mlp.up_proj.lora_B.default.weight, model.layers.0.mlp.down_proj.lora_A.default.weight, model.layers.0.mlp.down_proj.lora_B.default.weight, model.layers.1.self_attn.q_proj.lora_A.default.weight, model.layers.1.self_attn.q_proj.lora_B.default.weight, model.layers.1.self_attn.k_proj.lora_A.default.weight, model.layers.1.self_attn.k_proj.lora_B.default.weight, model.layers.1.self_attn.v_proj.lora_A.default.weight, model.layers.1.self_attn.v_proj.lora_B.default.weight, model.layers.1.self_attn.o_proj.lora_A.default.weight, model.layers.1.self_attn.o_proj.lora_B.default.weight,
...
COuld you indicate a fix for that (it maybe the transformers version, not sure)
- Can you provide a quick snippet to obtain a doc/query embedding and compute scores ?
Thanks in advance,
Hi Maxoul,
Sorry for the late reply.
For 1, please check the GitHub repo: https://github.com/HansiZeng/scaling-retriever, which contains the detailed instructions on how to load the sparse models.
For 2, could you please let me know what your transformer version is? My transformer version is 4.43.1.
For 3, you can also check the GitHub repo: https://github.com/HansiZeng/scaling-retriever. It contains the toy examples for computing scores.