8bit quantization error
#98
by
lovelyfrog
- opened
model = AutoModelForCausalLM.from_pretrained(DEFAULT_CKPT_PATH, device_map="auto", load_in_8bit=True, max_memory=max_memory_mapping)
when the prompt is only test, it will cause the errror:
File "/home/devbrain/miniconda3/lib/python3.11/site-packages/transformers/generation/utils.py", line 2897, in sample
next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: probability tensor contains either inf, nan or element < 0
Hi, I encountered the same issue, may I know did you manage to find the solution for this error?