deepseek-ai

/

DeepSeek-V3-0324

like 2.72k

Follow

DeepSeek 59k

(ktransformers) wrx80e@id2:/media/cookie/Designaire/_10_python_pkgs/_16_Chatbots/ktransformers$ ktransformers --model_path "deepseek-ai/DeepSeek-V3-0324" --gguf_path "/media/ cookie/NVMeArray/Downloads/Unsloth_DeepSeek-V3-0324-UD-IQ2_XXS/DeepSeek-V3-0324-UD-IQ2_XXS-00001-of-00004.gguf" --port 5002 --host 192.168.0.202 --total_context=32768 --cache _8bit True --temperature 0.3 --gpu_split "24 24 24 24 12" --max_new_tokens 32768
2025-03-27 04:16:56,082 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
found flashinfer
2025-03-27 04:16:56,239 INFO /home/wrx80e/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/main.py[29]: Creating SQL tables
2025-03-27 04:16:56,240 INFO /home/wrx80e/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating def ault assistant
Traceback (most recent call last):
  File "/home/wrx80e/anaconda3/envs/ktransformers/bin/ktransformers", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/wrx80e/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/main.py", line 115, in main
    create_interface(config=cfg, default_args=cfg)
  File "/home/wrx80e/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/utils/create_interface.py", line 27, in create_interface
    GlobalInterface.interface = BackendInterface(default_args)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wrx80e/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 48, in __init__
    self.model = custom_models[config.architectures[0]](config)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wrx80e/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 1605, in __init__
    self.model = DeepseekV3Model(config)
                 ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wrx80e/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 1367, in __init__
    [
  File "/home/wrx80e/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 1368, in <listcomp>
    DeepseekV3DecoderLayer(config, layer_idx)
  File "/home/wrx80e/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 1154, in __init__
    DeepseekV3MoE(config)
  File "/home/wrx80e/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 515, in __init__
    self.gate = MoEGate(config)
                ^^^^^^^^^^^^^^^
  File "/home/wrx80e/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 401, in __init__
    self.seq_aux = config.seq_aux
                   ^^^^^^^^^^^^^^
  File "/home/wrx80e/anaconda3/envs/ktransformers/lib/python3.11/site-packages/transformers/configuration_utils.py", line 264, in __getattribute__
    return super().__getattribute__(key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'DeepseekV3Config' object has no attribute 'seq_aux'
(ktransformers) wrx80e@id2:/media/cookie/Designaire/_10_python_pkgs/_16_Chatbots/ktransformers$

ktransformers now crashes after your latest commit.

Should I report this bug to ktransformers or transformers or here?