inference/README.md

DeepSeek V3.1

First convert huggingface model weight files to the format of this project.

export EXPERTS=256
python convert.py --hf-ckpt-path ${HF_CKPT_PATH} --save-path ${SAVE_PATH} --n-experts ${EXPERTS} --model-parallel ${MP}

Then chat with DeepSeek model at will!

export CONFIG=config_671B_v3.1.json
torchrun --nproc-per-node ${MP} generate.py --ckpt-path ${SAVE_PATH} --config ${CONFIG} --interactive