anikifoss

/

Kimi-K2-Instruct-DQ4_K

like 4

You're welcome, glad to hear it's useful!

Kimi just updated their chat template, so likely will need to re-upload. I'll wait for a couple of days so the the dust can settle, in case there are more changes.

Looks like the chat template is stored in the first GGUF part, so it may be possible to only re-upload the 00001 part with the new chat template. This should work, as long as llama-gguf-split keeps the same grouping of tensors for each part. I'll keep you posted when I make the changes.

@sousekd I updated Kimi-K2-Instruct-DQ4_K-00001-of-00014.gguf file with new metadata. You can re-download it, or run this patch locally:

cd llama.cpp/gguf-py
python -m venv venv
source venv/bin/activate
pip install -r ../requirements/requirements-gguf_editor_gui.txt
pip install --editable .

cd path-to-kimi-k2

gguf-new-metadata \
  --general-name 'Kimi K2 Instruct' \
  --special-token-by-id eos 163586 \
  --chat-template "{%- if tools -%}\n  <|im_system|>tool_declare<|im_middle|>{{ tools | tojson }}<|im_end|>\n{%- endif -%}\n{%- for message in messages -%}\n  {%- if loop.first and messages[0]['role'] != 'system' -%}\n    <|im_system|>system<|im_middle|>You are a helpful assistant<|im_end|>\n  {%- endif -%}\n  {%- if message['role'] == 'system' -%}\n    <|im_system|>system<|im_middle|>\n  {%- elif message['role'] == 'user' -%}\n    <|im_user|>user<|im_middle|>\n  {%- elif message['role'] == 'assistant' -%}\n    <|im_assistant|>assistant<|im_middle|>\n  {%- elif message['role'] == 'tool' -%}\n    <|im_system|>tool<|im_middle|>\n  {%- endif -%}\n  {%- if message['role'] == 'assistant' and message.get('tool_calls') -%}\n    {%- if message['content'] -%}{{ message['content'] }}{%- endif -%}\n    <|tool_calls_section_begin|>\n    {%- for tool_call in message['tool_calls'] -%}\n      {%- set func_name = tool_call['function']['name'] -%}\n      {%- set formatted_id = 'functions.' + func_name + ':' + loop.index0|string -%}\n      <|tool_call_begin|>{{ formatted_id }}<|tool_call_argument_begin|>{{ tool_call['function']['arguments'] | tojson}}<|tool_call_end|>\n    {%- endfor -%}\n    <|tool_calls_section_end|>\n  {%- elif message['role'] == 'tool' -%}\n    ## Return of {{ message.tool_call_id }}\n{{ message['content'] }}\n  {%- elif message['content'] is string -%}\n    {{ message['content'] }}\n  {%- elif message['content'] is not none -%}\n    {% for content in message['content'] -%}\n      {% if content['type'] == 'image' or 'image' in content or 'image_url' in content -%}\n        <|media_start|>image<|media_content|><|media_pad|><|media_end|>\n      {% else -%}\n        {{ content['text'] }}\n      {%- endif -%}\n    {%- endfor -%}\n  {%- endif -%}\n  <|im_end|>\n{%- endfor -%}\n{%- if add_generation_prompt -%}\n  <|im_assistant|>assistant<|im_middle|>\n{%- endif -%}" \
  Kimi-K2-Instruct-DQ4_K-00001-of-00014.gguf \
  Kimi-K2-Instruct-DQ4_K-00001-of-00014-patched.gguf

rm Kimi-K2-Instruct-DQ4_K-00001-of-00014.gguf
mv Kimi-K2-Instruct-DQ4_K-00001-of-00014-patched.gguf Kimi-K2-Instruct-DQ4_K-00001-of-00014.gguf

Another chat_template update, see here for more details.