helenai/jais-family-13b-chat-ov-int4-sym

This is the inceptionai/jais-family-13b-chat model converted to OpenVINO with INT4 symmetric channel-wise weight compression.

Download the model

Install huggingface-hub

pip install huggingface-hub[cli]

Download the model

huggingface-cli download helenai/jais-family-13b-chat-ov-int4-sym --local-dir jais-family-13b-chat-ov-int4-sym

Run inference

The recommend way to run inference with this model is with OpenVINO GenAI. It is the only package needed for inference - no need to install Transformers or PyTorch.

Install OpenVINO GenAI nightly

pip install --pre --upgrade openvino-genai openvino openvino-tokenizers --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/pre-release

Download a chat sample script (curl -O works on Windows Command Prompt and most Linux terminals)

curl -O https://raw.githubusercontent.com/helena-intel/snippets/refs/heads/main/llm_chat/python/llm_chat.py

Run the chat script with the path to the model and the device as parameters. Change GPU to CPU to run on CPU. NPU is not yet supported for this model

python llm_chat.py jais-family-13b-chat-ov-int4-sym GPU

More information

Check out OpenVINO GenAI documentation and OpenVINO GenAI samples.

Model compression parameters

openvino_version         : 2025.2.0-18660-3ceeeb52d64

advanced_parameters      : {'statistics_path': None, 'awq_params': {'subset_size': 32, 'percent_to_apply': 0.002, 'alpha_min': 0.0, 'alpha_max': 1.0, 'steps': 100}, 'scale_estimation_params': {'subset_size': 64, 'initial_steps': 5, 'scale_steps': 5, 'weight_penalty': -1.0}, 'gptq_params': {'damp_percent': 0.1, 'block_size': 128, 'subset_size': 128}, 'lora_correction_params': {'adapter_rank': 8, 'num_iterations': 3, 'apply_regularization': True, 'subset_size': 128, 'use_int8_adapters': True}}
all_layers               : False
awq                      : False
backup_mode              : int8_asym
gptq                     : False
group_size               : -1
ignored_scope            : []
lora_correction          : False
mode                     : int4_sym
ratio                    : 1.0
scale_estimation         : False
sensitivity_metric       : weight_quantization_error

optimum_intel_version    : 1.22.0
optimum_version          : 1.24.0
pytorch_version          : 2.5.1+cpu
transformers_version     : 4.48.3