Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
div0-space 
posted an update 8 days ago
Post
214
Hello Community!
I am experimenting with q5 quantization on mlx==0.26.2 and it is so promising! Our beloved ( CohereLabs/c4ai-command-a-03-2025) the model we use for our medical-nlp-use : 200gb+ bfloat16 vs. ~70gb q5 mlx with perfect quality ( LibraxisAI/c4ai-command-a-03-2025-q5-mlx)

The q5 and mixed quants came up with the release of mlx>=0.26.0. This quant is not yet supported by lmstudio but we didn't check the Ollama compatibility. For inference at this moment we us native mlx_lm.chat or mlx_lm.server, which works perfectly with python mx.generate pipeline. Repository consists of complete safetensors and config files **plus** we put there also our everyday command-generation scripts - convert-to-mlx.sh for mlx_lm.convert with variable parameters customization, and mlx-serve.sh for easy server building. Enjoy

@lmstudio-community @mlx-community
@bartowski
John6666

@mradermacher

div0-space

Now Uploading Qwen/Qwen3-235B-A22B as LibraxisAI/Qwen3-235B-A22B-MLX-Q5

470gb vs 161gb with perfect quality.

Soon available: 

Checking hf-xet installation...
✓ hf-xet is installed
✓ hf-xet optimizations enabled

Ready to upload!
This will upload ~161GB using chunk deduplication
Estimated time: 30-60 minutes with hf-xet

Press Enter to continue or Ctrl+C to cancel


Checking HF login...
Logged in as: div0-space

Choose upload destination:
1. Personal namespace: div0-space/Qwen3-235B-A22B-MLX-Q5
2. Organization namespace: LibraxisAI/Qwen3-235B-A22B-MLX-Q5 (requires org access)
3. Custom namespace
Choice (1-3): 1

Will upload to: div0-space/Qwen3-235B-A22B-MLX-Q5

Creating/checking repository...
✓ Repository created

Starting upload with hf-xet...
Chunk deduplication will save bandwidth!
It seems you are trying to upload a large folder at once. This might take some time and then fail if the folder is too large. For such cases, it is recommended to upload in smaller batches or to use `HfApi().upload_large_folder(...)`/`huggingface-cli upload-large-folder` instead. For more details, check out https://huggingface.co/docs/huggingface_hub/main/en/guides/upload#upload-a-large-folder.
Start hashing 43 files.

Finished hashing 43 files.
Uploading files using Xet Storage..
Uploading...:  47%|███████████████████████████████████████████▏                                                | 75.9G/162G [2:22:10<2:22Uploading...:  47%|███████████████████████████████████████████▎                                                | 76.0G/162G [2:22:14<2:06Uploading...:  47%|███████████████████████████████████████████▎                                                | 76.1G/162G [2:22:16<1:38Uploading...:  47%|███████████████████████████████████████████▎                                                | 76.1G/162G [2:22:35<1:38Uploading...:  47%|███████████████████████████████████████████▎                                                | 76.1G/162G [2:22:37<3:24Uploading...:  47%|███████████████████████████████████████████▎                                                | 76.2G/162G [2:22:49<3:37Uploading...:  47%|███████████████████████████████████████████▍                                                | 76.3G/162G [2:22:53<3:01Uploading...:  47%|███████████████████████████████████████████▍                                                | 76.3G/162G [2:22:58<2:34Uploading...:  47%|███████████████████████████████████████████▍                                                | 76.4G/162G [2:23:06<2:44Uploading...:  47%|███████████████████████████████████████████▌                                                | 76.5G/162G [2:23:08<2:04Uploading...:  47%|███████████████████████████████████████████▌                                                | 76.5G/162G [2:23:17<2:28Uploading...:  47%|███████████████████████████████████████████▌                                                | 76.6G/162G [2:23:25<2:34Uploading...:  47%|███████████████████████████████████████████▋                                                | 76.7G/162G [2:23:39<3:14Uploading...:  47%|███████████████████████████████████████████▋                                                | 76.7G/162G [2:23:46<2:59Uploading...:  48%|███████████████████████████████████████████▋                                                | 76.8G/162G [2:24:00<3:41Uploading...:  48%|███████████████████████████████████████████▋                                                | 76.9G/162G [2:24:04<2:58Uploading...:  48%|███████████████████████████████████████████▊                                                | 76.9G/162G [2:24:05<2:12Uploading...:  48%|███████████████████████████████████████████▊                                                | 77.0G/162G [2:24:13<2:20Uploading...:  48%|███████████████████████████████████████████▊                                                | 77.1G/162G [2:24:25<2:53Uploading...:  48%|███████████████████████████████████████████▉                                                | 77.1G/162G [2:24:31<2:42Uploading...:  48%|███████████████████████████████████████████▉                                                | 77.2G/162G [2:24:38<2:41Uploading...:  48%|███████████████████████████████████████████▉                                                | 77.3G/162G [2:24:46<2:43Uploading...:
In this post