Convert in bf16 or fp16?

by remember2015 - opened Jul 30

remember2015

Jul 30

I noticed https://github.com/ml-explore/mlx-lm/issues/193 which say : "For many models it's fine to use float16 so in your case you can convert the model using"

so I was wondering which dtype used to quant model

warshanks

MLX Community org Jul 30

I converted this with mlx-lm 0.24.1 so it should have been inferred from the original model, so bf16.

I'd have to double check but that should be the case!

remember2015

Jul 30

I converted this with mlx-lm 0.24.1 so it should have been inferred from the original model, so bf16.

I'd have to double check but that should be the case!

thank you for ur reply~

remember2015 changed discussion status to closed Jul 30

remember2015 changed discussion status to open Jul 30

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment