Update README.md

README.md

@@ -1,13 +1,51 @@
 ---
 base_model:
-- davidkim205/komt-solar-10.7b-sft-v5
-- colable/LDCC-CCK-slerp
-library_name: transformers
+- freewheelin/free-solar-slerp-v0.3
 tags:
 - mergekit
 - merge
 license: mit
+inference: false
+language:
+- ko
 ---
+# free-solar-0.3-exl2
+Original model: [free-solar-slerp-v0.3](https://huggingface.co/freewheelin/free-solar-slerp-v0.3)  
+Model creator: [freewheelin](https://huggingface.co/freewheelin/)
+
+# Quants
+[4bpw h6 (main)](https://huggingface.co/cgus/free-solar-slerp-v0.3/tree/main)  
+[4.25bpw h6](https://huggingface.co/cgus/free-solar-slerp-v0.3/tree/4.25bpw-h6)  
+[4.65bpw h6](https://huggingface.co/cgus/free-solar-slerp-v0.3/tree/4.65bpw-h6)  
+[5bpw h6](https://huggingface.co/cgus/free-solar-slerp-v0.3/tree/5bpw-h6)  
+[6bpw h6](https://huggingface.co/cgus/free-solar-slerp-v0.3/tree/6bpw-h6)  
+[8bpw h8](https://huggingface.co/cgus/free-solar-slerp-v0.3/tree/8bpw-h8)  
+
+# Quantization notes
+Made with Exllamav2 0.0.15 with the default dataset.  
+This model has unusually long loading times, normal 11B models take about 30s on my PC but this one loads in 130s.  
+I have no clue why it has such long loading times, maybe because it was originally FP32 instead of usual FP16.  
+But overall VRAM usage and generation speed seems to be rather normal.  
+  
+This seems to be primarily a Korean language model.  
+I didn't realize it at first when I tried it since the language wasn't explictly listed.  
+I'm unable to evaluate it in its main area but it seems to be usable in English and to some degree in other languages.  
+When using it in English, sometimes it seems to randomly switch topics or starts writing in Korean as if it occasionally forgets to write the stopping token.  
+But it has an interesting writing style in English and overall seems to be quite reasonable, so I dedided to make full quants.  
+And I was curious to try to quantize a FP32 model, RAM requirements were higher but overall process went smooth without any issues.  
+
+## How to run
+
+This quantization method uses GPU and requires Exllamav2 loader which can be found in following applications:
+
+[Text Generation Webui](https://github.com/oobabooga/text-generation-webui)  
+
+[KoboldAI](https://github.com/henk717/KoboldAI)  
+
+[ExUI](https://github.com/turboderp/exui)  
+
+
+# Original model card
 # free-solar-0.3
 
 This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
@@ -21,4 +59,4 @@ This model was merged using the SLERP merge method.
 
 The following models were included in the merge:
 * [davidkim205/komt-solar-10.7b-sft-v5](https://huggingface.co/davidkim205/komt-solar-10.7b-sft-v5)
-* [colable/LDCC-CCK-slerp](https://huggingface.co/davidkim205/colable/LDCC-CCK-slerp)
+* [colable/LDCC-CCK-slerp](https://huggingface.co/davidkim205/colable/LDCC-CCK-slerp)