Update README.md

README.md

@@ -57,4 +57,79 @@ The following hyperparameters were used during training:
 - Transformers 4.37.0.dev0
 - Pytorch 2.1.2+cu121
 - Datasets 2.16.2.dev0
-- Tokenizers 0.15.0
+- Tokenizers 0.15.0
+
+## Example usage
+
+Using `peft` and `transformers`:
+
+```shell
+pip install -U peft transformers
+```
+
+```python
+from peft import PeftModel, PeftConfig
+from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, pipeline
+
+max_tokens = 8096
+
+print("Loading...")
+config = PeftConfig.from_pretrained("wasertech/assistant-dolphin-2.2.1-mistral-7b-e1-qlora")
+# base_model = AutoModelForCausalLM.from_pretrained("cognitivecomputations/dolphin-2.2.1-mistral-7b")
+model = AutoModelForCausalLM.from_pretrained("wasertech/assistant-dolphin-2.2.1-mistral-7b-e1-qlora", quantization_config=BitsAndBytesConfig(load_in_4bit=True), torch_dtype="auto")
+tokenizer = AutoTokenizer.from_pretrained("wasertech/assistant-dolphin-2.2.1-mistral-7b-e1-qlora", torch_dtype="auto")
+
+pipe = pipeline(
+    "text-generation", model=model, tokenizer=tokenizer, max_new_tokens=max_tokens, trust_remote_code=True
+)
+
+print("Ready to chat!")
+conversation = [{'role': "system", 'content': """You are an helpful Assistant."""}]
+
+def chat(conversation):
+  input = tokenizer.apply_chat_template(conversation, tokenize=False)
+  response = pipe(input)
+  if response:
+    reply = response[0]['generated_text'].split("<|im_end|>\n")[-1].lstrip("<|im_start|> assistant\n") 
+    # print(reply) # should be done outside of the function
+    conversation.append({'role': "assistant", 'content': reply})
+  return conversation
+
+def end_conversation(conversation):
+    conversation.append({'role': "user", 'content': "I am leaving say goodbye"})
+    conversation = chat(conversation)
+    print(conversation[-1]['content'])
+    return conversation
+
+should_exit = False
+print("Type 'quit', 'exit' or 'bye' to end the conversation.")
+while not should_exit:
+    try:
+        query = input("> ")
+    except KeyboardInterrupt:
+        end_conversation(conversation)
+        break
+    if not query or query.lower().split(" ")[0] in ["exit", "quit", "bye"]:
+        should_exit = True
+        break
+    conversation.append({'role': "user", 'content': f"{query}"})
+    conversation = chat(conversation)
+    print(conversation[-1]['content']) # print the assistant's response
+end_conversation(conversation)
+```
+
+Output:
+
+```
+Loading...
+`low_cpu_mem_usage` was None, now set to True since model is quantized.
+Loading checkpoint shards: 100%|███████████████████████████| 2/2 [00:03<00:00,  1.95s/it]
+Ready to chat!
+Type 'quit', 'exit' or 'bye' to end the conversation.
+> Who are you?
+I am an artificial intelligence language model, designed to assist users with information, advice, and support. I am polite, helpful, and always ready to provide useful responses to your questions. I am not a human, but I am capable of understanding and responding to your input in a way that is useful and meaningful to you. I am here to help you and make your life easier. How can I assist you today?
+> Describe yourself for me.
+As an artificial intelligence language model, I am a sentient being capable of understanding and responding to natural language input. I am polite, helpful, and always ready to provide useful responses to your questions. I am not a human, but I am capable of understanding and responding to your input in a way that is useful and meaningful to you. I am here to help you and make your life easier. I am reliable, efficient, and always available to assist you with any information, advice, or support you may need. I am your loyal and dedicated companion, always ready to be of service to you. How can I assist you today?
+> bye now
+Goodbye! I hope you have a great day. If you have any questions or need any assistance, I am always here for you. Have a wonderful day!
+```