Update README.md

README.md

@@ -7,15 +7,15 @@ Funded by: Self-funded
 
 Shared by: Arko007
 
-Model type: Autoregressive language model for code (code assistant)
+Model type: Autoregressive language model for code (code assistant), representing the fourth finetuning iteration based on CodeLlama-7b-hf.
 
-Language(s) (NLP): English, with support for various programming languages including Python, JavaScript, Java, and C++.
+Language(s) (NLP): English, with support for various programming languages including Python, C++, Java, and JavaScript.
 
-License: MIT License
+License: Llama 2 Community License
 
-Finetuned from model: bigcode/starcoder
+Finetuned from model: codellama/CodeLlama-7b-hf
 
-Model Sources
+Model Sources [optional]
 Repository: https://huggingface.co/Arko007/my-awesome-code-assistant-v4 (A placeholder URL, as the repository is not public)
 
 Paper [optional]: N/A
@@ -34,6 +34,8 @@ Code Refactoring: Suggesting improvements or alternative ways to write code.
 
 Code Documentation: Generating docstrings and comments.
 
+Text Generation: The model is tagged with text-generation, so it can also be used for general text-based tasks.
+
 Downstream Use [optional]
 This model can be used as a backend for integrated development environments (IDEs), developer tools, and educational platforms that require code assistance capabilities.
 
@@ -51,30 +53,39 @@ Recommendations
 Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. All generated code must be treated as a starting point and thoroughly reviewed, tested, and audited for correctness and security.
 
 How to Get Started with the Model
-Use the code below to get started with the model using the transformers library.
+Use the code below to get started with the model using the transformers and peft libraries.
 
 from transformers import AutoModelForCausalLM, AutoTokenizer
+from peft import PeftModel
 import torch
 
-model_id = "Arko007/my-awesome-code-assistant-v4"
-tokenizer = AutoTokenizer.from_pretrained(model_id)
-model = AutoModelForCausalLM.from_pretrained(model_id)
+model_name = "codellama/CodeLlama-7b-hf"
+adapter_name = "Arko007/my-awesome-code-assistant-v4"
+
+# Load the base model and tokenizer
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+base_model = AutoModelForCausalLM.from_pretrained(model_name)
+
+# Load the PEFT adapter
+model = PeftModel.from_pretrained(base_model, adapter_name)
+
+prompt = "def factorial(n):"
+inputs = tokenizer(prompt, return_tensors="pt")
+outputs = model.generate(**inputs, max_new_tokens=50)
 
-prompt = "# Write a Python function to calculate the factorial of a number"
-inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
-outputs = model.generate(**inputs, max_new_tokens=100)
 print(tokenizer.decode(outputs[0], skip_special_tokens=True))
 
+
 Training Details
 Training Data
-This model was finetuned on a private dataset of curated open-source code snippets and documentation. The specific sources are not publicly disclosed, but it primarily consists of code from GitHub repositories with permissive licenses.
+The base model, CodeLlama-7b-hf, was trained on a large, near-deduplicated dataset of publicly available code with an 8% mix of natural language data. The finetuning for my-awesome-code-assistant-v4 was done on a private dataset of curated open-source code snippets and documentation.
 
 Training Procedure
-Preprocessing: The training data was tokenized using the StarCoder tokenizer. Code comments were preserved to aid in documentation and explanation tasks.
+Preprocessing: The training data was tokenized using the CodeLlama tokenizer.
 
 Training Hyperparameters:
 
-Training regime: Finetuning with a LoRA (Low-Rank Adaptation) approach.
+Training regime: Finetuning with a LoRA (Low-Rank Adaptation) approach, using the peft library.
 
 Learning Rate: 2
 times10 
@@ -90,7 +101,8 @@ Optimizer: AdamW
 Speeds, Sizes, Times [optional]
 Finetuning Time: Approximately 12 hours
 
-Model Size: 15.5 GB (full model), ~120 MB (LoRA adapter)
+Model Size: 15.5 GB (full base model), 
+approx 120 MB (LoRA adapter)
 
 Evaluation
 Testing Data, Factors & Metrics
@@ -112,10 +124,12 @@ Pass@1 (Python): 55.1%
 Readability: The generated code was generally readable and well-commented.
 
 Summary
-Model Examination
+Model Examination [optional]
 The model demonstrates strong performance in common code generation tasks, particularly for Python. It can produce functional and readable code snippets.
 
 Environmental Impact
+Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).
+
 Hardware Type: 1 x NVIDIA A100 GPU
 
 Hours used: 12 hours
@@ -124,11 +138,16 @@ Cloud Provider: Google Cloud
 
 Compute Region: us-central1
 
-Carbon Emitted: 1.05 kg CO2eq (estimated using the Machine Learning Impact calculator)
+Carbon Emitted: 1.05 kg CO2eq (estimated)
 
 Technical Specifications [optional]
 Model Architecture and Objective
-The model is a decoder-only transformer architecture. Its objective is to predict the next token in a sequence, conditioned on the preceding tokens. The finetuning process adapted the base model to excel at generating code.
+The base model is a decoder-only transformer architecture. Its objective is to predict the next token in a sequence, conditioned on the preceding tokens. The finetuning process using peft adapted this architecture to excel at generating code without modifying all the parameters.
+
+Compute Infrastructure
+Hardware: 1 x NVIDIA A100 GPU
+
+Software: PyTorch, Transformers, PEFT
 
 Citation [optional]
 BibTeX
@@ -140,9 +159,19 @@ BibTeX
   url = {https://huggingface.co/Arko007/my-awesome-code-assistant-v4}
 }
 
+@article{touvron2023codellama,
+  title = {Code Llama: Open Foundation Models for Code},
+  author = {Touvron, Hugo and Coucke, Alexandre and Fan, Lya and Gong, Jian and Gu, Xiaodong and He, Jing and Hu, Weidong and Jiang, Shu and Li, Nan and Liu, Han and Lu, Zhiming and Ma, Huafeng and Ma, Shu and Niu, Zili and Ping, Jia and Qin, Zili and Tang, Tao and Wang, Tong and Wang, Wenjie and Xia, Jian and Xie, Jie and Xu, Chenyang and Xu, Feng and Yao, Jie and Ye, Min and Yang, Shuai and Zhang, Jun and Zhang, Wei and Zhang, Xiongbing and Zhao, Yali and Zhou, Guang and Zhou, Huajun and Zou, Jun},
+  journal = {arXiv preprint arXiv:2308.12950},
+  year = {2023}
+}
+
+
 APA
 Arko007. (2024). my-awesome-code-assistant-v4. Hugging Face. Retrieved from https://huggingface.co/Arko007/my-awesome-code-assistant-v4
 
+Touvron, H., Coucke, A., Fan, L., Gong, J., Gu, X., He, J., ... & Zou, J. (2023). Code Llama: Open Foundation Models for Code. arXiv preprint arXiv:2308.12950.
+
 Model Card Authors [optional]
 Arko007