Llama-3-8B-Instruct-ov-fp16-int4-sym
Built with Meta Llama 3
Model Description
This model is an OpenVINO IR FP16-INT4-ASYM optimized version of the original Meta-Llama-3-8B-Instruct, tailored for efficient inference while maintaining high performance. This version is optimized using OpenVINO, enhancing deployment on Intel hardware architectures. The model was optimized by following guidlines from OpenVINO Notebooks
Intended Use
This model is designed for advanced natural language understanding and generation tasks, ideal for academic researchers and developers in commercial settings looking to integrate efficient AI capabilities into their applications. It is not to be used for creating or promoting harmful or illegal content as per the guidelines outlined in the Meta Llama 3 Acceptable Use Policy.
Model Architecture
Like the original model, Llama-3-8B-Instruct-ov-fp16-int4-sym is based on an auto-regressive transformer architecture, fine-tuned with a focus on instruction-based tasks. The int8 quantization ensures it runs efficiently on compatible hardware without significant loss in performance.
Carbon Footprint and Sustainability
Our model training processes are committed to sustainability. The original training utilized Meta’s Research SuperCluster, significantly offsetting carbon emissions to ensure environmentally responsible AI development.
Licensing and Redistribution
This model is released under the Meta Llama 3 Community License. Redistribution requires inclusion of this license and a citation to the original model. Modifications and derivative works must prominently display "Built with Meta Llama 3" and adhere to the redistribution policies detailed in the original license terms.
How to Use
This model is optimized for inference using Intel's Optimum library with OpenVINO 2024.1, which enables enhanced performance on Intel hardware. Below are the steps to set up and run the model using Optimum and OpenVINO 2024.1 :
from optimum.intel.openvino import OVModelForCausalLM
from transformers import AutoTokenizer
model_id = "nsbendre25/Llama-3-8B-Instruct-ov_fp16-int4_sym"
# Initialize the tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = OVModelForCausalLM.from_pretrained(model_id)
pipeline = transformers.pipeline("text-generation", model=model_id, model_kwargs={"torch_dtype": torch.bfloat16}, device_map="auto")
pipeline("Hey how are you doing today?")