| 4-Bit Quant of: https://huggingface.co/Qwen/Qwen2-Audio-7B-Instruct | |
| Since there exists no other quant of this I guess I upload it here. | |
| Worked fine with english and japanese in my tests. | |
| --- | |
| license: apache-2.0 | |
| language: | |
| - en | |
| tags: | |
| - chat | |
| - audio | |
| --- | |
| # Qwen2-Audio-7B-Instruct | |
| ## Introduction | |
| Qwen2-Audio is the new series of Qwen large audio-language models. Qwen2-Audio is capable of accepting various audio signal inputs and performing audio analysis or direct textual responses with regard to speech instructions. We introduce two distinct audio interaction modes: | |
| * voice chat: users can freely engage in voice interactions with Qwen2-Audio without text input; | |
| * audio analysis: users could provide audio and text instructions for analysis during the interaction; | |
| We release Qwen2-Audio-7B and Qwen2-Audio-7B-Instruct, which are pretrained model and chat model respectively. | |
| For more details, please refer to our [Blog](https://qwenlm.github.io/blog/qwen2-audio/), [GitHub](https://github.com/QwenLM/Qwen2-Audio), and [Report](https://www.arxiv.org/abs/2407.10759). | |
| <br> | |
| ## Requirements | |
| The code of Qwen2-Audio has been in the latest Hugging face transformers and we advise you to build from source with command `pip install git+https://github.com/huggingface/transformers`, or you might encounter the following error: | |
| ``` | |
| KeyError: 'qwen2-audio' | |
| ``` | |
| ## Quickstart | |
| In the following, we demonstrate how to use `Qwen2-Audio-7B-Instruct` for the inference, supporting both voice chat and audio analysis modes. Note that we have used the ChatML format for dialog, in this demo we show how to leverage `apply_chat_template` for this purpose. | |
| ### Voice Chat Inference | |
| In the voice chat mode, users can freely engage in voice interactions with Qwen2-Audio without text input: | |
| ```python | |
| from io import BytesIO | |
| from urllib.request import urlopen | |
| import librosa | |
| from transformers import Qwen2AudioForConditionalGeneration, AutoProcessor | |
| processor = AutoProcessor.from_pretrained("Qwen/Qwen2-Audio-7B-Instruct") | |
| model = Qwen2AudioForConditionalGeneration.from_pretrained("Qwen/Qwen2-Audio-7B-Instruct", device_map="auto") | |
| conversation = [ | |
| {"role": "user", "content": [ | |
| {"type": "audio", "audio_url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/guess_age_gender.wav"}, | |
| ]}, | |
| {"role": "assistant", "content": "Yes, the speaker is female and in her twenties."}, | |
| {"role": "user", "content": [ | |
| {"type": "audio", "audio_url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/translate_to_chinese.wav"}, | |
| ]}, | |
| ] | |
| text = processor.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False) | |
| audios = [] | |
| for message in conversation: | |
| if isinstance(message["content"], list): | |
| for ele in message["content"]: | |
| if ele["type"] == "audio": | |
| audios.append(librosa.load( | |
| BytesIO(urlopen(ele['audio_url']).read()), | |
| sr=processor.feature_extractor.sampling_rate)[0] | |
| ) | |
| inputs = processor(text=text, audios=audios, return_tensors="pt", padding=True) | |
| inputs.input_ids = inputs.input_ids.to("cuda") | |
| generate_ids = model.generate(**inputs, max_length=256) | |
| generate_ids = generate_ids[:, inputs.input_ids.size(1):] | |
| response = processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] | |
| ``` | |
| ### Audio Analysis Inference | |
| In the audio analysis, users could provide both audio and text instructions for analysis: | |
| ```python | |
| from io import BytesIO | |
| from urllib.request import urlopen | |
| import librosa | |
| from transformers import Qwen2AudioForConditionalGeneration, AutoProcessor | |
| processor = AutoProcessor.from_pretrained("Qwen/Qwen2-Audio-7B-Instruct") | |
| model = Qwen2AudioForConditionalGeneration.from_pretrained("Qwen/Qwen2-Audio-7B-Instruct", device_map="auto") | |
| conversation = [ | |
| {'role': 'system', 'content': 'You are a helpful assistant.'}, | |
| {"role": "user", "content": [ | |
| {"type": "audio", "audio_url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/glass-breaking-151256.mp3"}, | |
| {"type": "text", "text": "What's that sound?"}, | |
| ]}, | |
| {"role": "assistant", "content": "It is the sound of glass shattering."}, | |
| {"role": "user", "content": [ | |
| {"type": "text", "text": "What can you do when you hear that?"}, | |
| ]}, | |
| {"role": "assistant", "content": "Stay alert and cautious, and check if anyone is hurt or if there is any damage to property."}, | |
| {"role": "user", "content": [ | |
| {"type": "audio", "audio_url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/1272-128104-0000.flac"}, | |
| {"type": "text", "text": "What does the person say?"}, | |
| ]}, | |
| ] | |
| text = processor.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False) | |
| audios = [] | |
| for message in conversation: | |
| if isinstance(message["content"], list): | |
| for ele in message["content"]: | |
| if ele["type"] == "audio": | |
| audios.append( | |
| librosa.load( | |
| BytesIO(urlopen(ele['audio_url']).read()), | |
| sr=processor.feature_extractor.sampling_rate)[0] | |
| ) | |
| inputs = processor(text=text, audios=audios, return_tensors="pt", padding=True) | |
| inputs.input_ids = inputs.input_ids.to("cuda") | |
| generate_ids = model.generate(**inputs, max_length=256) | |
| generate_ids = generate_ids[:, inputs.input_ids.size(1):] | |
| response = processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] | |
| ``` | |
| ### Batch Inference | |
| We also support batch inference: | |
| ```python | |
| from io import BytesIO | |
| from urllib.request import urlopen | |
| import librosa | |
| from transformers import Qwen2AudioForConditionalGeneration, AutoProcessor | |
| processor = AutoProcessor.from_pretrained("Qwen/Qwen2-Audio-7B-Instruct") | |
| model = Qwen2AudioForConditionalGeneration.from_pretrained("Qwen/Qwen2-Audio-7B-Instruct", device_map="auto") | |
| conversation1 = [ | |
| {"role": "user", "content": [ | |
| {"type": "audio", "audio_url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/glass-breaking-151256.mp3"}, | |
| {"type": "text", "text": "What's that sound?"}, | |
| ]}, | |
| {"role": "assistant", "content": "It is the sound of glass shattering."}, | |
| {"role": "user", "content": [ | |
| {"type": "audio", "audio_url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/f2641_0_throatclearing.wav"}, | |
| {"type": "text", "text": "What can you hear?"}, | |
| ]} | |
| ] | |
| conversation2 = [ | |
| {"role": "user", "content": [ | |
| {"type": "audio", "audio_url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/1272-128104-0000.flac"}, | |
| {"type": "text", "text": "What does the person say?"}, | |
| ]}, | |
| ] | |
| conversations = [conversation1, conversation2] | |
| text = [processor.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False) for conversation in conversations] | |
| audios = [] | |
| for conversation in conversations: | |
| for message in conversation: | |
| if isinstance(message["content"], list): | |
| for ele in message["content"]: | |
| if ele["type"] == "audio": | |
| audios.append( | |
| librosa.load( | |
| BytesIO(urlopen(ele['audio_url']).read()), | |
| sr=processor.feature_extractor.sampling_rate)[0] | |
| ) | |
| inputs = processor(text=text, audios=audios, return_tensors="pt", padding=True) | |
| inputs['input_ids'] = inputs['input_ids'].to("cuda") | |
| inputs.input_ids = inputs.input_ids.to("cuda") | |
| generate_ids = model.generate(**inputs, max_length=256) | |
| generate_ids = generate_ids[:, inputs.input_ids.size(1):] | |
| response = processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False) | |
| ``` | |
| ## Citation | |
| If you find our work helpful, feel free to give us a cite. | |
| ```BibTeX | |
| @article{Qwen2-Audio, | |
| title={Qwen2-Audio Technical Report}, | |
| author={Chu, Yunfei and Xu, Jin and Yang, Qian and Wei, Haojie and Wei, Xipin and Guo, Zhifang and Leng, Yichong and Lv, Yuanjun and He, Jinzheng and Lin, Junyang and Zhou, Chang and Zhou, Jingren}, | |
| journal={arXiv preprint arXiv:2407.10759}, | |
| year={2024} | |
| } | |
| ``` | |
| ```BibTeX | |
| @article{Qwen-Audio, | |
| title={Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models}, | |
| author={Chu, Yunfei and Xu, Jin and Zhou, Xiaohuan and Yang, Qian and Zhang, Shiliang and Yan, Zhijie and Zhou, Chang and Zhou, Jingren}, | |
| journal={arXiv preprint arXiv:2311.07919}, | |
| year={2023} | |
| } | |
| ``` |