kyutai/moshika-vis-pytorch-bf16 v0.1

Co-authored-by: mboehle <[email protected]>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,117 @@
+---
+license: cc-by-4.0
+language:
+- en
+base_model:
+- google/paligemma2-3b-pt-448
+- kyutai/moshika-pytorch-bf16
+---
+
+# Model Card for MoshiVis
+
+
+## Model Details
+
+### Model Description
+
+MoshiVis is a perceptually augmented version of Moshi, giving it the ability to freely discuss images whilst maintaining its natural conversation style and low latency.
+To achieve this, Moshi has been extended with a visual backbone and a cross-attention mechanism to infuse the visual information into the language model.
+To train MoshiVis, we add a few parameters (~200M) on top of a frozen Moshi backbone (for the text/speech modeling aspect, ~7B params) 
+and a PaliGemma2 vision encoder (for the image encoding part, ~400M parameters).
+
+This model page contains the `Moshika` (female voice)  model weights for the `Pytorch` backend of the MoshiVis repo, in `bfloat16`.
+We provide the same model weights for other backends and quantization formats in the associated model collection. 
+
+- **Developed by:** Kyutai
+- **Model type:** Multimodal speech+vision+text foundation model
+- **Language(s) (NLP):** English
+- **License:** Apache License 2.0
+- **Finetuned from model:** [Moshika](https://huggingface.co/kyutai/moshika-pytorch-bf16) and [PaliGemma2](https://huggingface.co/google/paligemma2-3b-pt-448)
+
+
+### Model Sources 
+
+- **Repository:** [Github kyutai-labs/moshivis](https://github.com/kyutai-labs/moshivis)
+- **Demo:** [Talk to Moshi](http://vis.moshi.chat)
+
+## Uses
+
+### Direct Use
+
+Similar to Moshi itself, MoshiVis can be used as a conversational agent for casual conversations, basic facts and advice (e.g. recipes, trivia), roleplay, etc. 
+In addition, MoshiVis is able to recognize and discuss images in a natural way, whilst still allowing for low-latency interactions.
+
+
+### Downstream Use 
+
+Since MoshiVis was designed to infuse visual signal in a frozen Moshi backbone with  only a few trainable parameters, 
+ the model could be adapted to different downstream scenarios by further finetuning these parameters : 
+for instance adapting MoshiVis for a different off-the-shelf image encoder or different visual domains.
+
+### Out-of-Scope Use
+
+The model is not intended to be used to impersonate other people or any malicious use of any kind. 
+This model is for research only and we do not recommend it for providing advices or to perform any professionnal duty.
+
+
+## Bias, Risks, and Limitations
+
+MoshiVis has been designed to perceptually augment the original [Moshi]((https://huggingface.co/kyutai/moshika-pytorch-bf16)) 
+model with vision capabilities and is expected to inherit similar biases and limitations. 
+
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+See our [github repository](https://github.com/kyutai-labs/moshivis) for getting started.
+
+
+## Training Details
+
+Stay tuned for our technical report, in which we will describe the training procedure in detail as well as report evaluation results.
+
+### Training Data
+
+For information on the training data used for the base models, see [Pixtral](https://mistral.ai/news/pixtral-12b/) and 
+[Moshi](https://huggingface.co/kyutai/moshika-pytorch-bf16) respectively.
+To train the cross-attention and gating mechanism that MoshiVis uses for processing images, 
+we rely on a collection of publicly available datasets, namely:
+- [DOCCI](https://google.github.io/docci/)   
+- [PixMo](https://huggingface.co/datasets/allenai/pixmo-cap)   
+- [Pixelprose](https://arxiv.org/abs/2406.10328)   
+- [TallyQA](https://arxiv.org/abs/1810.12440)   
+- [OCR-VQA](https://ocr-vqa.github.io/)   
+- [RenderedText](https://huggingface.co/datasets/wendlerc/RenderedText)   
+- [DocVQA](https://arxiv.org/abs/2007.00398)    
+
+
+
+## Technical Specifications 
+
+
+### Compute Infrastructure
+
+MoshiVis was designed as a relatively low-cost adaptation of Moshi (~200M extra trainable parameters)
+and was trained on a single DGX node with 8 H100 GPUs.
+
+
+#### Software
+
+Our training code was implemented in Pytorch. Our inference code is available for Pytorch, Rust and MLX.
+
+## Citation
+
+
+Blog post: https://kyutai.org/
+
+
+
+## Model Card Authors and Contact
+
+  * Amelie Royer
+  * Moritz Boehle

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:73d2e92ff89b99c200d8c6e625d3649022481da2fb9d10fd85b5ae12fcc6226b
+size 17445080792

tokenizer-e351c8d8-checkpoint125.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:09b782f0629851a271227fb9d36db65c041790365f11bbe5d3d59369cf863f50
+size 384644900

tokenizer_spm_32k_3.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:78d4336533ddc26f9acf7250d7fb83492152196c6ea4212c841df76933f18d2d
+size 552778