Update README.md

README.md

@@ -1,128 +1,99 @@
 ---
-license: cc-by-4.0
+license: apache-2.0
 language:
 - en
 base_model:
-- google/paligemma2-3b-pt-448
 - kyutai/moshika-pytorch-bf16
+- mistralai/Pixtral-12B-2409
+- mistral-community/pixtral-12b
 ---
+# Model Card for Moshika Vision 
 
-# Model Card for MoshiVis
+<!-- Provide a quick summary of what the model is/does. -->
 
 
 ## Model Details
 
 ### Model Description
 
-**MoshiVis** ([Project Page](https://kyutai.org/moshivis) | [arXiv](https://arxiv.org/abs/2503.15633)) is a perceptually augmented version of Moshi, giving it the ability to freely discuss images whilst maintaining its natural conversation style and low latency.
+<!-- Provide a longer summary of what this model is. -->
+MoshiVis is a perceptually augmented version of Moshi, giving it the ability to freely discuss images whilst maintaining its natural conversation style and low latency.
 To achieve this, Moshi has been extended with a visual backbone and a cross-attention mechanism to infuse the visual information into the language model.
-To train MoshiVis, we add a few parameters (~200M) on top of a frozen Moshi backbone (for the text/speech modeling aspect, ~7B params) 
-and a PaliGemma2 vision encoder (for the image encoding part, ~400M parameters).
-
-This model page contains the `Moshika` (female voice)  model weights for the `Pytorch` backend of the MoshiVis repo, in `bfloat16`.
-We provide the same model weights for other backends and quantization formats in the associated model collection. 
 
 - **Developed by:** Kyutai
 - **Model type:** Multimodal speech+vision+text foundation model
 - **Language(s) (NLP):** English
-- **License:** CC-BY-4.0
-- **Finetuned from model:** [Moshika](https://huggingface.co/kyutai/moshika-pytorch-bf16) and [PaliGemma2](https://huggingface.co/google/paligemma2-3b-pt-448)
+- **License:** Apache License 2.0
+- **Finetuned from model:** [Moshika](https://huggingface.co/kyutai/moshika-vis-pytorch-bf16) and [Pixtral](https://huggingface.co/mistral-community/pixtral-12b)
 
+### Model Sources [optional]
 
-### Model Sources 
+<!-- Provide the basic links for the model. -->
 
-- **Project Page** [kyutai.org/moshivis](https://kyutai.org/moshivis)
-- **Preprint** ([arXiv/abs/2503.15633](https://arxiv.org/abs/2503.15633))
-- **Repository:** [Github kyutai-labs/moshivis](https://github.com/kyutai-labs/moshivis)
-- **Demo:** [Talk to Moshi](http://vis.moshi.chat)
+- **Repository:** [Github kyutai-labs/moshivis](https://github.com/kyutai-labs/moshivis) <-- TODO: Update / check link
+- **Demo [optional]:** [moshi.chat](https://moshi.chat/)
 
 ## Uses
 
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
 ### Direct Use
 
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
 Similar to Moshi itself, MoshiVis can be used as a conversational agent for casual conversations, basic facts and advice (e.g. recipes, trivia), roleplay, etc. 
 In addition, MoshiVis is able to recognize and discuss images in a natural way, whilst still allowing for low-latency interactions.
 
 
-### Downstream Use 
-
-Since MoshiVis was designed to infuse visual signal in a frozen Moshi backbone with  only a few trainable parameters, 
- the model could be adapted to different downstream scenarios by further finetuning these parameters : 
-for instance adapting MoshiVis for a different off-the-shelf image encoder or different visual domains.
-
 ### Out-of-Scope Use
 
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
 The model is not intended to be used to impersonate other people or any malicious use of any kind. 
 This model is for research only and we do not recommend it for providing advices or to perform any professionnal duty.
 
-
 ## Bias, Risks, and Limitations
 
-MoshiVis has been designed to perceptually augment the original [Moshi]((https://huggingface.co/kyutai/moshika-pytorch-bf16)) 
-model with vision capabilities and is expected to inherit similar biases and limitations. 
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+MoshiVis has been designed to perceptually augment the original Moshi model with vision capabilities and is expected to inherit similar biases and limitations, see also [Moshika](https://huggingface.co/kyutai/moshika-vis-pytorch-bf16).
+Our analysis with respect to how much MoshiVis diverges from the original model is still ongoing.
 
 
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
 ## How to Get Started with the Model
 
-See our [github repository](https://github.com/kyutai-labs/moshivis) for getting started.
-
+See the [README file](https://github.com/kyutai-labs/moshivis) for getting started.  <-- TODO: Update / check link
 
 ## Training Details
 
-Stay tuned for our technical report, in which we will describe the training procedure in detail as well as report evaluation results.
-
-### Training Data
-
-For information on the training data used for the base models, see [Pixtral](https://mistral.ai/news/pixtral-12b/) and 
-[Moshi](https://huggingface.co/kyutai/moshika-pytorch-bf16) respectively.
-To train the cross-attention and gating mechanism that MoshiVis uses for processing images, 
-we rely on a collection of publicly available datasets, namely:
-- [DOCCI](https://google.github.io/docci/)   
-- [PixMo](https://huggingface.co/datasets/allenai/pixmo-cap)   
-- [Pixelprose](https://arxiv.org/abs/2406.10328)   
-- [TallyQA](https://arxiv.org/abs/1810.12440)   
-- [OCR-VQA](https://ocr-vqa.github.io/)   
-- [RenderedText](https://huggingface.co/datasets/wendlerc/RenderedText)   
-- [DocVQA](https://arxiv.org/abs/2007.00398)    
-
-
-
-## Technical Specifications 
+### Model Architecture and Objective
+Our goal was to design an efficient and effective adaptation mechanism that allows Moshi to discuss images whilst maintaining its previous conversational capabilities.
+To achieve this, we train a cross-attention mechanism to insert image information from a pretrained and frozen vision backbone into the underlying language model, which is also kept frozen.
+An additional gating mechanism ensures that the insertion of visual information does not impact the interaction with Moshi outside of discussions of images, allowing for a seamless back and forth between general and image-specific conversations.
 
 
-### Compute Infrastructure
-
-MoshiVis was designed as a relatively low-cost adaptation of Moshi (~200M extra trainable parameters)
-and was trained on a single DGX node with 8 H100 GPUs.
-
+### Training Procedure
 
-#### Software
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+Stay tuned for our technical report, in which we will describe the training procedure in detail!
 
-Our training code was implemented in Pytorch. Our inference code is available for Pytorch, Rust and MLX.
+### Training Data
 
-## Citation
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+For information on the training data used for the base models, see [Pixtral](https://mistral.ai/news/pixtral-12b/) and [Moshi](https://huggingface.co/kyutai/moshika-pytorch-bf16) respectively.
+To train the cross-attention and gating mechanism that MoshiVis uses for processing images, we rely on a collection of publicly available datasets:
+- [Pixelprose](https://arxiv.org/abs/2406.10328)
+- [DOCCI](https://arxiv.org/abs/2404.19753)
+- [TallyQA](https://arxiv.org/abs/1810.12440)
+- [OCRVQA](https://ocr-vqa.github.io/)
+- [RenderedText](https://huggingface.co/datasets/wendlerc/RenderedText)
+- [DocVQA](https://arxiv.org/abs/2007.00398)
+- [ChartQA](https://aclanthology.org/2022.findings-acl.177/)
 
-```
-@article{kyutai2025moshivis,
-  author = {Amélie Royer and Moritz Böhle and Gabriel de Marmiesse and
-  Laurent Mazaré and Alexandre Défossez and Neil Zeghidour and Patrick Pérez},
-  year = {2025},
-  title = {Vision-Speech Models: Teaching Speech Models to Converse about Images},
-  journal = {ArXiv},
-  url = {https://arxiv.org/abs/2503.15633}
-}
-```
+We will share additional details soon, stay tuned!
 
 
+### Compute Infrastructure
 
+MoshiVis was designed as a relatively low-cost adaptation of Moshi and was trained on a single DGX node with 8 H100 GPUs provided by Scaleway.
 
-## Model Card Authors and Contact
+## Model Card Authors
 
-  * Amelie Royer
-  * Moritz Boehle
+Amélie Royer, Moritz Böhle

model.safetensors

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:73d2e92ff89b99c200d8c6e625d3649022481da2fb9d10fd85b5ae12fcc6226b
-size 17445080792

tokenizer-e351c8d8-checkpoint125.safetensors

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:09b782f0629851a271227fb9d36db65c041790365f11bbe5d3d59369cf863f50
-size 384644900

tokenizer_spm_32k_3.model

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:78d4336533ddc26f9acf7250d7fb83492152196c6ea4212c841df76933f18d2d
-size 552778