Hola world.

LICENSE.md

@@ -0,0 +1,36 @@
+MICROSOFT RESEARCH LICENSE TERMS
+IF YOU LIVE IN THE UNITED STATES, PLEASE READ THE “BINDING ARBITRATION AND CLASS ACTION WAIVER” SECTION BELOW. IT AFFECTS HOW DISPUTES ARE RESOLVED.
+These license terms are an agreement between you and Microsoft Corporation (or one of its affiliates). They apply to the source code, object code, machine learning models, or data (collectively “Materials”) that accompany this license. IF YOU COMPLY WITH THESE LICENSE TERMS, YOU HAVE THE RIGHTS BELOW. BY USING THE MATERIALS, YOU ACCEPT THESE TERMS.
+1) INSTALLATION AND USE RIGHTS TO THE MATERIALS.
+Subject to the terms of this agreement, you have the below rights, if applicable, to use the Materials solely for non-commercial, non-revenue generating, research purposes:
+a) Source Code. If source code is included, you may use and modify the source code, but you may not distribute the source code. 
+b) Object Code. If object code is included, you may use the object code, but you may not distribute the object code.
+c) Models.  If machine learning model(s) are included, you may use the model(s), but you may not distribute the models.
+d) Data. If data is included, you may use and modify the data, but your use and modification must be consistent with the consent under which the data was provided and/or gathered and you may not distribute the data or your modifications to the data.
+2) SCOPE OF LICENSE. The Materials are licensed, not sold. Microsoft reserves all other rights. Unless applicable law gives you more rights despite this limitation, you will not (and have no right to):
+a) work around any technical limitations in the Materials that only allow you to use it in certain ways;
+b) reverse engineer, decompile or disassemble the Materials;
+c) remove, minimize, block, or modify any notices of Microsoft or its suppliers in the Materials;
+d) use the Materials in any way that is against the law or to create or propagate malware; or
+e) share, publish, distribute or lend the Materials, provide the Materials as a stand-alone hosted solution for others to use, or transfer the Materials or this agreement to any third party.
+3) PERSONAL DATA.  If the data (set forth in Section 1(c) above) includes or is found to include any data that enables any ability to identify an individual (“Personal Data”), you will not use such Personal Data for any purpose other than was authorized and consented to by the data subject/research participant.  You will not use Personal Data to contact any person.  You will keep Personal Data in strict confidence.  You will not share any Personal Data that is collected or in your possession with any third party for any reason and as required under the original consent agreement.  Further, you will destroy the Personal Data and any backup or copies, immediately upon the completion of your research.  
+4) LICENSE TO MICROSOFT.  Notwithstanding the limitations in Section 1, you may distribute your modifications back to Microsoft, and if you do provide Microsoft with modifications of the Materials, you hereby grant Microsoft, without any restrictions or limitations, a non-exclusive, perpetual, irrevocable, royalty-free, assignable and sub-licensable license, to reproduce, publicly perform or display, install, use, modify, post, distribute, make and have made, sell and transfer such modifications and derivatives for any purpose.
+5) PUBLICATION.  You may publish (or present papers or articles) on your results from using the Materials provided that no material or substantial portion of the Materials is included in any such publication or presentation.
+6) FEEDBACK. Any feedback about the Materials provided by you to us is voluntarily given, and Microsoft shall be free to use the feedback as it sees fit without obligation or restriction of any kind, even if the feedback is designated by you as confidential.  Such feedback shall be considered a contribution and licensed to Microsoft under the terms of Section 4 above.
+7) COMPLIANCE WITH TRADE LAWS. You acknowledge that the Materials may be subject to applicable trade laws in one or more countries.  You will comply with all relevant laws and regulations applicable to the import or export of the Materials, including but not limited to, trade laws such as the U.S. Export Administration Regulations or other end-user, end use, and destination restrictions by the U.S. and other governments, as well as sanctions regulations administered by the U.S. Office of Foreign Assets Control. Microsoft may suspend or terminate the agreement immediately to the extent that Microsoft reasonably concludes that continued performance would violate trade laws or put it at risk of becoming subject to sanctions or penalties under trade laws. For additional information, see www.microsoft.com/exporting.
+8) SUPPORT SERVICES. Microsoft is not obligated under this agreement to provide any support services for the Materials. Any support provided is “as is”, “with all faults”, and without warranty of any kind.
+9) BINDING ARBITRATION AND CLASS ACTION WAIVER. This Section applies if you live in (or, if a business, your principal place of business is in) the United States.  If you and Microsoft have a dispute, you and Microsoft agree to try for 60 days to resolve it informally. If you and Microsoft can’t, you and Microsoft agree to binding individual arbitration before the American Arbitration Association under the Federal Arbitration Act (“FAA”), and not to sue in court in front of a judge or jury. Instead, a neutral arbitrator will decide. Class action lawsuits, class-wide arbitrations, private attorney-general actions, and any other proceeding where someone acts in a representative capacity are not allowed; nor is combining individual proceedings without the consent of all parties. The complete Arbitration Agreement contains more terms and is at aka.ms/arb-agreement-1. You and Microsoft agree to these terms.
+10) ENTIRE AGREEMENT. This agreement, and any other terms Microsoft may provide for supplements, updates, or third-party applications, is the entire agreement for the Materials.
+11) APPLICABLE LAW AND PLACE TO RESOLVE DISPUTES. If you acquired the Materials in the United States or Canada, the laws of the state or province where you live (or, if a business, where your principal place of business is located) govern the interpretation of this agreement, claims for its breach, and all other claims (including consumer protection, unfair competition, and tort claims), regardless of conflict of laws principles, except that the FAA governs everything related to arbitration. If you acquired the Materials in any other country, its laws apply, except that the FAA governs everything related to arbitration. If U.S. federal jurisdiction exists, you and Microsoft consent to exclusive jurisdiction and venue in the federal court in King County, Washington for all disputes heard in court (excluding arbitration). If not, you and Microsoft consent to exclusive jurisdiction and venue in the Superior Court of King County, Washington for all disputes heard in court (excluding arbitration).
+12) CONSUMER RIGHTS; REGIONAL VARIATIONS. This agreement describes certain legal rights. You may have other rights, including consumer rights, under the laws of your state, province, or country. Separate and apart from your relationship with Microsoft, you may also have rights with respect to the party from which you acquired the Materials. This agreement does not change those other rights if the laws of your state, province, or country do not permit it to do so. For example, if you acquired the Materials in one of the below regions, or mandatory country law applies, then the following provisions apply to you:
+a) Australia. You have statutory guarantees under the Australian Consumer Law and nothing in this agreement is intended to affect those rights.
+b) Canada. If you acquired this software in Canada, you may stop receiving updates by turning off the automatic update feature, disconnecting your device from the Internet (if and when you re-connect to the Internet, however, the Materials will resume checking for and installing updates), or uninstalling the Materials. The product documentation, if any, may also specify how to turn off updates for your specific device or software.
+c) Germany and Austria.
+ i. Warranty. The properly licensed software will perform substantially as described in any Microsoft materials that accompany the Materials. However, Microsoft gives no contractual guarantee in relation to the licensed software.
+ ii. Limitation of Liability. In case of intentional conduct, gross negligence, claims based on the Product Liability Act, as well as, in case of death or personal or physical injury, Microsoft is liable according to the statutory law.
+Subject to the foregoing clause (ii), Microsoft will only be liable for slight negligence if Microsoft is in breach of such material contractual obligations, the fulfillment of which facilitate the due performance of this agreement, the breach of which would endanger the purpose of this agreement and the compliance with which a party may constantly trust in (so-called "cardinal obligations"). In other cases of slight negligence, Microsoft will not be liable for slight negligence.
+13) DISCLAIMER OF WARRANTY. THE MATERIALS ARE LICENSED “AS IS.” YOU BEAR THE RISK OF USING THEM. MICROSOFT GIVES NO EXPRESS WARRANTIES, GUARANTEES, OR CONDITIONS. TO THE EXTENT PERMITTED UNDER APPLICABLE LAWS, MICROSOFT EXCLUDES ALL IMPLIED WARRANTIES, INCLUDING MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT.
+
+14) LIMITATION ON AND EXCLUSION OF DAMAGES. IF YOU HAVE ANY BASIS FOR RECOVERING DAMAGES DESPITE THE PRECEDING DISCLAIMER OF WARRANTY, YOU CAN RECOVER FROM MICROSOFT AND ITS SUPPLIERS ONLY DIRECT DAMAGES UP TO U.S. $5.00. YOU CANNOT RECOVER ANY OTHER DAMAGES, INCLUDING CONSEQUENTIAL, LOST PROFITS, SPECIAL, INDIRECT OR INCIDENTAL DAMAGES.
+This limitation applies to (a) anything related to the Materials, services, content (including code) on third party Internet sites, or third party applications; and (b) claims for breach of contract, warranty, guarantee, or condition; strict liability, negligence, or other tort; or any other claim; in each case to the extent permitted by applicable law.
+It also applies even if Microsoft knew or should have known about the possibility of the damages. The above limitation or exclusion may not apply to you because your state, province, or country may not allow the exclusion or limitation of incidental, consequential, or other damages.

README.md

@@ -1,5 +1,251 @@
 ---
 license: other
-license_name: microsoft-research
-license_link: LICENSE
 ---
+
+# LLaVA-Rad
+
+This is the official model checkpoint for **LLaVA-Rad**, described in “A Clinically Accessible Small Multimodal Radiology Model and Evaluation Metric for Chest X-Ray
+Findings”.
+
+|  |  |
+|:---------|:--------|
+| **Developed by** | Microsoft Research |
+| **Model type** | [small multimodal transformer model](#model-architecture) |
+| **Languages** | English |
+| **License** | [![Model License](https://img.shields.io/badge/License-Microsoft%20Research-red)](LICENSE) |
+| **Data** | [![Data](https://img.shields.io/badge/Dataset-Annotations-228B22)](https://physionet.org/content/llava-rad-mimic-cxr-annotation/1.0.0/) |
+| **Code** | [![Code](https://img.shields.io/badge/GitHub-LLaVARad-FFA500)](https://github.com/microsoft/llava-rad) |
+| **Evaluation** | [![Eval](https://img.shields.io/badge/GitHub-CheXprompt-purple)](https://github.com/microsoft/chexprompt) |
+| **Preprint** | [![Preprint](https://img.shields.io/badge/Paper-arXiv-blue)](https://arxiv.org/abs/2403.08002) |
+| **Peer Reviewed Paper** | [![Paper](https://img.shields.io/badge/status-In%20Press-cyan)]() |
+
+
+LlaVA-Rad is a 7 billion parameter small multimodal model trained to produce findings given an input chest X-ray. Its architecture follows that of [LLaVA](https://arxiv.org/abs/2310.03744) and [LLaVA-Med](https://arxiv.org/abs/2306.00890), differing in the use of a specialized chest X-ray image encoder, BiomedCLIP-CXR, built with the [BiomedCLIP](https://arxiv.org/abs/2303.00915) framework. LLaVA-Rad offers outstanding performance at relatively small model size.
+
+
+<p align="center">
+    <img src="results_top.jpg" alt="Evaluation Overview" width="400" height="400">
+</p>
+
+## Contents
+- [Usage](#usage)
+  - [Installation](#installation)
+  - [Example inference](#example-inference)
+  - [Usage and License Notices](#usage-and-license-notices)
+  - [Ethical Considerations and Limitations](#ethical-considerations-and-limitations)
+- [LLaVA-Rad Basics](#llava-rad-basics)
+  - [Abstract](#abstract)
+  - [Model Architecture](#model-architecture)
+  - [Evaluation Results](#evaluation-results)
+  - [Citation](#citation)
+- [Further Information](#further-information)
+  - [Data Specification](#data-specification)
+  - [Related Models](#related-models)
+
+
+
+## Usage
+
+### Installation
+Follow these steps to set up LLaVA-Rad:
+
+1. Clone the repository and navigate to the project folder:
+   ```Shell
+   git clone https://github.com/microsoft/llava-rad.git
+   cd llava-rad
+   ```
+2. Create and activate a virtual environment (Python 3.10):
+   ```Shell
+   conda create -n llavarad python=3.10 -y
+   conda activate llavarad
+   ```
+3. Upgrade pip and install the package:
+   ```Shell
+   pip install --upgrade pip
+   pip install -e .
+   ```
+
+
+### Example inference
+
+```python
+import requests
+import torch
+from PIL import Image
+from io import BytesIO
+
+from llava.constants import IMAGE_TOKEN_INDEX
+from llava.conversation import conv_templates
+from llava.model.builder import load_pretrained_model
+from llava.utils import disable_torch_init
+from llava.mm_utils import tokenizer_image_token, KeywordsStoppingCriteria
+
+
+def load_image(image_file):
+    if image_file.startswith('http') or image_file.startswith('https'):
+        response = requests.get(image_file)
+        image = Image.open(BytesIO(response.content)).convert('RGB')
+    else:
+        image = Image.open(image_file).convert('RGB')
+    return image
+
+# Model
+disable_torch_init()
+
+model_path = "microsoft/llava-rad"
+model_base = "lmsys/vicuna-7b-v1.5"
+model_name = "llavarad"
+conv_mode = "v1"
+
+tokenizer, model, image_processor, context_len = load_pretrained_model(model_path, model_base, model_name)
+
+# Prepare query
+image_file = "https://openi.nlm.nih.gov/imgs/512/253/253/CXR253_IM-1045-1001.png" # CXR w pneumothorax from Open-I
+query = "<image>\nDescribe the findings of the chest x-ray.\n"
+
+conv = conv_templates[conv_mode].copy()
+conv.append_message(conv.roles[0], query)
+conv.append_message(conv.roles[1], None)
+prompt = conv.get_prompt()
+
+image = load_image(image_file)
+image_tensor = image_processor.preprocess(image, return_tensors='pt')['pixel_values'][0].half().unsqueeze(0).cuda()
+
+input_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors='pt').unsqueeze(0).cuda()
+
+stopping_criteria = KeywordsStoppingCriteria(["</s>"], tokenizer, input_ids)
+
+with torch.inference_mode():
+    output_ids = model.generate(
+        input_ids,
+        images=image_tensor,
+        do_sample=False,
+        temperature=0.0,
+        max_new_tokens=1024,
+        use_cache=True,
+        stopping_criteria=[stopping_criteria])
+
+outputs = tokenizer.batch_decode(output_ids[:, input_ids.shape[1]:], skip_special_tokens=True)[0]
+outputs = outputs.strip()
+print(outputs)
+# Large left pneumothorax is present with apical pneumothorax component
+#  measuring approximately 3.4 cm in craniocaudal dimension, and a basilar
+#  component overlying the left hemidiaphragm, with visceral pleural line just
+#  below the left seventh posterior rib.  Cardiomediastinal contours are normal.
+#  The lungs are clear.  No pleural effusion.
+```
+
+### Usage and License Notices
+The model’s intended use is to generate draft findings of chest X-ray images in English. It is provided for reproducibility and to enable further research.
+
+**The data, code, and model checkpoints are licensed and intended for research use only.** The code and model checkpoints are subject to additional restrictions as determined by the Terms of Use of LLaMA, Vicuna, and GPT-4 respectively. Code and model checkpoints may be used for **research purposes only** and should **not** be used in direct clinical care or for any clinical decision making purpose.  You bear sole responsibility for use of the code and model checkpoints, including incorporation into any product used for clinical purposes.
+
+### Ethical Considerations and Limitations
+
+
+Microsoft believes Responsible AI is a shared responsibility and we have identified six principles and practices to help organizations address risks, innovate, and create value: fairness, reliability and safety, privacy and security, inclusiveness, transparency, and accountability. When downloaded or used in accordance with our terms of service, developers should work with their supporting model team to ensure this model meets requirements for the relevant use case and addresses unforeseen product misuse. 
+While testing the model with images and/or text, ensure that the data is PHI free and that there are no patient information or information that can be tracked to a patient identity.
+
+#### Out-of-scope use
+The model is NOT designed for the following use cases:
+*	Use by clinicians to inform clinical decision-making, as a diagnostic tool or as a medical device. Although LLaVA-Rad attains state-of-the-art performance, it is not designed or intended to be deployed in clinical settings as-is nor is it for use in the diagnosis, cure, mitigation, treatment, or prevention of disease or other conditions (including to support clinical decision-making), or as a substitute of professional medical advice, diagnosis, treatment, or clinical judgment of a healthcare professional. 
+*	Scenarios without consent for data - Any scenario that uses health data for a purpose for which consent was not obtained.  
+*	Use outside of health scenarios - Any scenario that uses non-medical related image and/or serving purposes outside of the healthcare domain.
+Please see Microsoft's Responsible AI Principles and approach available at https://www.microsoft.com/en-us/ai/principles-and-approach/
+
+#### Data provenance and potential biases
+BiomedCLIP-CXR was trained with datasets containing chest X-ray images acquired from adult patients in the United States (Stanford Hospital in CA, Beth Israel Deaconess Medical Center in MA), New Zealand, Brazil, Vietnam, Spain, and China. Synthetic free-text reports were generated for datasets without readily available corresponding reports, or translated from other languages such as Spanish into English. 
+
+The projector and decoder layers of LLaVA-Rad were trained using MIMIC-CXR data, containing English only reports for patients from Boston, MA.  Please see [our manuscript](https://arxiv.org/abs/2403.08002v5) for references to individual publications that describe demographic characteristics of patients from whom chest X-ray images were obtained in detail.
+
+Radiology reporting styles can vary within and across health systems and regions. This may impact the generalizability of the model to unseen populations and as such the model should always be tested on the intended deployment population.
+
+This model may produce errors in the generated findings (see CheXprompt evaluation and other results in [manuscript](https://arxiv.org/pdf/2403.08002) for details).
+
+## LLaVA-Rad Basics
+
+
+### Abstract
+The scaling laws and extraordinary performance of large foundation models motivate the development and utilization of such models in biomedicine. However, despite early promising results on some biomedical benchmarks, there are still major challenges that need to be addressed before these models can be used in real-world clinics.
+This work aims to address these challenges by training small multimodal models (SMMs) to bridge competency gaps for unmet clinical need in radiology. To maximize data efficiency, we adopt a modular approach by incorporating state-of-the-art pre-trained models for image and text modalities, and focusing on training a lightweight adapter to ground each modality to the text embedding space. For training, we assemble a large dataset of over 697 thousand radiology image-text pairs.
+LLaVA-Rad (7B) model attains state-of-the-art results on standard radiology tasks such as report generation and cross-modal retrieval, even outperforming much larger models such as GPT-4V and Med-PaLM M (84B). Moreover, LLaVA-Rad requires only one day to be trained on over 697 thousand image-text pairs using a standard 8-A100 GPU cluster, allowing further fine-tuning by clinicians using their own data. The inference of LLaVA-Rad is fast and can be performed on a single V100 GPU in private settings, offering a promising state-of-the-art tool for real-world clinical applications.
+
+
+### Model Architecture
+LLaVA-Rad follows the LLaVA v1.5 architecture, which employs an image encoder, a transformer-based language decoder, and a multilayer perceptron connector. For the image encoder, it uses a custom model named BiomedCLIP-CXR, based on [BiomedCLIP](https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224). For the language decoder it uses [lmsys/vicuna-7b-v1.5](https://huggingface.co/lmsys/vicuna-7b-v1.5).
+
+### Evaluation Results
+
+![Evaluation Results](results.jpg)
+
+To reproduce the results above, please follow the steps outlined in the official code implementation: [setup](https://github.com/microsoft/llava-rad#requirements), [inference and eval](https://github.com/microsoft/llava-rad#inference) .
+
+Please see the paper for detailed information about methods and results: https://arxiv.org/pdf/2412.10337.
+
+You may also find the CheXprompt library useful for evaluation of generated reports. It is accessible at https://github.com/microsoft/chexprompt.
+
+To use this model to reproduce results in the manuscript, follow these steps using the [supported code](https://github.com/microsoft/llava-rad):
+
+1. Preparation 
+   Before running the commands below, you need to have the data, image folder, and the above checkpoints ready. 
+   1. Text Data
+   
+   To download the data, sign the data use agreement and follow the instructions for download at LLaVA-Rad MIMIC-CXR Annotations on PhysioNet. This will include reports with extracted sections in LLaVA format, split into train/dev/test. 
+   
+   2. Images 
+   
+   You need to download the MIMIC-CXR-JPG images from PhysioNet by signing the data use agreement and following the instructions. 
+   
+   3. Model weights 
+   You can find the pretrained model weights for BiomedCLIP-CXR and LLaVA-Rad at https://huggingface.co/microsoft/llava-rad (this repository).
+
+ 
+Before proceeding, change the paths in the scripts below according to where you downloaded the data. 
+Batch size is set for 4-GPU machines. If your machine has a difference number of GPUs, please change batch size.
+
+2. Inference 
+
+To perform inference, update the paths in the evaluation script, found in the [official codebase](https://github.com/microsoft/llava-rad).
+
+```python
+bash scripts/eval.sh
+```
+
+3. Evaluation
+
+If you have run inference using multiple GPUs and have a resulting set of chunks with results, make sure you concatenate prediction chunks into a single file before running the following command:
+
+```python
+cd llava/eval/rr_eval 
+python run.py ${YOUR_PREDICTION_FILE} 
+```
+
+
+
+### Citation
+Please cite our paper if you use the code, model, or data.
+
+```
+Zambrano Chaves JM, Huang S-C, Xu Y, Xu H, Usuyama N, Zhang S, et al. Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation. arXiv preprint arXiv:2403.08002. 2024.
+```
+
+Bibtex:
+```bibtex
+@article{zambranochaves2024llavarad,
+  title = {Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation},
+  author = {Zambrano Chaves, JM and Huang, S-C and Xu, Y and Xu, H and Usuyama, N and Zhang, S, et al.},
+  journal = {arXiv preprint arXiv:2403.08002},
+  year = {2024},
+  url = {https://arxiv.org/pdf/2403.08002}
+}
+```
+
+
+## Further Information
+
+### Data Specification
+Additional details regarding the data used can be found in the corresponding data release, LLaVA-Rad MIMIC-CXR Annotations: https://physionet.org/content/llava-rad-mimic-cxr-annotation/1.0.0/ 
+
+### Related Models
+CxrReportGen (https://aka.ms/CXRReportGenModelCard) offers similar functionality based on the [MAIRA-2](https://arxiv.org/abs/2406.04449) framework.
+

adapter_config.json

@@ -0,0 +1,26 @@
+{
+  "auto_mapping": null,
+  "base_model_name_or_path": "lmsys/vicuna-7b-v1.5",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "lora_alpha": 128,
+  "lora_dropout": 0.05,
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "down_proj",
+    "gate_proj",
+    "o_proj",
+    "v_proj",
+    "q_proj",
+    "up_proj"
+  ],
+  "task_type": "CAUSAL_LM"
+}

adapter_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b8d0649a878cff912d0347775d047fec1215e22c1aed740454d2f32165331103
+size 319971402

biomedclipcxr_518.json

@@ -0,0 +1,17 @@
+{
+    "embed_dim": 512,
+    "vision_cfg": {
+        "timm_model_name": "vit_base_patch14_dinov2.lvd142m",
+        "timm_model_pretrained": false,
+        "timm_pool": "",
+        "timm_proj": "linear",
+        "image_size": 518
+    },
+    "text_cfg": {
+        "hf_model_name": "microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract",
+        "hf_tokenizer_name": "microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract",
+        "hf_proj_type": "mlp",
+        "hf_pooler_type": "cls_last_hidden_state_pooler",
+        "context_length": 256
+    }
+}

biomedclipcxr_518_checkpoint.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:698b1afcaf07365ec60bcaba3f1e96f556f5c40e5f405db424bb4a9221144319
+size 2363054683

config.json

@@ -0,0 +1,40 @@
+{
+  "_name_or_path": "lmsys/vicuna-7b-v1.5",
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "freeze_mm_mlp_adapter": false,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "image_aspect_ratio": "square",
+  "image_grid_pinpoints": null,
+  "initializer_range": 0.02,
+  "intermediate_size": 11008,
+  "max_position_embeddings": 4096,
+  "mm_hidden_size": 768,
+  "mm_projector_type": "mlp2x_gelu",
+  "mm_use_im_patch_token": false,
+  "mm_use_im_start_end": false,
+  "mm_vision_select_feature": "patch",
+  "mm_vision_select_layer": -2,
+  "mm_vision_tower": "biomedclip_cxr_518",
+  "mm_vision_tower_checkpoint": "biomedclipcxr_518_checkpoint.pt",
+  "mm_vision_tower_config": "llava/model/multimodal_encoder/open_clip_encoder/model_configs/biomedclipcxr_518.json",
+  "model_type": "llava",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 32,
+  "pad_token_id": 0,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float16",
+  "transformers_version": "4.31.0",
+  "tune_mm_mlp_adapter": false,
+  "use_cache": true,
+  "use_mm_proj": true,
+  "vocab_size": 32000
+}

non_lora_trainables.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1b87a97a84e803896db2720857d71f1a782f4f108c9f2d5eba06cb1ba660f9f8
+size 39864496