Create README.md

README.md

@@ -0,0 +1,120 @@
+---
+# For reference on model card metadata, see the spec: https://github.com/huggingface/hub-docs/blob/main/modelcard.md?plain=1
+# Doc / guide: https://huggingface.co/docs/hub/model-cards
+{}
+---
+
+# LLaVA Model Card
+
+## Model Details
+
+Model type: LLaVA is an open-source chatbot trained by fine-tuning LLM on multimodal instruction-following data. It is an auto-regressive language model, based on the transformer architecture. 
+
+Base LLM: Qwen/Qwen1.5-72B-Chat
+
+### Model Description
+
+**Repository:** https://github.com/EvolvingLMMs-Lab/LLaVA-NEXT
+
+**Primary intended uses:** The primary use of LLaVA is research on large multimodal models and chatbots.
+
+**Primary intended users:** The primary intended users of the model are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence.
+
+### License Notices
+  
+This project utilizes certain datasets and checkpoints that are subject to their respective original licenses. Users must comply with all terms and conditions of these original licenses, including but not limited to the OpenAI Terms of Use for the dataset and the specific licenses for base language models for checkpoints trained using the dataset (e.g. Llama-1/2 community license for LLaMA-2 and Vicuna-v1.5, Tongyi Qianwen RESEARCH LICENSE AGREEMENT and Llama-3 Research License). This project does not impose any additional constraints beyond those stipulated in the original licenses. Furthermore, users are reminded to ensure that their use of the dataset and checkpoints is in compliance with all applicable laws and regulations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Procedure
+
+We conducted the training on LLaVA-1.6's codebase with adding support of Llama-3 and Qwen model.
+
+### Training Hyperparameters
+
+```shell
+LLM_VERSION="Qwen/Qwen1.5-72B-Chat"
+LLM_VERSION_CLEAN="${LLM_VERSION//\//_}"
+VISION_MODEL_VERSION="openai/clip-vit-large-patch14-336"
+VISION_MODEL_VERSION_CLEAN="${VISION_MODEL_VERSION//\//_}"
+
+PROMPT_VERSION=plain
+PRETRAIN_DATA_VERSION="blip558k"
+############### Pretrain ################
+
+BASE_RUN_NAME="llavanext-${LLM_VERSION_CLEAN}-${VISION_MODEL_VERSION_CLEAN}-pretrain_${PRETRAIN_DATA_VERSION}_plain"
+echo "BASE_RUN_NAME: ${BASE_RUN_NAME}"
+
+PROMPT_VERSION="qwen_1_5"
+MID_RUN_NAME="llavanext-${LLM_VERSION_CLEAN}-${VISION_MODEL_VERSION_CLEAN}-pretrain_${PRETRAIN_DATA_VERSION}_plain-ft_la1_6mix_d32k"
+echo "MID_RUN_NAME: ${MID_RUN_NAME}"
+
+torchrun # with necessary torchrun information for distributed training\
+    llava/train/train_mem.py \
+    --deepspeed scripts/zero3.json \
+    --model_name_or_path $LLM_VERSION \
+    --version $PROMPT_VERSION \
+    --data_path="/path/to/data/llava_instruct/llava1_6mix.json" \
+    --image_folder /path/to/data/llava_data \
+    --pretrain_mm_mlp_adapter="./checkpoints/projectors/${BASE_RUN_NAME}/mm_projector.bin" \
+    --mm_tunable_parts="mm_vision_tower,mm_mlp_adapter,mm_language_model" \
+    --mm_vision_tower_lr=2e-6 \
+    --vision_tower ${VISION_MODEL_VERSION} \
+    --mm_projector_type mlp2x_gelu \
+    --mm_vision_select_layer -2 \
+    --mm_use_im_start_end False \
+    --mm_use_im_patch_token False \
+    --group_by_modality_length True \
+    --image_aspect_ratio anyres \
+    --image_grid_pinpoints "[(336, 672), (672, 336), (672, 672), (1008, 336), (336, 1008)]" \
+    --mm_patch_merge_type spatial_unpad \
+    --bf16 True \
+    --run_name $MID_RUN_NAME \
+    --output_dir ./checkpoints/$MID_RUN_NAME \
+    --num_train_epochs 1 \
+    --per_device_train_batch_size 1 \
+    --per_device_eval_batch_size 4 \
+    --gradient_accumulation_steps 2 \
+    --evaluation_strategy "no" \
+    --save_strategy "steps" \
+    --save_steps 3000 \
+    --save_total_limit 1 \
+    --learning_rate 1e-5 \
+    --weight_decay 0. \
+    --warmup_ratio 0.03 \
+    --lr_scheduler_type "cosine" \
+    --logging_steps 1 \
+    --tf32 True \
+    --model_max_length 32768 \
+    --gradient_checkpointing True \
+    --dataloader_num_workers 8 \
+    --lazy_preprocess True \
+    --report_to wandb \
+    --torch_compile True \
+    --torch_compile_backend "inductor"
+    --dataloader_drop_last True
+```
+
+### Training Data
+
+- 558K filtered image-text pairs from LAION/CC/SBU, captioned by BLIP.
+- 158K GPT-generated multimodal instruction-following data.
+- 500K academic-task-oriented VQA data mixture.
+- 50K GPT-4V data mixture.
+- 40K ShareGPT data.
+
+#### Speeds, Sizes, Times [optional]
+
+The training cost is ~50-60 hours on 8 x 8 NVIDIA A100-SXM4-80GB (may vary due to hardware differences). 
+
+[More Information Needed]
+
+## Evaluation
+
+The evaluation is conducted with the support of [`lmms-eval`](https://github.com/EvolvingLMMs-Lab/lmms-eval)