Instructions to use lmms-lab/llava-onevision-qwen2-7b-ov-chat with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use lmms-lab/llava-onevision-qwen2-7b-ov-chat with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="lmms-lab/llava-onevision-qwen2-7b-ov-chat")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("lmms-lab/llava-onevision-qwen2-7b-ov-chat", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use lmms-lab/llava-onevision-qwen2-7b-ov-chat with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "lmms-lab/llava-onevision-qwen2-7b-ov-chat"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lmms-lab/llava-onevision-qwen2-7b-ov-chat",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/lmms-lab/llava-onevision-qwen2-7b-ov-chat

SGLang

How to use lmms-lab/llava-onevision-qwen2-7b-ov-chat with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "lmms-lab/llava-onevision-qwen2-7b-ov-chat" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lmms-lab/llava-onevision-qwen2-7b-ov-chat",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "lmms-lab/llava-onevision-qwen2-7b-ov-chat" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lmms-lab/llava-onevision-qwen2-7b-ov-chat",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use lmms-lab/llava-onevision-qwen2-7b-ov-chat with Docker Model Runner:
```
docker model run hf.co/lmms-lab/llava-onevision-qwen2-7b-ov-chat
```

Upload folder using huggingface_hub

by luodian - opened Sep 13, 2024

base: refs/heads/main

←

from: refs/pr/1

Discussion Files changed

+458265

-0

luodian

LMMs-Lab org Sep 13, 2024

•

edited Sep 13, 2024

Upload folder using huggingface_hub

Multi commit ID: 78a9d015959e22b9bfd9480a9a2aaea8570e10b5875699839743d2a687e62fc0

Scheduled commits:

Upload 1 file(s) totalling 4.9G (dd8cb4e66ac637ecac45642b0956514fdeb25f9ac7a22c01c802064fdbe7e0ae)
Upload 1 file(s) totalling 4.9G (93539cacda63c3fdcab43981fdd314df79ee89bc3e965ada9594e78e93d0ffe4)
Upload 1 file(s) totalling 5.0G (fb8ad8969f84e9e2e60e75bc4d872ba6e1ac289a8449b0b1f8d990c9435b84fa)
Upload 12 file(s) totalling 1.3G (be0f70caefe53dd92deb6041cda716d9ca689fb7882a9c006dec3d49976dbb51)

This is a PR opened using the huggingface_hub library in the context of a multi-commit. PR can be commented as a usual PR. However, please be aware that manually updating the PR description, changing the PR status, or pushing new commits, is not recommended as it might corrupt the commit process. Learn more about multi-commits in this guide.

dd8cb4e66ac637ecac45642b0956514fdeb25f9ac7a22c01c802064fdbe7e0aeeceb0476

93539cacda63c3fdcab43981fdd314df79ee89bc3e965ada9594e78e93d0ffe41c521dcd

fb8ad8969f84e9e2e60e75bc4d872ba6e1ac289a8449b0b1f8d990c9435b84fad4538760

be0f70caefe53dd92deb6041cda716d9ca689fb7882a9c006dec3d49976dbb51859f01f0

luodian changed pull request title from [WIP] Upload folder using huggingface_hub (multi-commit 78a9d015959e22b9bfd9480a9a2aaea8570e10b5875699839743d2a687e62fc0) to Upload folder using huggingface_hub Sep 13, 2024

luodian

LMMs-Lab org Sep 13, 2024

Multi-commit is now completed! You can ping the repo owner to review the changes. This PR can now be commented or modified without risking to corrupt it.

This is a comment posted using the huggingface_hub library in the context of a multi-commit. Learn more about multi-commits in this guide.

luodian changed pull request status to open Sep 13, 2024

luodian

LMMs-Lab org Sep 13, 2024

create_pr=False has been passed so PR is automatically merged.

This is a comment posted using the huggingface_hub library in the context of a multi-commit. Learn more about multi-commits in this guide.

luodian changed pull request status to merged Sep 13, 2024

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment