Instructions to use HuggingFaceM4/idefics-9b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use HuggingFaceM4/idefics-9b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="HuggingFaceM4/idefics-9b")

# Load model directly
from transformers import AutoProcessor, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained("HuggingFaceM4/idefics-9b")
model = AutoModelForImageTextToText.from_pretrained("HuggingFaceM4/idefics-9b")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use HuggingFaceM4/idefics-9b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "HuggingFaceM4/idefics-9b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HuggingFaceM4/idefics-9b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/HuggingFaceM4/idefics-9b

SGLang

How to use HuggingFaceM4/idefics-9b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "HuggingFaceM4/idefics-9b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HuggingFaceM4/idefics-9b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "HuggingFaceM4/idefics-9b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HuggingFaceM4/idefics-9b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use HuggingFaceM4/idefics-9b with Docker Model Runner:
```
docker model run hf.co/HuggingFaceM4/idefics-9b
```

Why is the use of the <fake_token_around_image> token different from Flamingo's <EOC> token?

#12

by gigant - opened Oct 26, 2023

Discussion

gigant

Oct 26, 2023

•

edited Oct 26, 2023

From my understanding, the <fake_token_around_image> replaces the <EOC> end of chunk token from the original Flamingo paper. According to the latter, the <EOC> token is used at the end of a text chunk: "prior to any image and at the end of the document".
However, if I am not mistaken, the <fake_token_around_image> is to be used before and after the image token, and is not used at the end of the document.
Why this difference? Did you do or refer to experiments/ablation studies on this added token?

Thank you

VictorSanh

Oct 27, 2023

Hey,
We use:

<fake_token_around_image> to wrap the <image> tokens. As such, if we have consecutive images, the token sequence will be <fake_token_around_image><image><fake_token_around_image><image><fake_token_around_image>, if you only have one image, it will be <fake_token_around_image><image><fake_token_around_image>.
<eos> tokens to mark the end of a document.
<bos> tokens to mark the beginning of a document.

At the end of the day, the reason why we wrap image tokens around other ones is to ensure that that we always have tokens associated with all the images even when they are consecutive. it ensures the model can reason across all images. Beyond that, the exact implementation (EOC or not) depends on other factors.
For instance, we started using \n\n instead of a new learned <fake_token_around_image> token but the model was confusing the \n\n as in double line breaks and \n\n as in an image is next.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment