多模態輸入

多模態輸入#

本頁教您如何將多模態輸入傳遞給 vLLM 中的多模態模型。

注意

我們正在積極迭代多模態支援。請參閱此 RFC 瞭解即將到來的更改，如果您有任何反饋或功能請求，請在 GitHub 上開啟一個 issue。

離線推理#

要輸入多模態資料，請遵循 vllm.inputs.PromptType 中的此模式

prompt：prompt 應遵循 HuggingFace 上記錄的格式。
multi_modal_data：這是一個字典，它遵循 vllm.multimodal.inputs.MultiModalDataDict 中定義的模式。

影像輸入#

您可以將單個影像傳遞到多模態字典的 'image' 欄位，如下例所示

from vllm import LLM

llm = LLM(model="llava-hf/llava-1.5-7b-hf")

# Refer to the HuggingFace repo for the correct format to use
prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"

# Load the image using PIL.Image
image = PIL.Image.open(...)

# Single prompt inference
outputs = llm.generate({
    "prompt": prompt,
    "multi_modal_data": {"image": image},
})

for o in outputs:
    generated_text = o.outputs[0].text
    print(generated_text)

# Batch inference
image_1 = PIL.Image.open(...)
image_2 = PIL.Image.open(...)
outputs = llm.generate(
    [
        {
            "prompt": "USER: <image>\nWhat is the content of this image?\nASSISTANT:",
            "multi_modal_data": {"image": image_1},
        },
        {
            "prompt": "USER: <image>\nWhat's the color of this image?\nASSISTANT:",
            "multi_modal_data": {"image": image_2},
        }
    ]
)

for o in outputs:
    generated_text = o.outputs[0].text
    print(generated_text)

完整示例：examples/offline_inference/vision_language.py

要在同一文字 prompt 中替換多個影像，您可以傳遞影像列表

from vllm import LLM

llm = LLM(
    model="microsoft/Phi-3.5-vision-instruct",
    trust_remote_code=True,  # Required to load Phi-3.5-vision
    max_model_len=4096,  # Otherwise, it may not fit in smaller GPUs
    limit_mm_per_prompt={"image": 2},  # The maximum number to accept
)

# Refer to the HuggingFace repo for the correct format to use
prompt = "<|user|>\n<|image_1|>\n<|image_2|>\nWhat is the content of each image?<|end|>\n<|assistant|>\n"

# Load the images using PIL.Image
image1 = PIL.Image.open(...)
image2 = PIL.Image.open(...)

outputs = llm.generate({
    "prompt": prompt,
    "multi_modal_data": {
        "image": [image1, image2]
    },
})

for o in outputs:
    generated_text = o.outputs[0].text
    print(generated_text)

完整示例：examples/offline_inference/vision_language_multi_image.py

多影像輸入可以擴充套件到執行影片字幕。我們使用 Qwen2-VL 展示這一點，因為它支援影片

from vllm import LLM

# Specify the maximum number of frames per video to be 4. This can be changed.
llm = LLM("Qwen/Qwen2-VL-2B-Instruct", limit_mm_per_prompt={"image": 4})

# Create the request payload.
video_frames = ... # load your video making sure it only has the number of frames specified earlier.
message = {
    "role": "user",
    "content": [
        {"type": "text", "text": "Describe this set of frames. Consider the frames to be a part of the same video."},
    ],
}
for i in range(len(video_frames)):
    base64_image = encode_image(video_frames[i]) # base64 encoding.
    new_image = {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}
    message["content"].append(new_image)

# Perform inference and log output.
outputs = llm.chat([message])

for o in outputs:
    generated_text = o.outputs[0].text
    print(generated_text)

影片輸入#

您可以將 NumPy 陣列列表直接傳遞到多模態字典的 'video' 欄位，而不是使用多影像輸入。

完整示例：examples/offline_inference/vision_language.py

音訊輸入#

您可以將元組 (array, sampling_rate) 傳遞到多模態字典的 'audio' 欄位。

完整示例：examples/offline_inference/audio_language.py

嵌入輸入#

要將屬於資料型別（即影像、影片或音訊）的預計算嵌入直接輸入到語言模型，請將形狀為 (num_items, feature_size, LM 的 hidden_size) 的張量傳遞到多模態字典的相應欄位。

from vllm import LLM

# Inference with image embeddings as input
llm = LLM(model="llava-hf/llava-1.5-7b-hf")

# Refer to the HuggingFace repo for the correct format to use
prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"

# Embeddings for single image
# torch.Tensor of shape (1, image_feature_size, hidden_size of LM)
image_embeds = torch.load(...)

outputs = llm.generate({
    "prompt": prompt,
    "multi_modal_data": {"image": image_embeds},
})

for o in outputs:
    generated_text = o.outputs[0].text
    print(generated_text)

對於 Qwen2-VL 和 MiniCPM-V，我們接受與嵌入一起的附加引數

# Construct the prompt based on your model
prompt = ...

# Embeddings for multiple images
# torch.Tensor of shape (num_images, image_feature_size, hidden_size of LM)
image_embeds = torch.load(...)

# Qwen2-VL
llm = LLM("Qwen/Qwen2-VL-2B-Instruct", limit_mm_per_prompt={"image": 4})
mm_data = {
    "image": {
        "image_embeds": image_embeds,
        # image_grid_thw is needed to calculate positional encoding.
        "image_grid_thw": torch.load(...),  # torch.Tensor of shape (1, 3),
    }
}

# MiniCPM-V
llm = LLM("openbmb/MiniCPM-V-2_6", trust_remote_code=True, limit_mm_per_prompt={"image": 4})
mm_data = {
    "image": {
        "image_embeds": image_embeds,
        # image_sizes is needed to calculate details of the sliced image.
        "image_sizes": [image.size for image in images],  # list of image sizes
    }
}

outputs = llm.generate({
    "prompt": prompt,
    "multi_modal_data": mm_data,
})

for o in outputs:
    generated_text = o.outputs[0].text
    print(generated_text)

線上服務#

我們的 OpenAI 相容伺服器透過 Chat Completions API 接受多模態資料。

重要

使用 Chat Completions API 需要聊天模板。

雖然大多數模型都帶有聊天模板，但對於其他模型，您必須自己定義一個。聊天模板可以根據模型 HuggingFace 倉庫上的文件推斷出來。例如，LLaVA-1.5 (llava-hf/llava-1.5-7b-hf) 需要一個聊天模板，可以在這裡找到：examples/template_llava.jinja

影像輸入#

影像輸入根據 OpenAI Vision API 獲得支援。這是一個使用 Phi-3.5-Vision 的簡單示例。

首先，啟動 OpenAI 相容伺服器

vllm serve microsoft/Phi-3.5-vision-instruct --task generate \
  --trust-remote-code --max-model-len 4096 --limit-mm-per-prompt image=2

然後，您可以按如下方式使用 OpenAI 客戶端

from openai import OpenAI

openai_api_key = "EMPTY"
openai_api_base = "https://:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

# Single-image input inference
image_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"

chat_response = client.chat.completions.create(
    model="microsoft/Phi-3.5-vision-instruct",
    messages=[{
        "role": "user",
        "content": [
            # NOTE: The prompt formatting with the image token `<image>` is not needed
            # since the prompt will be processed automatically by the API server.
            {"type": "text", "text": "What’s in this image?"},
            {"type": "image_url", "image_url": {"url": image_url}},
        ],
    }],
)
print("Chat completion output:", chat_response.choices[0].message.content)

# Multi-image input inference
image_url_duck = "https://upload.wikimedia.org/wikipedia/commons/d/da/2015_Kaczka_krzy%C5%BCowka_w_wodzie_%28samiec%29.jpg"
image_url_lion = "https://upload.wikimedia.org/wikipedia/commons/7/77/002_The_lion_king_Snyggve_in_the_Serengeti_National_Park_Photo_by_Giles_Laurent.jpg"

chat_response = client.chat.completions.create(
    model="microsoft/Phi-3.5-vision-instruct",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "What are the animals in these images?"},
            {"type": "image_url", "image_url": {"url": image_url_duck}},
            {"type": "image_url", "image_url": {"url": image_url_lion}},
        ],
    }],
)
print("Chat completion output:", chat_response.choices[0].message.content)

完整示例：examples/online_serving/openai_chat_completion_client_for_multimodal.py

提示

vLLM 也支援從本地檔案路徑載入：您可以在啟動 API 伺服器/引擎時透過 --allowed-local-media-path 指定允許的本地媒體路徑，並在 API 請求中將檔案路徑作為 url 傳遞。

提示

無需在 API 請求的文字內容中放置影像佔位符 - 它們已由影像內容表示。實際上，您可以透過交錯文字和影像內容將影像佔位符放置在文字中間。

注意

預設情況下，透過 HTTP URL 獲取影像的超時時間為 5 秒。您可以透過設定環境變數來覆蓋此設定

export VLLM_IMAGE_FETCH_TIMEOUT=<timeout>

影片輸入#

除了 image_url 之外，您還可以透過 video_url 傳遞影片檔案。這是一個使用 LLaVA-OneVision 的簡單示例。

首先，啟動 OpenAI 相容伺服器

vllm serve llava-hf/llava-onevision-qwen2-0.5b-ov-hf --task generate --max-model-len 8192

然後，您可以按如下方式使用 OpenAI 客戶端

from openai import OpenAI

openai_api_key = "EMPTY"
openai_api_base = "https://:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

video_url = "http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerFun.mp4"

## Use video url in the payload
chat_completion_from_url = client.chat.completions.create(
    messages=[{
        "role":
        "user",
        "content": [
            {
                "type": "text",
                "text": "What's in this video?"
            },
            {
                "type": "video_url",
                "video_url": {
                    "url": video_url
                },
            },
        ],
    }],
    model=model,
    max_completion_tokens=64,
)

result = chat_completion_from_url.choices[0].message.content
print("Chat completion output from image url:", result)

完整示例：examples/online_serving/openai_chat_completion_client_for_multimodal.py

注意

預設情況下，透過 HTTP URL 獲取影片的超時時間為 30 秒。您可以透過設定環境變數來覆蓋此設定

export VLLM_VIDEO_FETCH_TIMEOUT=<timeout>

音訊輸入#

音訊輸入根據 OpenAI Audio API 獲得支援。這是一個使用 Ultravox-v0.5-1B 的簡單示例。

首先，啟動 OpenAI 相容伺服器

vllm serve fixie-ai/ultravox-v0_5-llama-3_2-1b

然後，您可以按如下方式使用 OpenAI 客戶端

import base64
import requests
from openai import OpenAI
from vllm.assets.audio import AudioAsset

def encode_base64_content_from_url(content_url: str) -> str:
    """Encode a content retrieved from a remote url to base64 format."""

    with requests.get(content_url) as response:
        response.raise_for_status()
        result = base64.b64encode(response.content).decode('utf-8')

    return result

openai_api_key = "EMPTY"
openai_api_base = "https://:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

# Any format supported by librosa is supported
audio_url = AudioAsset("winning_call").url
audio_base64 = encode_base64_content_from_url(audio_url)

chat_completion_from_base64 = client.chat.completions.create(
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "What's in this audio?"
            },
            {
                "type": "input_audio",
                "input_audio": {
                    "data": audio_base64,
                    "format": "wav"
                },
            },
        ],
    }],
    model=model,
    max_completion_tokens=64,
)

result = chat_completion_from_base64.choices[0].message.content
print("Chat completion output from input audio:", result)

或者，您可以傳遞 audio_url，它是影像輸入的 image_url 的音訊對應物

chat_completion_from_url = client.chat.completions.create(
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "What's in this audio?"
            },
            {
                "type": "audio_url",
                "audio_url": {
                    "url": audio_url
                },
            },
        ],
    }],
    model=model,
    max_completion_tokens=64,
)

result = chat_completion_from_url.choices[0].message.content
print("Chat completion output from audio url:", result)

完整示例：examples/online_serving/openai_chat_completion_client_for_multimodal.py

注意

預設情況下，透過 HTTP URL 獲取音訊的超時時間為 10 秒。您可以透過設定環境變數來覆蓋此設定

export VLLM_AUDIO_FETCH_TIMEOUT=<timeout>

嵌入輸入#

要將屬於資料型別（即影像、影片或音訊）的預計算嵌入直接輸入到語言模型，請將形狀張量傳遞到多模態字典的相應欄位。

影像嵌入輸入#

對於影像嵌入，您可以將 base64 編碼的張量傳遞到 image_embeds 欄位。以下示例演示瞭如何將影像嵌入傳遞到 OpenAI 伺服器

image_embedding = torch.load(...)
grid_thw = torch.load(...) # Required by Qwen/Qwen2-VL-2B-Instruct

buffer = io.BytesIO()
torch.save(image_embedding, buffer)
buffer.seek(0)
binary_data = buffer.read()
base64_image_embedding = base64.b64encode(binary_data).decode('utf-8')

client = OpenAI(
    # defaults to os.environ.get("OPENAI_API_KEY")
    api_key=openai_api_key,
    base_url=openai_api_base,
)

# Basic usage - this is equivalent to the LLaVA example for offline inference
model = "llava-hf/llava-1.5-7b-hf"
embeds =  {
    "type": "image_embeds",
    "image_embeds": f"{base64_image_embedding}" 
}

# Pass additional parameters (available to Qwen2-VL and MiniCPM-V)
model = "Qwen/Qwen2-VL-2B-Instruct"
embeds =  {
    "type": "image_embeds",
    "image_embeds": {
        "image_embeds": f"{base64_image_embedding}" , # Required
        "image_grid_thw": f"{base64_image_grid_thw}"  # Required by Qwen/Qwen2-VL-2B-Instruct
    },
}
model = "openbmb/MiniCPM-V-2_6"
embeds =  {
    "type": "image_embeds",
    "image_embeds": {
        "image_embeds": f"{base64_image_embedding}" , # Required
        "image_sizes": f"{base64_image_sizes}"  # Required by openbmb/MiniCPM-V-2_6
    },
}
chat_completion = client.chat.completions.create(
    messages=[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": [
        {
            "type": "text",
            "text": "What's in this image?",
        },
        embeds,
        ],
    },
],
    model=model,
)

注意

只有一個訊息可以包含 {"type": "image_embeds"}。如果與需要附加引數的模型一起使用，您還必須為每個引數提供一個張量，例如 image_grid_thw、image_sizes 等。

多模態輸入

目錄

多模態輸入#

離線推理#

影像輸入#

影片輸入#

音訊輸入#

嵌入輸入#

線上服務#

影像輸入#

影片輸入#

音訊輸入#

嵌入輸入#

影像嵌入輸入#