YAML Metadata Warning: empty or missing yaml metadata in repo card (https://huggingface.co/docs/hub/model-cards#model-card-metadata)

Introduction

MiniCPM_o_2.6-FlagOS-NVIDIA provides an all-in-one deployment solution, enabling execution of MiniCPM_o_2.6 on NVIDIA GPUs. As the first-generation release for the NVIDIA-H100, this package delivers two key features:

  1. Comprehensive Integration:
    • Integrated with FlagScale (https://github.com/FlagOpen/FlagScale).
    • Open-source inference execution code, preconfigured with all necessary software and hardware settings.
    • Pre-built Docker image for rapid deployment on NVIDIA-H100.
  2. Consistency Validation:
    • Evaluation tests verifying consistency of results between the official and ours.

Technical Summary

Serving Engine

We use FlagScale as the serving engine to improve the portability of distributed inference.

FlagScale is an end-to-end framework for large models across multiple chips, maximizing computational resource efficiency while ensuring model effectiveness. It ensures both ease of use and high performance for users when deploying models across different chip architectures:

  • One-Click Service Deployment: FlagScale provides a unified and simple command execution mechanism, allowing users to fast deploy services seamlessly across various hardware platforms using the same command. This significantly reduces the entry barrier and enhances user experience.
  • Automated Deployment Optimization: FlagScale automatically optimizes distributed parallel strategies based on the computational capabilities of different AI chips, ensuring optimal resource allocation and efficient utilization, thereby improving overall deployment performance.
  • Automatic Operator Library Switching: Leveraging FlagScale's unified Runner mechanism and deep integration with FlagGems, users can seamlessly switch to the FlagGems operator library for inference by simply adding environment variables in the configuration file.

Triton Support

We validate the execution of DeepSeed-R1 model with a Triton-based operator library as a PyTorch alternative.

We use a variety of Triton-implemented operation kernelsโ€”approximately 70%โ€”to run the MiniCPM_o_2.6 model. These kernels come from two main sources:

  • Most Triton kernels are provided by FlagGems (https://github.com/FlagOpen/FlagGems). You can enable FlagGems kernels by setting the environment variable USE_FLAGGEMS. For more details, please refer to the "How to Run Locally" section.

  • Also included are Triton kernels from vLLM.

Bundle Download

Usage Nvidia
Basic Image basic software environment that supports model running 'docker pull flagrelease-registry.cn-beijing.cr.aliyuncs.com/flagrelease/flagrelease:deepseek-flagos-nvidia

Evaluation Results

Benchmark Result

Metrics MiniCPM_o_2.6-H100-CUDA MiniCPM_o_2.6-H100-FlagOS
mmmu_val 48.11 48.33
math_vision_test 22.89 22.30
ocrbench_test 85.80 85.70
blink_val 54.87 55.81
mmmvet_v2 57.66 59.03
mmmu_pro_vision_test 70.46 69.77
mmmu_pro_standard_test 30.46 30.81
cmmmu_val 39.33 39.33
cii_bench_test 50.07 50.33

How to Run Locally

๐Ÿ“Œ Getting Started

Download open-source weights

pip install modelscope
modelscope download --model <Model Name> --local_dir <Cache Path>

Download the FlagOS image

docker pull <IMAGE>

Start the inference service

docker run -itd --name flagrelease_nv  --privileged --gpus all --net=host --ipc=host --device=/dev/infiniband --shm-size 512g --ulimit memlock=-1 -v <CKPT_PATH>:<CKPT_PATH> flagrelease-registry.cn-beijing.cr.aliyuncs.com/flagrelease/flagrelease:deepseek-flagos-nvidia /bin/bash

docker exec -it flagrelease_nv /bin/bash

conda activate flagscale-inference

Download and install FlagGems

git clone https://github.com/FlagOpen/FlagGems.git
cd FlagGems
pip install .
cd ../

Modify the configuration

cd FlagScale/examples/minicpm_o_2.6/conf
# Modify the configuration in config_minicpm_o_2.6.yaml
defaults:
  - _self_
  - serve: minicpm_o_2.6
experiment:
  exp_name: minicpm_o_2.6
  exp_dir: outputs/${experiment.exp_name}
  task:
    type: serve
  deploy:
    use_fs_serve: false
  runner:
    ssh_port: 22
  envs:
    CUDA_DEVICE_MAX_CONNECTIONS: 1
  cmds:
    before_start: source /root/miniconda3/bin/activate flagscale-inference && export USE_FLAGGEMS=1
action: run
hydra:
  run:
    dir: ${experiment.exp_dir}/hydra
cd FlagScale/examples/minicpm_o_2.6/conf/serve
# Modify the configuration in minicpm_o_2.6.yaml
- serve_id: vllm_model
  engine: vllm
  engine_args:
    model: /models/MiniCPM_o_2 # path of weight of DeepSeek-R1-Distill-Qwen-32B
    served_model_name: minicpmo26-flagos
    tensor_parallel_size: 1
    pipeline_parallel_size: 1
    gpu_memory_utilization: 0.9
    max_num_seqs: 256
    limit_mm_per_prompt: image=18
    port: 9010
    trust_remote_code: true
    enable_chunked_prefill: true
# install flagscale
cd FlagScale/
pip install .

#ใ€Verifiable on a single machineใ€‘

Serve

flagscale serve <Model>

Contributing

We warmly welcome global developers to join us:

  1. Submit Issues to report problems
  2. Create Pull Requests to contribute code
  3. Improve technical documentation
  4. Expand hardware adaptation support

๐Ÿ“ž Contact Us

Scan the QR code below to add our WeChat group

send "FlagRelease"

WeChat

License

This project and related model weights are licensed under the MIT License.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support