An Overview of Alibaba’s Multimodal Model: Ovis-U1

The advancement of Artificial General Intelligence (AGI) towards human-level task performance is largely driven by Multimodal Large Language Models (MLLMs). Combining multiple modalities allows for greater information density in inputs and enhanced capabilities during inference. We’ve covered multiple recent multimodal image generation models such as OmniGen2, BAGEL, Ming-lite-omni, ICEdit, etc. In this article, we will cover Ovis-U1, an open-source 3-billion parameter model released by the Alibaba Ovis team, with capabilities that span across understanding multimodal inputs, generating images from text, and editing uploaded images.

• Ovis-U1 is an open-source 3-billion parameter multimodal LLM from Alibaba.
• Capabilities include multimodal understanding, text-to-image generation, and image editing.
• The model was trained on a diverse mix of datasets for various tasks (linked below).
• The model can be implemented on a DigitalOcean GPU Droplet or tested on HuggingFace Spaces.

Let’s take a look at the data used to train the model.

In the paper’s conclusion, they acknowledge “that Ovis-U1 currently lacks a reinforcement learning stage, which has proven crucial for large model optimization. Developing effective methods to align unified multimodal models with human preferences remains an important open research question in this domain.” We recently covered MMADA which introduces UniGRPO and are curious if there’s an application here. Let us know what you think in the comments.

Now that we’ve went over the model architecture and training process, let’s run the model on DigitalOcean.

Begin by spinning up a GPU Droplet. Once that’s completed, clone the repo into and install the required packages. You can do this using the following shell commands in the terminal. Alternatively, you can also try out the model on HuggingFace Spaces here.

# Install git-lfs for handling large files
apt install git-lfs

# Clone the Ovis-U1-3B repository from HuggingFace Spaces
git-lfs clone https://huggingface.co/spaces/AIDC-AI/Ovis-U1-3B

# Change directory into the cloned repository
cd Ovis-U1-3B

# Install pip for Python package management
apt install python3-pip

# Install required Python packages from requirements.txt
pip install -r requirements.txt

# Install additional Python packages for wheel and spaces
pip install wheel spaces

# Install PyTorch with CUDA 12.8 support and upgrade existing installations
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128 --upgrade

# Install xformers for optimized transformer operations
pip install -U xformers

# Install flash_attn for attention mechanism optimization
pip install flash_attn==2.7.4.post1

# Run the main application script
python app.py

We’re very excited about MLLMs. The datasets researchers decide to leverage, architectural modification, and how those translate to incremental improvements in capabilities is fascinating. We encourage you to test the model out. How are you using multimodal models and what use cases do you care about?

Learn more about DigitalOcean’s AI offerings. We have GPU Droplets you can spin up to train your models/run inference!