InternVL — Open-Source Vision-Language Model Rivaling GPT-4o

Introduction

InternVL is a family of open-source vision-language models developed by Shanghai AI Laboratory that achieve competitive performance with proprietary models like GPT-4o on multimodal benchmarks. The models support image and video understanding, OCR, document analysis, and multi-turn visual dialogue.

What InternVL Does

• Performs visual question answering on images, charts, documents, and screenshots
• Extracts text from images using built-in OCR capabilities without external tools
• Supports multi-image and video understanding with temporal reasoning
• Provides multi-turn conversational interaction grounded in visual context
• Scales from 1B to 108B parameters to fit different hardware constraints

Architecture Overview

InternVL uses a vision encoder based on InternViT coupled with a large language model backbone through a pixel-shuffle connector. The vision encoder processes images at dynamic resolution by splitting them into tiles, extracting features from each tile independently. These visual tokens are projected into the language model's embedding space and concatenated with text tokens for joint reasoning.

Self-Hosting & Configuration

• Requires Python 3.9+ with PyTorch and Transformers library
• Models are hosted on Hugging Face and range from 2 GB to 200 GB in size
• Run inference on a single GPU for smaller variants (1B-8B) or multi-GPU for larger ones
• Supports quantization with BNB 4-bit and AWQ for reduced memory usage
• Compatible with vLLM and LMDeploy for high-throughput serving

Key Features

• Achieves top scores on OCRBench, MathVista, and DocVQA benchmarks among open models
• Dynamic resolution support processes images from 448 to 4096 pixels without fixed aspect ratio
• Bilingual support for English and Chinese across all model sizes
• Progressive training pipeline from vision pretraining to supervised fine-tuning
• Open weights and training recipes for full reproducibility

Comparison with Similar Tools

• GPT-4o — Proprietary with broader general knowledge; InternVL matches or exceeds it on specific vision benchmarks
• LLaVA — Pioneer open VLM but InternVL offers better OCR and document understanding
• CogVLM — Strong on visual grounding; InternVL has better multi-resolution handling
• Qwen-VL — Competitive alternative; InternVL provides more model size options

FAQ

Q: What GPU is needed to run InternVL? A: The 8B model runs on a single 24 GB GPU; the 2B model fits on 8 GB with quantization.

Q: Can InternVL process PDF documents? A: Yes, by rendering PDF pages as images, InternVL can extract and reason over document content.

Q: Does InternVL support video input? A: Yes, it samples frames from video and performs temporal reasoning across the frame sequence.

Q: Is InternVL suitable for production deployment? A: Yes, it can be served with vLLM or LMDeploy for high-throughput inference in production.

Sources

• https://github.com/OpenGVLab/InternVL
• https://internvl.github.io/