Image-Text-to-Text
Image-text-to-text exploded from a research curiosity to the dominant AI interface in under two years. GPT-4V (2023) proved multimodal LLMs could reason over images, Gemini 1.5 scaled to million-token contexts mixing text and vision, and Claude 3 showed that careful RLHF produces models that refuse to hallucinate about image content. MMMU and MMBench have become the standard evaluation gauntlet, but the real challenge is grounding — models still confabulate spatial relationships and struggle with fine-grained visual reasoning. This is the task that turned chatbots into visual assistants.
Image-text-to-text (vision-language) models accept images and text prompts to generate text — powering visual Q&A, document understanding, chart interpretation, and visual reasoning. This is the backbone task of multimodal AI, now dominated by frontier models like GPT-4o, Gemini, and Claude that treat vision as a first-class modality.
History
ViLBERT introduces dual-stream vision-language pretraining with co-attentional transformer layers
CLIP (OpenAI) demonstrates that contrastive vision-language pretraining creates powerful zero-shot classifiers from 400M image-text pairs
Flamingo (DeepMind) shows few-shot visual reasoning by interleaving frozen vision encoders with language models
GPT-4V launches multimodal reasoning at scale — first frontier model with production-grade vision understanding
LLaVA demonstrates that visual instruction tuning on synthetic data can match GPT-4V on many benchmarks with a 13B model
Claude 3.5 Sonnet, Gemini 1.5 Pro, and GPT-4o all achieve >60% on MMMU (expert-level multimodal reasoning)
InternVL2 and Qwen2-VL push open-source vision-language models to within 5% of proprietary frontier on most benchmarks
Gemini 2.0, GPT-4.5, and Claude 3.6 Opus compete on complex visual reasoning; open-source models like Qwen2.5-VL-72B close the gap further
How Image-Text-to-Text Works
Visual Encoding
Input images are processed through a vision encoder — typically a Vision Transformer (ViT) pretrained with SigLIP or CLIP objectives. The encoder produces a grid or sequence of patch embeddings representing visual features at multiple scales.
Resolution Handling
Modern VLMs support dynamic resolution: images are tiled into sub-images at native resolution (e.g., Qwen2-VL uses naive dynamic resolution, InternVL2 uses dynamic tiling up to 4K). This preserves fine-grained detail for OCR and small-object tasks.
Cross-modal Projection
Visual features are projected into the LLM's token embedding space via an adapter — MLP projector (LLaVA-style), cross-attention layers (Flamingo-style), or a perceiver/Q-Former that compresses visual tokens to a fixed budget.
Autoregressive Generation
The LLM processes the interleaved sequence of visual and text tokens, attending over both to generate text outputs. Multi-image and video understanding follow the same pattern with additional positional encodings.
Current Landscape
Vision-language models are the most mature multimodal category in 2025. The proprietary frontier (GPT-4o, Gemini 2.0, Claude 3.6) all score above 65% on MMMU and handle complex multi-step visual reasoning reliably. Open-source has closed the gap dramatically — Qwen2.5-VL-72B and InternVL2.5-78B match or exceed GPT-4o-mini across most benchmarks. The field has moved beyond simple VQA to challenging tasks like mathematical diagram reasoning (MathVista), expert-level exam questions (MMMU), and real-world visual grounding. Dynamic resolution handling and efficient visual token compression are the key architectural differentiators.
Key Challenges
Hallucination — models confidently describe objects, text, or relationships not present in the image; POPE and CHAIR metrics track this
Spatial and counting reasoning — models struggle with precise object counting, relative positioning, and spatial relationship questions
Fine-grained OCR — reading small, rotated, or stylized text in documents and natural scenes remains error-prone
Multi-image reasoning — comparing, contrasting, or tracking entities across multiple images degrades significantly vs. single-image tasks
Evaluation saturation — benchmarks like VQAv2 are nearly solved; newer benchmarks (MMMU, MathVista, RealWorldQA) better test true understanding
Quick Recommendations
Best overall
GPT-4o
Strongest all-around vision understanding — leads on MMMU, MathVista, and document comprehension with fast inference
Best for documents and charts
Claude 3.5 Sonnet
Exceptional at structured document understanding, table extraction, and following complex visual instructions
Best for long visual context
Gemini 2.0 Pro
1M token context window natively handles hundreds of images or long documents in a single prompt
Open source (large)
Qwen2.5-VL-72B
Matches GPT-4o-mini on most benchmarks; best open-weight VLM for production use with dynamic resolution support
Open source (small)
InternVL2.5-8B
Best performance-per-parameter in the 7-8B class; strong on OCR, charts, and visual reasoning
On-device
PaliGemma-2-3B
Google's 3B vision-language model optimized for edge deployment with solid OCR and captioning capabilities
What's Next
The next wave is agentic vision — models that can interact with GUIs, navigate websites, and operate software by seeing and clicking. Expect vision-language models to become the perception backbone of computer-use agents (Claude Computer Use, GPT-4o with tools). Video understanding will merge with image understanding into unified temporal-visual reasoning. Efficiency breakthroughs in visual token compression will enable real-time streaming visual understanding on edge devices.
Benchmarks & SOTA
MMBench
Comprehensive multimodal LLM evaluation across 20 ability dimensions
No results tracked yet
MMMU
Massive multi-discipline multimodal understanding across 30 subjects
No results tracked yet
MMStar
Vision-indispensable multimodal benchmark filtering text-solvable questions
No results tracked yet
Related Tasks
Audio-Text-to-Text
Audio-text-to-text is the backbone of voice assistants that actually understand context — models that jointly process speech and text to generate grounded responses. Whisper (2022) cracked robust transcription, but the real leap came when Gemini 1.5 and GPT-4o (2024) began reasoning natively over audio tokens alongside text, eliminating the lossy ASR-then-LLM pipeline. The key challenges are handling overlapping speakers, noisy environments, and preserving prosodic cues like sarcasm or hesitation that pure transcription destroys. Benchmarks like SUPERB and Dynamic-SUPERB are expanding, but real-world spoken dialogue understanding remains far ahead of what leaderboards capture.
Image-Text-to-Image
Image-text-to-image covers instruction-guided image editing — taking a source image plus a text command and producing a modified result. InstructPix2Pix (2023) demonstrated this could work zero-shot, and subsequent models like DALL-E 3's inpainting, Ideogram, and Stable Diffusion's img2img pipelines made it practical. The core difficulty is surgical precision: users want "change the dress to red" without altering the face, background, or lighting, which requires disentangling content from style at a level current architectures still fumble. Benchmarks are fragmented across editing fidelity, instruction-following, and identity preservation, making unified comparison difficult.
Image-Text-to-Video
Image-text-to-video is generative AI's hardest unsolved frontier — animating a still image according to a text prompt while maintaining temporal coherence and physical plausibility. Stable Video Diffusion (2023) and Runway Gen-2 showed early promise, Sora (2024) raised the bar dramatically with minute-long physically consistent clips, and Kling and Veo 2 pushed quality further. The fundamental challenge is that video generation requires implicit world models: objects must persist, lighting must evolve consistently, and motion must obey approximate physics across dozens of frames. Evaluation is still largely human-judged, with FVD and CLIP-temporal scores poorly correlating with perceived quality.
Visual Question Answering
Visual question answering (VQA) is the original multimodal reasoning task — given an image and a natural language question, produce the correct answer. VQAv2 (2017) defined the field, but modern benchmarks like GQA, OK-VQA, and TextVQA have pushed toward compositional reasoning, external knowledge, and OCR-dependent understanding. The task was largely "solved" in its classic form once multimodal LLMs arrived, with GPT-4V and Gemini saturating standard benchmarks, but adversarial and compositional variants still expose systematic failures in spatial reasoning and counting. VQA's legacy is establishing that vision-language models need more than pattern matching — they need genuine visual understanding.
Something wrong or missing?
Help keep Image-Text-to-Text benchmarks accurate. Report outdated results, missing benchmarks, or errors.