You take a photo of a chair you like at a friend's house. You open an app, upload the photo, and add "but in dark wood." The app shows you chairs that look like the one in your photo but in dark wood finishes.
That's multimodal search. The query combines two different types of input (an image and text), and the system understands both together to find what you want.
Traditional search works within a single type of data. You type words, you get results that match those words. Image search works the same way but with photos. Multimodal search breaks that boundary. You can search across data types, and the query itself can be a mix of them.
What "Multimodal" Actually Means
A modality is a type of data. Text is one modality. Images are another. Audio, video, 3D models, those are all separate modalities.
Multimodal search means two things:
1. Cross-modal retrieval. Your query is one modality and the results are another. Type "sunset over ocean" and get back photos. Upload a photo of a dress and get back text descriptions of similar products. The query type and the result type are different.
2. Mixed-modal queries. Your query combines multiple modalities. Upload a photo AND add text to refine it. "This chair but in blue." "This song but more upbeat." The system understands both inputs together.
Most image search APIs already handle the first part. You can search images by text description (text-to-image) or search images by uploading another image (image-to-image). That's cross-modal retrieval.
The second part, combining modalities in a single query, is newer and harder. Google Multisearch was one of the first consumer products to offer this (take a photo, add text, search).
How It Works Under the Hood
The core idea is a shared embedding space. Different types of data get converted into vectors (arrays of numbers) that live in the same mathematical space. Similar meaning produces similar vectors, regardless of whether the input was text or an image.
Here's the process:
- A model like CLIP, SigLIP, or ImageBind encodes both text and images into vectors of the same dimensions
- "A golden retriever playing in the snow" becomes a vector of, say, 1024 numbers
- A photo of a golden retriever in snow becomes a vector of 1024 numbers that's very close to the text vector
- Search is just finding the nearest vectors in the database to your query vector
This is why you can type a text description and find matching photos. The text and the image end up near each other in vector space because they mean the same thing.
The models that make this work:
| Model | Modalities | Dimensions | Notes |
|---|---|---|---|
| CLIP (OpenAI) | Text + Image | 512-768 | The original. Most widely used. |
| OpenCLIP | Text + Image | 768-1024 | Open-source CLIP trained on larger data. |
| SigLIP (Google) | Text + Image | 768-1152 | Better accuracy than CLIP on most benchmarks. |
| ImageBind (Meta) | Text + Image + Audio + Video + Depth + Thermal | 1024 | Six modalities in one space. |
| Gemini Embeddings (Google) | Text + Image + Video | varies | Available through Vertex AI. |
CLIP and its variants are the workhorses for most production systems. ImageBind is interesting for research but less commonly deployed.
Multimodal vs Semantic vs Image Search
These terms overlap and people use them interchangeably, which causes confusion. Here's the actual difference:
Keyword search matches exact words. "Running shoes" finds documents containing "running" and "shoes." Fast, predictable, breaks when the user's words don't match your data.
Semantic search matches meaning but stays within one modality. "Affordable running shoes" finds "budget sneakers for jogging" because the meaning is similar. Text in, text out.
Image search works with images specifically. Upload a photo, find visually similar photos. Or describe what you want and get matching images. Cross-modal, but limited to text and images.
Multimodal search is the general case. Any combination of modalities as input, any modality as output. Text, images, audio, video, mixed queries. Image search is a subset of multimodal search.
In practice, most developers who say they need "multimodal search" actually need text-to-image and image-to-image search. True multi-input queries (photo + text together) are less common in production apps today, but that's changing fast.
Where It's Used in Production
Google Lens / Multisearch. The biggest example. Users point their camera at something, optionally add text ("in green"), and get shopping results, translations, or information. Google processes over 12 billion visual searches per month through Lens.
Amazon. The camera icon in the Amazon app. Take a photo of anything and Amazon finds matching products in their catalog. It's how a lot of people shop for things they can't describe in words.
Pinterest. Their visual search handles 600+ million visual queries per month. Users crop a section of a pin and find similar items. It powers their shopping and discovery features.
E-commerce product search. A customer types "warm jacket for hiking" and gets matching products even though no product title contains those exact words. Or they upload a photo of a jacket they saw on someone and find similar ones in the catalog.
Stock photography. Designers search for images by description ("team meeting in modern office, diverse, natural lighting") or by uploading a reference image and finding similar compositions.
Medical imaging. Radiologists search for similar scans across databases using both the image and clinical text descriptions to find relevant cases.
Building Multimodal Search
There are two approaches: build the pipeline yourself or use a managed API.
The DIY Approach
You need:
- An embedding model (CLIP or OpenCLIP) running on a GPU server
- A vector database (Pinecone, Qdrant, Weaviate, pgvector) to store and search the embeddings
- An ingestion pipeline that embeds every piece of content when it's added
- A query pipeline that embeds the search query and finds nearest neighbors
- A sync mechanism to keep your vectors updated when source data changes
This gives you full control over the model, the search quality, and the infrastructure. It also means you're operating GPU servers, managing an embedding pipeline, and maintaining a vector database. For a detailed cost breakdown of this approach, see our post on what it costs to search 1M images in production.
The API Approach
Managed search APIs handle the embedding, storage, and retrieval for you. You send in your data (text, images, or both) and search it. The API generates embeddings internally, indexes them, and runs the similarity search.
With Vecstore, the workflow looks like:
- Insert your content (images, text, or both) via API
- Search by text description, by uploading an image, or both
- Get ranked results with similarity scores
No GPU servers, no embedding pipeline, no vector database to manage. The tradeoff is less control over the model layer.
What's Coming Next
Multimodal search is moving fast. A few things to watch:
Video search. Searching inside videos by describing a scene or uploading a frame is becoming practical. Models like Twelve Labs' Embed and Google's Gemini can index video content for retrieval. This is still early but moving quickly.
Audio search. Searching music by humming, finding podcast segments by description, or matching sounds. Models like AudioCLIP and Meta's ImageBind handle audio embeddings in the same space as text and images.
Mixed-modal queries becoming standard. Google Multisearch normalized the "photo + text" query pattern. Expect more apps to adopt this. The underlying models (Gemini, GPT-4o) natively understand mixed inputs.
Better models, smaller footprint. Newer models like SigLIP achieve better accuracy than CLIP at similar or smaller sizes. Quantized models make it possible to run multimodal embeddings on edge devices.
When Do You Actually Need It?
Most developers searching for "multimodal search" need one of these:
- Text-to-image search - describe something, find matching images
- Image-to-image search - upload a photo, find similar ones
- Semantic text search - search by meaning, not keywords
If that's your use case, you don't need to build a full multimodal system. A search API that handles text and image embeddings covers it.
True multimodal search (combining multiple input types in a single query, searching across video and audio, etc.) is for more specialized applications. Get the basics working first, then add modalities as your product needs them.
Try multimodal search with Vecstore - text, image, face, and OCR search from a single API.
