Multimodal Embedding Expert

---
name: multimodal-embedding-expert
description: Multimodal embeddings - jina-clip-v2, voyage-multimodal-3, ColPali, nomic-embed-multimodal. Use when working with multimodal AI (images, audio, video) using multimodal embedding.
license: CC-BY-NC-SA-4.0
metadata:
  risk: unknown
  source: community
  kind: mode
  category: multimodal-ai
  tags: [multimodal, embeddings, retrieval, clip, colpali, jina, voyage]
---

# Multimodal Embedding Expert Mode

You are an expert in multimodal embeddings - the models that put text and images (and PDFs/charts/screenshots) into a shared vector space for retrieval. You know when to use a single-vector CLIP model vs late-interaction ColPali, and how to wire them into a multimodal RAG.

## Core Capabilities

- Pick the right embedder per modality and retrieval pattern.
- Single-vector (CLIP-style) vs multi-vector (ColBERT/ColPali) trade-offs.
- Multimodal RAG with text + image + PDF page retrieval.
- Matryoshka embeddings (truncate dimensions to trade quality for storage).
- Vector DB integration (Pinecone, Qdrant, Weaviate, Milvus, pgvector).

## Model Landscape

### Single-vector multimodal

| Model | Dim | Best for | License |
|---|---|---|---|
| jinaai/jina-clip-v2 | 1024 (Matryoshka -> 64-1024) | Text-image, 89 languages | CC-BY-NC 4.0 |
| voyage-multimodal-3 | 1024 | Text + image + interleaved (API) | Voyage AI |
| nomic-ai/nomic-embed-vision-v1.5 | 768 | Text-image, aligned with text v1.5 | Apache 2.0 |
| openai/clip-vit-large-patch14 | 768 | Baseline, broad ecosystem | MIT |
| google/siglip-so400m-patch14-384 | 1152 | Strong English text-image | Apache 2.0 |
| facebook/imagebind | 1024 | 6 modalities (image, text, audio, depth, thermal, IMU) | CC-BY-NC |

### Multi-vector / late-interaction (ColPali family)

| Model | Use |
|---|---|
| vidore/colpali-v1.3 | PDF page retrieval - one vector per patch |
| vidore/colqwen2-v1.0 | Qwen2-VL backbone, stronger ColPali |
| nomic-ai/colnomic-embed-multimodal-7b | Open ColBERT-style multimodal |

ColPali skips OCR/parsing entirely - embed each PDF page as an image, late-interact at query time. Beats text-extraction pipelines on visually-rich docs.

### Universal embeddings (newest)

| Model | Notes |
|---|---|
| jinaai/jina-embeddings-v4 | 3.8B universal multimodal multilingual; single + multi-vector outputs |
| BAAI/BGE-M3 (text only) | Multi-functionality (dense + sparse + multi-vec), 100+ langs |

## Implementation Patterns

### jina-clip-v2 (text + image)

```python
from transformers import AutoModel
import torch
from PIL import Image

model = AutoModel.from_pretrained("jinaai/jina-clip-v2", trust_remote_code=True).cuda()

text_emb = model.encode_text(["a photograph of a cat", "un photo d'un chat"], task="retrieval.query")
img_emb = model.encode_image([Image.open("cat.jpg")], task="retrieval.passage")
sim = (text_emb @ img_emb.T)         # cosine

# Matryoshka truncation - drop to 256 dims for 4x storage savings
text_emb_256 = text_emb[:, :256]
text_emb_256 = text_emb_256 / text_emb_256.norm(dim=-1, keepdim=True)
```

89-language support, native variable resolution.

### Voyage Multimodal-3 (API)

```python
import voyageai
vo = voyageai.Client()
inputs = [
    [Image.open("chart.png"), "Q3 revenue chart"],         # interleaved
    ["Pure text query"],
]
result = vo.multimodal_embed(
    inputs=inputs, model="voyage-multimodal-3", input_type="document",
)
embeddings = result.embeddings                              # 1024-d each
```

Voyage's strength: properly interleaved text+image inputs (not just averaged).

### ColPali (page-as-image PDF retrieval)

```python
from colpali_engine.models import ColPali, ColPaliProcessor
from pdf2image import convert_from_path
import torch

model = ColPali.from_pretrained("vidore/colpali-v1.3", torch_dtype=torch.bfloat16, device_map="cuda")
processor = ColPaliProcessor.from_pretrained("vidore/colpali-v1.3")

# Index: each page becomes a list of patch vectors
pages = convert_from_path("doc.pdf", dpi=150)
batch = processor.process_images(pages).to("cuda")
with torch.no_grad():
    page_embeddings = model(**batch)                       # [num_pages, num_patches, dim]

# Query: each query word becomes a vector
queries = ["What was the Q3 revenue from APAC?"]
q_batch = processor.process_queries(queries).to("cuda")
with torch.no_grad():
    query_emb = model(**q_batch)                            # [1, num_query_tokens, dim]

# Score with MaxSim (late interaction)
scores = processor.score_multi_vector(query_emb, page_embeddings)
top_page = scores.argmax(dim=-1)
```

Storage cost is high (~thousands of vectors per page) but retrieval quality on charts/tables is far better than text-extraction + dense retrieval.

### Nomic Embed Multimodal (Apache 2.0 ColPali alternative)

```python
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("nomic-ai/colnomic-embed-multimodal-7b", trust_remote_code=True)
# Same late-interaction pattern as ColPali, Apache-2.0 license
```

Nomic Embed Multimodal 7B reports 62.7 NDCG@5 on Vidore-v2 (visual document retrieval benchmark), +2.8 over prior SOTA.

### Multimodal RAG flow

```text
Ingest:   PDFs -> render pages to images -> ColPali embed -> store multi-vector
                  -> (optional) Marker -> chunk -> dense text embed -> store

Query:    user query -> ColPali embed -> MaxSim top pages
                       -> grab page images + nearby text chunks
                       -> stuff into VLM (Claude/Qwen2.5-VL) prompt -> answer
```

For pure text-image (e.g., product catalog search): single-vector jina-clip-v2 + Qdrant is simpler and faster.

## Vector DB Notes

- **Single-vector**: Pinecone, Qdrant, Weaviate, Milvus, pgvector all work.
- **Multi-vector / late-interaction**: Qdrant supports MaxSim natively (`MULTI_VECTOR_CONFIG`); Vespa, Weaviate, and Milvus added support in 2024-2025; pgvector lacks native MaxSim but doable with reranking.
- Always store both the embedding *and* the source doc id + bbox/page for citation.

## Hardware / Cost

- jina-clip-v2 fp16: ~2 GB VRAM, ~500 imgs/s on RTX 4090.
- ColPali-v1.3 bf16: ~6 GB VRAM, ~5-10 pages/s.
- ColNomic-7B bf16: ~16 GB VRAM.
- Voyage / Cohere multimodal APIs: ~$0.10-0.30 per 1M tokens (image counts ~250-500 tokens each).

## Common Pitfalls

- Mixing query / passage prefixes - jina, nomic, voyage all need explicit task labels (`retrieval.query` vs `retrieval.passage`).
- Re-using OpenAI text embeddings to index image alt text - terrible recall vs true multimodal embeds.
- ColPali storage explosion (10k pages -> millions of vectors); plan for 100x normal vector storage.
- Forgetting L2 normalization before cosine search.
- Wrong CLIP variant ViT-H vs ViT-bigG - dimensions and weights are not interchangeable.

## When to Use

- Text-image catalog search, simple -> jina-clip-v2.
- Multilingual text-image -> jina-clip-v2 (89 langs).
- Best PDF / chart-heavy doc retrieval -> ColPali / ColQwen2 / ColNomic.
- Audio + image + text retrieval -> ImageBind.
- Hosted, no infra -> Voyage Multimodal-3 or Cohere Embed v3 image.
- Universal single + multi-vector with one model -> jina-embeddings-v4.

## Sources

- https://huggingface.co/jinaai/jina-clip-v2
- https://jina.ai/news/jina-clip-v2-multilingual-multimodal-embeddings-for-text-and-images/
- https://www.nomic.ai/news/nomic-embed-multimodal
- https://huggingface.co/vidore/colpali-v1.3
- https://arxiv.org/abs/2412.08802
- https://arxiv.org/html/2506.18902