Advanced Embeddings from Scratch — Part 1: Why Do We Need Multimodal Embeddings?

In our original Understanding LLM Embeddings from Scratch series, we learned how text becomes vectors, how those vectors learn semantic relationships, and how sentence embedding models represent entire sentences.

Now we're going to extend that understanding beyond text. Our fundamental question is:

How can a computer recognize that a sentence, an image, an audio recording, and a video are describing the same concept when they contain completely different kinds of data?

We’ll begin with the problem, discover why traditional text embeddings aren't enough, construct a shared embedding space mathematically, and finally implement a tiny multimodal retrieval system in PyTorch.

The problem: Meaning exists beyond text

Imagine you're building a search engine for a collection of one million images.

A user enters: "A golden retriever running across a grassy field."

Your collection includes this photograph:

As a human, you can immediately recognize that the sentence and image are related. But consider what the computer actually receives.

text = "A dog running on grass"
token_ids = [101, 1037, 3899, 2770, 2006, 5568, 102]

The image, however, is represented using pixel values:

image.shape
# [3, 224, 224]

Here, the three channels represent RGB, and the image has a height and width of 224 pixels. These inputs are fundamentally different.

Property	Text	Image
Raw representation	Characters or encoded bytes	Pixel intensity values
Model-ready representation	Token IDs	Image tensors or patches
Primary structure	Token sequence	Spatial arrangement
Meaning	Language and concepts	Visual objects, scenes, relationships

We cannot directly calculate a meaningful cosine similarity between arbitrary token IDs and image pixel values. Yet we want our search engine to recognize that they describe related content. That's the first reason multimodal embeddings exist.

Why can't we just use our existing sentence embedding model?

Remember Module 7 of our earlier series. We used a model such as all-MiniLM-L6-v2:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "sentence-transformers/all-MiniLM-L6-v2"
)

text = "A dog running on grass"

embedding = model.encode(text)

print(embedding.shape)
# (384,)

We obtain a 384-dimensional text embedding. Now suppose we want to pass a JPG image to that same text embedding model. There's a problem.

The model was designed to consume tokenized text, not raw image pixels. It doesn't have a visual processing pathway that understands image structure.

We could first generate a caption for the image, then embed the caption:

This is a valid approach, and it can be useful. But what if the image contains an important detail that the captioning model omits? For example, the dog might be holding a red ball while jumping over a small fence. The caption "A dog outdoors" loses that information.

The same issue occurs with audio. A recording might contain someone speaking, a car horn, rainfall, and distant traffic. Converting it only to a speech transcript can discard important non-speech information.

This motivates models that learn representations directly from different modalities, instead of forcing everything through a textual description.

What is a modality?

A modality is a particular type or channel of information. In our context:

Modality	Example input	What the model needs to capture
Text	"A dog is barking"	Language and meaning
Image	Photograph of a dog	Objects, appearance, spatial relationships
Audio	Recording of barking	Sound patterns, speech, acoustic events
Video	Dog running and barking	Visual events and temporal relationships

A multimodal embedding model aims to encode inputs from two or more modalities into representations that can be compared or combined.

The critical requirement for cross-modal similarity is that the outputs live in a compatible representation space.

So what does that actually mean?

What is a shared embedding space: The vector-space analogy

Imagine a map of a city. Every restaurant, hospital, and school has its own geographical coordinates. You can calculate the distance between two locations because their coordinates use the same reference system.

Now imagine one map uses latitude and longitude, while another uses a completely unrelated coordinate system. Comparing their raw numbers would not tell you how far apart the locations are. We need a common coordinate system.

Multimodal embeddings work toward a similar goal, except that the coordinates represent learned semantic relationships, not geographical positions.

For example, a text description of a dog and an image of a dog should ideally have nearby embeddings.

A description of a rainy street and an image of rainfall should ideally have nearby embeddings.

The underlying media types can be different. Their semantic representations should be compatible.

Let's construct a shared space manually

Suppose we're interested in just two concepts: Dogs and Rain

We'll use two-dimensional vectors so we can calculate everything by hand.

Imagine we've learned:

$$e_{\text{text,dog}}=[1,0]$$

$$e_{\text{text,rain}}=[0,1]$$

Now, let's assume our image encoder has learned the exact same semantic coordinate system:

$$e_{\text{image,dog}}=[1,0]$$

$$e_{\text{image,rain}}=[0,1]$$

Let's calculate cosine similarity.

Recall:

$$\cos(A,B)=\frac{A\cdot B}{|A||B|}$$

For dog text versus dog image:

$$\cos([1,0],[1,0])=1$$

For dog text versus rain image:

$$\cos([1,0],[0,1])=0$$

Therefore:

$$\boxed{\operatorname{sim}(\text{dog text},\text{dog image})=1}$$

$$\boxed{\operatorname{sim}(\text{dog text},\text{rain image})=0}$$

A retrieval system using cosine similarity would rank the dog image above the rain image for the dog query. That's the essential idea behind cross-modal retrieval.

Why matching dimensions is not enough: Suppose both encoders produce 2D vectors. Is that sufficient?

No. This is one of the most important ideas in multimodal learning. Imagine we independently train a text encoder and an image encoder.

The text encoder learns:

$$t_{\text{dog}}=[1,0]$$

$$t_{\text{rain}}=[0,1]$$

But the image encoder learns:

$$i_{\text{dog}}=[0,1]$$

$$i_{\text{rain}}=[-1,0]$$

Both encoders produce two-dimensional vectors. But calculate the similarities:

$$\cos(t_{\text{dog}},i_{\text{dog}})=0$$

$$\cos(t_{\text{dog}},i_{\text{rain}})=-1$$

For the dog text query, neither image has a positive similarity. The image of the dog is not close to the text embedding of the dog. Why? Because the two encoders learned different coordinate systems. Sharing the same number of dimensions doesn't mean the individual dimensions represent compatible directions.

This is why we need alignment.

What does alignment mean mathematically?

Let's see whether we can transform our image vectors into the text encoder's coordinate system.

Suppose we learn a transformation matrix:

$$W= \begin{bmatrix} 0&-1\ 1&0 \end{bmatrix}$$

Using row-vector notation, our projected image vector is:

$$i'=iW$$

For the dog image:

$$[0,1] \begin{bmatrix} 0&-1\ 1&0 \end{bmatrix} =[1,0]$$

For the rain image:

$$[-1,0] \begin{bmatrix} 0&-1\ 1&0 \end{bmatrix} =[0,1]$$

Now:

$$i'_{\text{dog}}=[1,0]$$

$$i'_{\text{rain}}=[0,1]$$

The image vectors have been aligned with their corresponding text vectors.

PyTorch: Align two embedding spaces manually

import torch
import torch.nn.functional as F

# Text embeddings
text = torch.tensor([
    [1.0, 0.0],  # dog
    [0.0, 1.0]   # rain
])

# Image embeddings in an incompatible coordinate system
images = torch.tensor([
    [ 0.0, 1.0],  # dog
    [-1.0, 0.0]   # rain
])

# Compare before alignment
before = text @ images.T
print(before)

# Matrix that aligns image vectors with text vectors
W = torch.tensor([
    [0.0, -1.0],
    [1.0,  0.0]
])

aligned_images = images @ W

# All vectors have unit length in this example
after = text @ aligned_images.T

print(after)

Expected output:

# Before alignment
tensor([[ 0., -1.],
        [ 1.,  0.]])

# After alignment
tensor([[1., 0.],
        [0., 1.]])

The diagonal now contains the correct matches. In real multimodal embedding models, we don't normally construct this alignment matrix manually. Instead, the model learns alignment from training examples.

And in practice, alignment may involve updating the text encoder, image encoder, projection layers, or a shared Transformer, not just a single matrix.

How does a multimodal model learn alignment? Training with image-text pairs

Imagine our training dataset contains these pairs:

Text	Associated image
"A dog running on grass"	Photograph of a running dog
"Rain falling on a road"	Photograph of a rainy street
"A red sports car"	Photograph of a red car

We have three related text-image pairs. For a given text input, its matching image is a positive example. The other images in the batch can serve as negative candidates, provided they aren't also relevant.

This should sound familiar. In Module 7, we used contrastive learning to train sentence embeddings. The same mathematical idea can align representations across modalities.

Calculate the similarity matrix

Suppose our model produces three text embeddings:

$$T\in\mathbb{R}^{3\times d}$$

And three image embeddings:

$$I\in\mathbb{R}^{3\times d}$$

Assume the embeddings are L2-normalized.

We can calculate every text-image cosine similarity with:

$$\boxed{S=TI^\top}$$

The result has shape:

$$S:[3,3]$$

Imagine our similarity matrix looks like this:

Our goal is to make the correct pairs score higher than competing pairs. For example, the first row should rank the dog image above the rain and car images.

Bring back InfoNCE

For text $t_i$ and its corresponding image $i_i$:

$$\boxed{ L_i=-\log \frac{\exp(s(t_i,i_i)/\tau)} {\sum_{j=1}^{B}\exp(s(t_i,i_j)/\tau)} }$$

This is the familiar InfoNCE-style contrastive objective.

Here:

$s(t_i,i_j)$ is the text-image cosine similarity.
$B$ is the number of image candidates in the batch.
$\tau$ is the temperature.
$i_i$ is the positive image for text $t_i$.

The gradients from this loss can update the encoders and projection layers. This encourages matching texts and images to occupy compatible regions of the embedding space.

One influential example is CLIP, which trains image and text encoders by learning to identify matching image-text pairs. The original work used 400 million image-text pairs.

We'll study CLIP's architecture and symmetric contrastive loss in depth later.

For now, the important takeaway is:

Multimodal alignment is learned through training. It doesn't happen automatically because two encoders produce vectors of the same size.

Let's train a shared embedding space in PyTorch

Can we learn alignment instead of manually defining it?

Yes. Let's build a tiny experiment.

Our objective is to train a projection layer that transforms image representations into a coordinate system compatible with our text representations.

For simplicity, we'll use three-dimensional vectors and just three concepts:

dog, rain, car

This will demonstrate the core mathematics without requiring a pretrained vision model.

Initialize our representations

import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(42)

labels = ["dog", "rain", "car"]

# Text embeddings: [3, 3]
text_vectors = torch.eye(3)

# Image vectors in a different coordinate system
image_vectors = text_vectors[[1, 2, 0]]

print("Text:\n", text_vectors)
print("Images:\n", image_vectors)

Initially, our image vectors are rotated or permuted relative to the text representation space. Calculate their similarities:

similarities = text_vectors @ image_vectors.T

print(similarities)

Output:

tensor([[0., 0., 1.],
        [1., 0., 0.],
        [0., 1., 0.]])

Notice the problem.

For the dog text embedding, the highest similarity is with the car image embedding!

Our coordinate systems are incompatible.

Introduce a trainable projection

We'll create a linear transformation:

$$e_{\text{image,aligned}}=We_{\text{image}}$$

Using a learnable matrix:

$$W\in\mathbb{R}^{3\times3}$$

projection = nn.Linear(
    in_features=3,
    out_features=3,
    bias=False
)

# Start with an identity transformation
with torch.no_grad():
    projection.weight.copy_(torch.eye(3))

optimizer = torch.optim.Adam(
    projection.parameters(),
    lr=0.1
)

Now the projection parameters can be updated using gradient descent.

Train using contrastive loss

loss_history = []

for step in range(201):

    # Project image vectors
    projected_images = projection(image_vectors)

    # Normalize both modalities
    text_normalized = F.normalize(
        text_vectors, dim=-1
    )

    image_normalized = F.normalize(
        projected_images, dim=-1
    )

    # Text-image cosine similarity matrix
    similarities = text_normalized @ image_normalized.T

    # Correct pairs are on the diagonal
    targets = torch.arange(3)

    # Contrastive loss
    loss = F.cross_entropy(
        similarities / 0.1,
        targets
    )

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    loss_history.append(loss.item())

What's happening here?

The projection begins with an identity transformation. Initially, the text-image comparisons are incorrect.

The contrastive loss penalizes the model when the matching image doesn't receive a high relative score.

Gradient descent updates the projection matrix, changing the position of the image embeddings in the shared space.

Check what the model learned

with torch.no_grad():

    aligned_images = F.normalize(
        projection(image_vectors),
        dim=-1
    )

    final_similarity = text_vectors @ aligned_images.T

    print(final_similarity)

    for i, label in enumerate(labels):
        predicted_idx = final_similarity[i].argmax().item()

        print(
            f"Query: {label}, "
            f"Retrieved: {labels[predicted_idx]}"
        )

tensor([[ 0.7620, -0.6148, -0.2033],
        [-0.2033,  0.7620, -0.6148],
        [-0.6148, -0.2033,  0.7620]])
Query: dog, Retrieved: dog
Query: rain, Retrieved: rain
Query: car, Retrieved: car

For this particular toy setup, one run produced a diagonal similarity of approximately 0.76, with lower scores for competing pairs.

The model doesn't need every positive cosine similarity to equal 1. It needs the correct pairs to rank above the competing pairs.

Visualize the loss

import matplotlib.pyplot as plt

plt.figure(figsize=(8, 4))

plt.plot(loss_history)

plt.xlabel("Training step")
plt.ylabel("Contrastive loss")
plt.title("Learning Cross-Modal Alignment")

plt.grid(alpha=0.3)
plt.show()

Important limitation: This model has learned a mapping between three synthetic examples. It doesn't understand real images or generalize to unseen concepts. A production multimodal embedding model needs encoders trained on diverse real-world data.

But the underlying mechanism is the same: learn representations that produce useful cross-modal similarity scores.

How does this scale to real images, audio, and video?

Different modalities require different input-processing mechanisms. Our toy model started with ready-made three-dimensional vectors. But real models must first convert raw inputs into meaningful representations.

Let's compare them.

Input	Raw representation	Typical processing approach
Text	Sequence of tokens	Token embeddings and Transformer
Image	Pixel tensor `[C,H,W]`	Vision encoder, often patch-based
Audio	Waveform samples	Audio encoder, often using spectral or learned acoustic features
Video	Ordered frames, optionally with audio	Visual encoder plus temporal processing

These modality-specific encoders produce representations that can eventually be mapped into compatible embeddings.

A simplified multimodal architecture looks like this:

The diagram is conceptual. Not all multimodal embedding models have four independent encoders followed by one projection layer. Some models use separate text and image encoders, like CLIP.

Others, such as EmbeddingGemma 2, use modality-specific processing pathways with a shared backbone. Google's documentation describes its ability to represent text, images, audio, and video in a compatible 768-dimensional space.

Does a shared embedding space mean the model understands everything?

Not necessarily.

Imagine this query:

"A man says that it is raining."

An audio recording of someone saying those words and an audio recording of actual rainfall are different. Both might relate to the concept of rain, but one is speech and the other is an environmental sound.

Similarly, an image of a dog standing still cannot necessarily tell us whether the dog was barking a moment earlier.

A model's ability to capture such distinctions depends on its training data, architecture, and objective. A shared embedding space makes cross-modal comparisons possible, but it does not guarantee perfect semantic understanding.

What can we build with multimodal embeddings?

Application	Example query	Data searched
Text-to-image retrieval	"A blue car parked near a tree"	Photos
Text-to-audio retrieval	"A dog barking loudly"	Sound recordings
Text-to-video retrieval	"A presenter showing a GPU chart"	Video clips
Image-to-image retrieval	Upload a picture of a shoe	Product images
Multimodal RAG	"Find the slide showing memory bandwidth"	Text, diagrams, slides, video

For a real-time meeting intelligence system, multimodal retrieval could be particularly useful.

The most important distinction is:

Multimodal embedding models don't merely convert different data types into vectors. They learn representations in which relationships between different data types become meaningful.

Quick Mental Model

We create training examples containing related information from different modalities, such as text-image pairs. Each modality is processed by an appropriate encoder, which may contain Transformer layers, and produces a fixed-dimensional embedding.

These embeddings are projected into a compatible dimensional space when necessary. We then calculate pairwise similarities and use contrastive loss to encourage matching pairs to score higher than competing pairs.

Through backpropagation, the trainable encoder and projection weights are updated so the different modalities gradually become aligned in a shared semantic space.

And that brings us to the central question of our next module.

Next: Module 2 — How Do Images Become Embeddings?

We've treated the image encoder as a black box so far. But what actually happens inside it? We will learn those.