Contrastive vision-text embeddings, InfoNCE loss, and joint cross-modal representation spaces.
For a small example, two images and two captions form correct and incorrect pairs. Calculate a similarity matrix, increase diagonal scores relative to negatives, and inspect the effect of temperature.
Users search images with natural-language descriptions. Create image and text embeddings, evaluate recall across product slices, rerank hard cases, and handle near-duplicate items.
Initializing Interactive Playground...
Predict how one control changes the Multimodal AI & CLIP result.
The starter deliberately follows a tempting but flawed assumption: Using random negatives that are actually valid matches
Check: Your saved hypothesis predicts a direction, the experiment tests it, and the result changes or supports your explanation.
Explanations after each answer · 80% to pass · Multimodal AI & CLIP
Chapter 1 of 5 · Core
Multimodal AI models align text, image, audio, and video representations into a unified cross-modal embedding space . OpenAI's CLIP (Contrastive Language-Image Pre-training - Radford et al., 2021) demonstrated that training dual encoders on 400M image-caption pairs enables zero-shot vision classification matching supervised ResNets.
Key equations
import torch
import torch.nn as nn
import torch.nn.functional as F
class CLIPContrastiveLoss(nn.Module):
def __init__(self, init_tau: float = 0.07):
super().__init__()
# Learnable log temperature parameter log(1/tau)
self.logit_scale = nn.Parameter(torch.tensor(torch.log(torch.tensor(1.0 / init_tau))))
def forward(self, vision_embeds: torch.Tensor, text_embeds: torch.Tensor):
# L2 Normalization
v_norm = F.normalize(vision_embeds, p=2, dim=-1)
t_norm = F.normalize(text_embeds, p=2, dim=-1)
# Cosine similarity matrix scaled by exp(logit_scale)
logit_scale = torch.exp(self.logit_scale)
logits_per_image = logit_scale * (v_norm @ t_norm.T) # [N, N]
logits_per_text = logits_per_image.T
# Ground truth diagonal targets
labels = torch.arange(vision_embeds.shape[0], device=vision_embeds.device)
# Symmetric Cross-Entropy Loss
loss_v = F.cross_entropy(logits_per_image, labels)
loss_t = F.cross_entropy(logits_per_text, labels)
return (loss_v + loss_t) / 2.0Two free references selected for Multimodal AI & CLIP.
Primary sources and free technical reading for Multimodal AI & CLIP.