Back to Curriculum
Intermediate•Modern AI
Multimodal AI & CLIP
Contrastive vision-text embeddings, InfoNCE loss, and joint cross-modal representation spaces.
Interactive Playground
Initializing Interactive Playground...
Research-Level Deep Dive & Equations
Multimodal AI models align text, image, audio, and video representations into a unified cross-modal embedding space . OpenAI's **CLIP** (Contrastive Language-Image Pre-training - Radford et al., 2021) demonstrated that training dual encoders on 400M image-caption pairs enables zero-shot vision classification matching supervised ResNets.
•Dual Encoder Topology:
- **Vision Encoder (ViT or ResNet)**: Encodes image into L2-normalized vector .
- **Text Encoder (Transformer)**: Encodes text caption into L2-normalized vector .
•Cosine Similarity Matrix:
Key Equations
PyTorch CLIP InfoNCE Loss Function Implementationpython
import torch
import torch.nn as nn
import torch.nn.functional as F
class CLIPContrastiveLoss(nn.Module):
def __init__(self, init_tau: float = 0.07):
super().__init__()
# Learnable log temperature parameter log(1/tau)
self.logit_scale = nn.Parameter(torch.tensor(torch.log(torch.tensor(1.0 / init_tau))))
def forward(self, vision_embeds: torch.Tensor, text_embeds: torch.Tensor):
# L2 Normalization
v_norm = F.normalize(vision_embeds, p=2, dim=-1)
t_norm = F.normalize(text_embeds, p=2, dim=-1)
# Cosine similarity matrix scaled by exp(logit_scale)
logit_scale = torch.exp(self.logit_scale)
logits_per_image = logit_scale * (v_norm @ t_norm.T) # [N, N]
logits_per_text = logits_per_image.T
# Ground truth diagonal targets
labels = torch.arange(vision_embeds.shape[0], device=vision_embeds.device)
# Symmetric Cross-Entropy Loss
loss_v = F.cross_entropy(logits_per_image, labels)
loss_t = F.cross_entropy(logits_per_text, labels)
return (loss_v + loss_t) / 2.0Test Your Knowledge
Check whether you have mastered this concept with a quick quiz.
Was this lesson helpful?
Your feedback helps us continuously improve the curriculum and interactive visualizations.