Debugging an In-Batch Contrastive Loss

HardML Coding

Reported in interviews at Exa

Problem statement

A two-tower retrieval model is trained with an in-batch contrastive (InfoNCE) loss. For a batch of $B$ pairs, z1 holds the query-side embeddings and z2 the document-side embeddings, both of shape [B, D], and row $i$ of z1 is paired with row $i$ of z2. Every other document in the batch is a negative for query $i$. A teammate wrote the loss below. Training crashes with a shape error at B = 256, D = 128; when they tried B = D = 128 the crash went away, but the loss curve looks wrong, and with temperature = 0.01 it turns into nan after a few hundred steps.

import torch
import torch.nn.functional as F

def info_nce(z1, z2, temperature=0.07, symmetric=False):
    sim = torch.matmul(z1, z2) / temperature
    target = torch.arange(z1.shape[1])
    log_p = sim - torch.log(torch.exp(sim).sum(dim=0, keepdim=True))
    loss = -log_p[target, target].mean()
    if symmetric:
        loss = 0.5 * (loss + F.cross_entropy(sim, target))
    return loss

Find every bug, say what each one does to shapes or values, and write a corrected version. The judge runs Python, so implement the fixed loss with numpy:

def contrastive_loss(z1: list, z2: list, temperature: float, symmetric: bool) -> float:
    ...

Define the score matrix $S_{ij} = (z_{1,i} \cdot z_{2,j}) / \tau$ using the raw dot product (the model does not L2-normalize). The one-directional loss is the mean over queries of $\log \sum_{j} e^{S_{ij}} - S_{ii}$. When symmetric is true, return the average of that loss and the same loss with documents as anchors (scored against every query in the batch).

Constraints

  • $1 \le B \le 64$, $1 \le D \le 64$; z1 and z2 have the same shape. $B$ and $D$ may or may not be equal.
  • Entries satisfy $|z| \le 50$; $0.01 \le \tau \le 2$.
  • The result must be finite; answers are compared with a relative tolerance of $10^{-6}$.

Example 1

z1 = [[1, 0], [0, 1], [1, 1]],  z2 = [[2, 0], [0, 2], [1, 1]],  temperature = 1.0,  symmetric = False
output: 0.637941...

$S$ has rows $[2, 0, 1]$, $[0, 2, 1]$, $[2, 2, 2]$; the per-query losses are $0.4076, 0.4076, \ln 3$, and their mean is 0.637941.

Example 2

same inputs, symmetric = True
output: 0.663753...

Example 3

z1 = [[30, 0], [0, 30]],  z2 = [[29, 2], [31, 0]],  temperature = 0.01,  symmetric = False
output: 6000.0

The scores reach $9.3 \times 10^4$, so exponentiating them directly overflows.

Show hint

Before reading the arithmetic, what shape should the score matrix have, and what does entry $(i, j)$ mean?

Relevant roles: ML Engineer, Deep Learning Engineer, Research Scientist and Applied Scientist.

Related free problems