Cross-Document Attention Mask Construction

MediumML Coding

Problem statement

When training a language model, multiple short documents are often packed into a single sequence to maximize GPU utilization. To prevent tokens from one document from attending to tokens of an unrelated document, the attention mask must be constructed to enforce two rules at once: standard causal masking (a token cannot attend to future positions) AND document boundary masking (a token cannot attend to any position belonging to a different document, even if that position is earlier in the packed sequence).

Write a function cross_doc_attention_mask(token_ids, doc_ids) that builds such a mask.

The function takes:

  • token_ids (list of int): the packed token id sequence (length N). Not used to compute the mask itself, but provided to mirror a realistic training pipeline.
  • doc_ids (list of int): the document id assigned to each token (length N). Tokens with the same doc_id belong to the same document.

Return a boolean numpy array of shape (N, N) where entry (i, j) is True if and only if the query token at position i is allowed to attend to the key token at position j.

Example

Input:

token_ids = [10, 11, 20, 21]
doc_ids   = [0, 0, 1, 1]

Output:

[[1, 0, 0, 0], [1, 1, 0, 0], [0, 0, 1, 0], [0, 0, 1, 1]]

Reasoning: Positions 0 and 1 belong to document 0; positions 2 and 3 belong to document 1. Within each document attention is causal: position 1 may attend to position 0 and itself, position 3 may attend to position 2 and itself. Across documents (e.g., position 2 attending to position 0 or 1) attention is blocked, even though position 0 and 1 come earlier in the packed sequence.

Show hint

Which two separate conditions must both hold for a query to see a key in a packed sequence?

Relevant roles: LLM Engineer, ML Engineer, Deep Learning Engineer and Gen AI Engineer.

Related free problems