Cross-Document Attention Mask Construction
Problem statement
When training a language model, multiple short documents are often packed into a single sequence to maximize GPU utilization. To prevent tokens from one document from attending to tokens of an unrelated document, the attention mask must be constructed to enforce two rules at once: standard causal masking (a token cannot attend to future positions) AND document boundary masking (a token cannot attend to any position belonging to a different document, even if that position is earlier in the packed sequence).
Write a function cross_doc_attention_mask(token_ids, doc_ids) that builds such a mask.
The function takes:
token_ids(list of int): the packed token id sequence (lengthN). Not used to compute the mask itself, but provided to mirror a realistic training pipeline.doc_ids(list of int): the document id assigned to each token (lengthN). Tokens with the samedoc_idbelong to the same document.
Return a boolean numpy array of shape (N, N) where entry (i, j) is True if and only if the query token at position i is allowed to attend to the key token at position j.
Example
Input:
token_ids = [10, 11, 20, 21]
doc_ids = [0, 0, 1, 1]
Output:
[[1, 0, 0, 0], [1, 1, 0, 0], [0, 0, 1, 0], [0, 0, 1, 1]]
Reasoning: Positions 0 and 1 belong to document 0; positions 2 and 3 belong to document 1. Within each document attention is causal: position 1 may attend to position 0 and itself, position 3 may attend to position 2 and itself. Across documents (e.g., position 2 attending to position 0 or 1) attention is blocked, even though position 0 and 1 come earlier in the packed sequence.
Show hint
Which two separate conditions must both hold for a query to see a key in a packed sequence?
Relevant roles: LLM Engineer, ML Engineer, Deep Learning Engineer and Gen AI Engineer.
Related free problems
- Causal and Padding Masks in Batched Decoder AttentionLLMs & GenAIMedium
- Multi-Head AttentionMedium