Memory of a Sharded Array on a Device Mesh
Problem statement
An array $A$ with dtype int8 and shape $[128, 2048]$ is placed on a device mesh with three named axes, $X = 2$, $Y = 8$ and $Z = 2$, for 32 devices in total. Its sharding is written $A[I_{XY}, J]$: the first dimension $I$ is split across the combined $X$ and $Y$ axes, and the second dimension $J$ is not split. Any mesh axis absent from the sharding spec replicates each shard.
- How many bytes of $A$ does each device hold?
- How many bytes does $A$ occupy in total, summed over all 32 devices?
- Describe which part of $A$ each device holds.
Use 1 KiB = 1024 bytes.
Show hint
How does a sharding spec tell you which array dimensions are cut, and by which mesh axes?
Relevant roles: Deep Learning Engineer, LLM Engineer and MLOps Engineer.
Related free problems
- Estimating KV-Cache Memory and Shrinking ItLLMs & GenAIMedium