Memory of a Sharded Array on a Device Mesh

Reported in interviews at Google DeepMind

Problem statement

An array $A$ with dtype int8 and shape $[128, 2048]$ is placed on a device mesh with three named axes, $X = 2$, $Y = 8$ and $Z = 2$, for 32 devices in total. Its sharding is written $A[I_{XY}, J]$: the first dimension $I$ is split across the combined $X$ and $Y$ axes, and the second dimension $J$ is not split. Any mesh axis absent from the sharding spec replicates each shard.

  1. How many bytes of $A$ does each device hold?
  2. How many bytes does $A$ occupy in total, summed over all 32 devices?
  3. Describe which part of $A$ each device holds.

Use 1 KiB = 1024 bytes.

Show hint

How does a sharding spec tell you which array dimensions are cut, and by which mesh axes?

Relevant roles: Deep Learning Engineer, LLM Engineer and MLOps Engineer.

Related free problems