Multiplying Replicated Matrices on Two GPUs

Reported in interviews at Google

Problem statement

How would you multiply large matrices across a pair of GPUs and deliver the result? Both devices already hold the inputs, stored row by row in reduced precision. The best partition depends on where the output is needed and whether moving it costs more than computing it again.

Choose a partition, calculate transfer time and decide whether to exchange results or recompute them.

Show hint

Which dimension of the product can you split so that each GPU's piece depends only on data it already holds?

Relevant roles: ML Engineer, Deep Learning Engineer and AI Architect.

Related free problems