Deep and Cross Network: Cross Layers v1 and v2
Problem statement
Explicit Feature Crosses
Implement the cross layers of DCN and DCN-v2 for a batch x0 of shape (n, d).
cross_layer_v1(x0, xl, w, b) with w and b of shape (d,):
$$x_{l+1} = x_0\,(x_l^T w) + b + x_l$$
where $x_l^T w$ is a scalar per example, so the first term is x0 scaled per row.
cross_layer_v2(x0, xl, W, b) with W of shape (d, d) and b of shape (d,):
$$x_{l+1} = x_0 \odot (W x_l + b) + x_l$$
with $\odot$ the elementwise product.
cross_network(x0, params, version)
paramsis a list of(w_or_W, b)tuples, one per layer. Apply the layers in order starting fromx_l = x_0and return the finalx_L.- Also return, as the second element of a tuple, the number of parameters used per layer:
2*dfor v1 andd*d + dfor v2.
Example
Input:
x0 = [[1, 2]], w = [1, 0], b = [0, 0], one v1 layer
Output:
[[2, 4]]
Reasoning: x0^T w = 1, so the cross term is x0 itself; adding the residual doubles it.
Show hint
What shape does the per-example projection $x_l^T w$ have when you compute it for a whole batch at once?
Relevant roles: ML Engineer, Applied Scientist, Deep Learning Engineer and Research Scientist.
Related free problems
- Multi-Head AttentionMedium
- Masked Self-AttentionMedium