Tool 02
Experiment with the linear algebra operations that power neural networks — multiply, add, transpose, and inspect shapes.
Enter valid matrices to compute.
Neural networks are, at their core, sequences of matrix operations. Every linear layer is a matrix multiply. Attention is matrix multiply plus softmax. If matrices feel opaque, transformers feel like magic — this playground makes the math tangible.
A matrix is a 2D grid of numbers — rows and columns. In deep learning, we call these tensors when they have more dimensions, but 2D matrices are the building block.
Each cell holds a learned weight or an activation value. When you see "shape [4, 768]" in transformer traces, that means 4 rows and 768 columns — 4 tokens each represented by a 768-dimensional vector.
To multiply A (m×n) by B (n×p), the inner dimensions must match: columns of A = rows of B. The result C has shape (m×p).
Each cell in C is a dot product — one row of A dotted with one column of B. This is how a row of input values gets combined with columns of weights to produce an output.
A linear (fully connected) layer computes: output = input × weights + bias. The weight matrix stores what the layer has learned. Input activations flow through thousands of these operations per forward pass.
In attention, Q·Kᵀ produces attention scores, then those scores multiply V to produce the output. Same primitive, different meaning.
Transpose flips rows and columns — needed when attention computes Kᵀ. Element-wise add combines tensors of the same shape (bias addition, residual connections).
Softmax turns a row of raw scores into probabilities that sum to 1. In attention, softmax normalizes scores so each token's attention weights over all keys add up to 100%.
Start with 2×2 matrices and multiply by hand, then verify here. Try incompatible shapes to see the error — it teaches why dimension matching matters.
Switch to transpose and shape operations to build intuition before moving to attention and transformer tools.