# Matrix multiplication Subject: Deep Learning Exam: TEE Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/6 Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/6/ Matrix multiplication implements affine transformations in a dense layer. Dimensions determine which products are valid; each output component is the dot product of a weight row and the input vector, plus its bias. $$ (AB)_{ij}=\sum_k A_{ik}B_{kj} $$ ## Matrix multiplication: every row has a job ### Row, column, vector and matrix — read these first A **row** runs horizontally, from left to right. A **column** runs vertically, from top to bottom. A **matrix** is a rectangular arrangement of numbers in rows and columns. Its size is written rows × columns. A 2 × 3 matrix therefore has two rows and three columns, not six rows. A **vector** is an ordered list of numbers. Written horizontally it is a row vector, such as [2, 1, 3]. Written vertically it is a column vector. Neural-network equations in these notes usually use column vectors. ### Transpose: rows become columns The superscript T means **transpose**, not a power. To obtain Aᵀ, copy row 1 of A as column 1, row 2 as column 2, and so on. Do not reverse the order within a row, change a sign or multiply anything. An m × n matrix becomes n × m. $$ [A^T]_{ij}=A_{ji} $$ The two indices are positions: i identifies a row and j a column. The rule says “swap the row and column addresses”. For example, the value at row 1, column 3 moves to row 3, column 1. **Yaad rakhne ka cue:** transpose = row ko column bana; number aur sign ko haath mat laga. ### Transpose example 1 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}2&1&3\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 1 row(s) and 3 column(s), so its size is 1 × 3. **Step 2: move original row 1 into new column 1.** Copy [2, 1, 3] down column 1, in the same order. **Step 3: write the transposed matrix.** $$ A^T=\begin{bmatrix}2\\1\\3\end{bmatrix} $$ **Step 4: state its new size.** The result has 3 row(s) and 1 column(s): 3 × 1. **Final answer:** Aᵀ has size 3 × 1. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 1 rows × 3 columns A: 1 rows × 3 columns | | | | | --- | --- | --- | | 2 | 1 | 3 | Read each colored row left to right. Aᵀ: 3 rows × 1 columns Aᵀ: 3 rows × 1 columns | | | --- | | 2 | | 1 | | 3 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Transpose example 2 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}1&2&3\\4&5&6\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 2 row(s) and 3 column(s), so its size is 2 × 3. **Step 2: move original row 1 into new column 1.** Copy [1, 2, 3] down column 1, in the same order. **Step 3: move original row 2 into new column 2.** Copy [4, 5, 6] down column 2, in the same order. **Step 4: write the transposed matrix.** $$ A^T=\begin{bmatrix}1&4\\2&5\\3&6\end{bmatrix} $$ **Step 5: state its new size.** The result has 3 row(s) and 2 column(s): 3 × 2. **Final answer:** Aᵀ has size 3 × 2. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 2 rows × 3 columns A: 2 rows × 3 columns | | | | | --- | --- | --- | | 1 | 2 | 3 | | 4 | 5 | 6 | Read each colored row left to right. Aᵀ: 3 rows × 2 columns Aᵀ: 3 rows × 2 columns | | | | --- | --- | | 1 | 4 | | 2 | 5 | | 3 | 6 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Transpose example 3 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}1&-2\\0&3\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 2 row(s) and 2 column(s), so its size is 2 × 2. **Step 2: move original row 1 into new column 1.** Copy [1, -2] down column 1, in the same order. **Step 3: move original row 2 into new column 2.** Copy [0, 3] down column 2, in the same order. **Step 4: write the transposed matrix.** $$ A^T=\begin{bmatrix}1&0\\-2&3\end{bmatrix} $$ **Step 5: state its new size.** The result has 2 row(s) and 2 column(s): 2 × 2. **Final answer:** Aᵀ has size 2 × 2. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 2 rows × 2 columns A: 2 rows × 2 columns | | | | --- | --- | | 1 | −2 | | 0 | 3 | Read each colored row left to right. Aᵀ: 2 rows × 2 columns Aᵀ: 2 rows × 2 columns | | | | --- | --- | | 1 | 0 | | −2 | 3 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Transpose example 4 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}-1\\4\\2\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 3 row(s) and 1 column(s), so its size is 3 × 1. **Step 2: move original row 1 into new column 1.** Copy [-1] down column 1, in the same order. **Step 3: move original row 2 into new column 2.** Copy [4] down column 2, in the same order. **Step 4: move original row 3 into new column 3.** Copy [2] down column 3, in the same order. **Step 5: write the transposed matrix.** $$ A^T=\begin{bmatrix}-1&4&2\end{bmatrix} $$ **Step 6: state its new size.** The result has 1 row(s) and 3 column(s): 1 × 3. **Final answer:** Aᵀ has size 1 × 3. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 3 rows × 1 columns A: 3 rows × 1 columns | | | --- | | −1 | | 4 | | 2 | Read each colored row left to right. Aᵀ: 1 rows × 3 columns Aᵀ: 1 rows × 3 columns | | | | | --- | --- | --- | | −1 | 4 | 2 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Dot product: multiply matching entries, then add A **dot product** turns two equally long lists into one number. Match first with first, second with second, and so on. It is the weighted sum inside a neuron. If both lists are columns, wᵀ turns the weight column into a row so row × column is allowed. $$ w^Tx=w_1x_1+w_2x_2+w_3x_3 $$ This formula is the three-input case. With two inputs there are only two products; with d inputs there are d products. A bias is an additional term, added after these products. ### Dot-product example 1 **Question. For w = [2, 3] and x = [4, 1], calculate wᵀx.** ### Given Weights w = [2, 3]. Matching inputs x = [4, 1]. Both lists contain 2 numbers. ### To find One scalar value wᵀx. ### Formula — which one and why A dot product is a sum of matching products. There is no bias in this question. Σ means add; the index i visits each matching pair. $$ w^Tx=\sum_i w_ix_i $$ **Yaad rakhne ka cue:** Same position multiply; sab products add. ### Solution — step by step **Step 1: multiply matching pair 1.** $$ (2)(4)=8 $$ **Step 2: multiply matching pair 2.** $$ (3)(1)=3 $$ **Step 3: add the products.** $$ w^Tx=(8)+(3)=11 $$ **Final answer:** wᵀx = 11. ### Dot-product example 2 **Question. For w = [1, -2] and x = [2, 3], calculate wᵀx.** ### Given Weights w = [1, -2]. Matching inputs x = [2, 3]. Both lists contain 2 numbers. ### To find One scalar value wᵀx. ### Formula — which one and why A dot product is a sum of matching products. There is no bias in this question. Σ means add; the index i visits each matching pair. $$ w^Tx=\sum_i w_ix_i $$ **Yaad rakhne ka cue:** Same position multiply; sab products add. ### Solution — step by step **Step 1: multiply matching pair 1.** $$ (1)(2)=2 $$ **Step 2: multiply matching pair 2.** $$ (-2)(3)=-6 $$ **Step 3: add the products.** $$ w^Tx=(2)+(-6)=-4 $$ **Final answer:** wᵀx = -4. ### Dot-product example 3 **Question. For w = [0.5, 1, -1] and x = [4, 2, 3], calculate wᵀx.** ### Given Weights w = [0.5, 1, -1]. Matching inputs x = [4, 2, 3]. Both lists contain 3 numbers. ### To find One scalar value wᵀx. ### Formula — which one and why A dot product is a sum of matching products. There is no bias in this question. Σ means add; the index i visits each matching pair. $$ w^Tx=\sum_i w_ix_i $$ **Yaad rakhne ka cue:** Same position multiply; sab products add. ### Solution — step by step **Step 1: multiply matching pair 1.** $$ (0.5)(4)=2 $$ **Step 2: multiply matching pair 2.** $$ (1)(2)=2 $$ **Step 3: multiply matching pair 3.** $$ (-1)(3)=-3 $$ **Step 4: add the products.** $$ w^Tx=(2)+(2)+(-3)=1 $$ **Final answer:** wᵀx = 1. ### Matrix multiplication: each row makes one output To multiply A of size m × n by B of size n × p, the inner sizes n must match. The result has size m × p. Output entry (i,j) is row i of A dotted with column j of B. $$ C_{ij}=\sum_{k=1}^{n}A_{ik}B_{kj} $$ For a layer receiving two inputs and producing two neuron outputs, W is 2 × 2, x is 2 × 1 and Wx is 2 × 1. Each row of W holds all weights entering one receiving neuron. Its matching bias is added once. **Yaad rakhne ka cue:** multiplication mein andar ke sizes match; answer ka size bahar ke sizes. Har output = ek row × ek column. **Definition.** A vector is an ordered collection of scalars; a matrix represents a linear transformation between vector spaces. A neural-network layer combines a linear transformation with an additive bias, giving an affine transformation. $$ z=Wx+b $$ For d input features and m receiving neurons, the dimensions are: $$ W\in\mathbb R^{m\times d},\quad x\in\mathbb R^{d\times1},\quad b,z\in\mathbb R^{m\times1} $$ The inner dimensions must match. Output component j is the dot product of row j of W with x, plus bias bⱼ. Wᵀ reverses the matrix dimensions. The Hadamard product u⊙v multiplies corresponding components and is different from a matrix product. ![Each row of the weight matrix calculates one receiving neuron's preactivation.](https://deep-learning-study-room.vercel.app/assets/figures/v_matrix.svg) Diagram: Each row of the weight matrix calculates one receiving neuron's preactivation.. Labels: A vector is a list. A matrix has rows and columns. | A vector is a list. A matrix has rows and columns. | The shape tells you how the values are arranged. | 2 | 1 | 3 | Vector | 3 × 1 | 1 | 2 | 3 | 4 | 5 | 6 | Matrix | 2 rows × 3 columns | Row 1 | Row 2 | A 2 × 3 matrix has six entries. The shape is not the multiplication result. Each row of the weight matrix calculates one receiving neuron's preactivation. Tap the diagram to enlarge. **Question.** Given W = $\begin{bmatrix}1 & -2\\0.5 & 3\end{bmatrix}$ , x = [2, 1]ᵀ and b = [0.1, −0.2]ᵀ, calculate z = Wx + b. ### Given Weights W = $\begin{bmatrix}1 & -2\\0.5 & 3\end{bmatrix}$ . Input column x = [2,1]ᵀ; bias column b = [0.1,−0.2]ᵀ. ### To find Find both components of z = Wx+b. ### Formula — which one and why **Row–column multiplication.** Each receiving neuron uses one weight row. Multiply that row by the input column, add the products, then add that neuron’s bias. Wⱼᵢ = weight from input i to receiver j; xᵢ = input i; bⱼ = bias of receiver j; zⱼ = its preactivation. A superscript T means transpose: exchange rows and columns. $$ z_j=\sum_i W_{ji}x_i+b_j $$ **Yaad rakhne ka cue:** Ek row × poora input column = ek receiving neuron ka answer. ### Solution — step by step **Step 1: verify compatible dimensions.** W has 2 rows and 2 columns; x has 2 rows and 1 column. The inner dimensions match, and the result has 2 rows and 1 column. **Step 2: calculate the first receiving row products.** $$ 1(2)=2 $$ $$ (-2)(1)=-2 $$ **Step 3: sum row 1 and add its bias.** $$ z_1=2+(-2)+0.1=0.1 $$ **Step 4: calculate the second receiving row products.** $$ 0.5(2)=1 $$ $$ 3(1)=3 $$ **Step 5: sum row 2 and add its bias.** $$ z_2=1+3-0.2=3.8 $$ $$ z_1=1(2)+(-2)(1)+0.1=0.1 $$ Row 1 produces output component z1 1. Select weight row 1 1. Select weight row 1 | | | | --- | --- | | 1 | −2 | | 0.5 | 3 | 2. Use the entire input column 2. Use the entire input column | | | --- | | 2 | | 1 | 3. Match and multiply 3. Match and multiply | | | | --- | --- | | 2 | −2 | First × first; second × second. Sum the products**0** + Bias for this row**0.1** = z1**0.1** One receiving neuron corresponds to one row of W and one bias. The other weight row is not used for this component. $$ z_2=0.5(2)+3(1)-0.2=3.8 $$ Row 2 produces output component z2 1. Select weight row 2 1. Select weight row 2 | | | | --- | --- | | 1 | −2 | | 0.5 | 3 | 2. Use the entire input column 2. Use the entire input column | | | --- | | 2 | | 1 | 3. Match and multiply 3. Match and multiply | | | | --- | --- | | 1 | 3 | First × first; second × second. Sum the products**4** + Bias for this row**−0.2** = z2**3.8** One receiving neuron corresponds to one row of W and one bias. The other weight row is not used for this component. $$ z=\begin{bmatrix}0.1\\3.8\end{bmatrix} $$ An activation is applied afterward if the question asks for the layer output. In backpropagation, the input gradient uses Wᵀ because it maps derivatives from m outputs back to d inputs. For a single example, an outer product δxᵀ produces a weight-gradient matrix with exactly the same dimensions as W. ### Two output columns **Question. Multiply A = $\begin{bmatrix}1 & 2\\3 & 4\end{bmatrix}$ by B = $\begin{bmatrix}2 & 0\\1 & 3\end{bmatrix}$ . Show all four output entries.** ### Given $$ A=\begin{bmatrix}1&2\\3&4\end{bmatrix} $$ $$ B=\begin{bmatrix}2&0\\1&3\end{bmatrix} $$ ### To find The 2 × 2 matrix C = AB. ### Formula — which one and why Matrix multiplication pairs one row of A with one column of B for each output. It is not entry-by-entry multiplication. Cᵢⱼ is the entry in output row i, column j. $$ C_{ij}=\sum_k A_{ik}B_{kj} $$ **Yaad rakhne ka cue:** Output ki position dekho: wahi row, wahi column. ### Solution — step by step **Step 1: check the dimensions.** (2 × 2)(2 × 2) is valid; output size = 2 × 2. **Step 2: row 1 with column 1.** $$ C_{11}=1(2)+2(1)=2+2=4 $$ **Step 3: row 1 with column 2.** $$ C_{12}=1(0)+2(3)=0+6=6 $$ **Step 4: row 2 with column 1.** $$ C_{21}=3(2)+4(1)=6+4=10 $$ **Step 5: row 2 with column 2.** $$ C_{22}=3(0)+4(3)=0+12=12 $$ **Step 6: assemble the output by position.** $$ C=\begin{bmatrix}4&6\\10&12\end{bmatrix} $$ **Final answer:** C = $\begin{bmatrix}4 & 6\\10 & 12\end{bmatrix}$ , with size 2 × 2. ### Two products that must not be confused **Question. Let a = [2,3]ᵀ and b = [4,5]ᵀ. Find the entry-by-entry product a ⊙ b and the outer product abᵀ.** ### Given a and b are both 2 × 1 columns: a contains 2,3 and b contains 4,5. ### To find The 2 × 1 entry-by-entry product and the 2 × 2 outer product. ### Formula — which one and why ⊙ multiplies corresponding positions; the outer product combines every value of a with every value of b. Backpropagation uses both operations for different reasons. ⊙ is called the Hadamard or elementwise product. bᵀ is a 1 × 2 row. $$ (a\odot b)_i=a_i b_i $$ $$ (ab^T)_{ij}=a_i b_j $$ **Yaad rakhne ka cue:** ⊙ mein same jagah; outer product mein har pair. ### Solution — step by step **Step 1: multiply matching entries.** $$ a\odot b=\begin{bmatrix}8\\15\end{bmatrix} $$ **Step 2: transpose b.** $$ b^T=\begin{bmatrix}4&5\end{bmatrix} $$ **Step 3: form row 1 of the outer product.** $$ 2(4)=8,\qquad2(5)=10 $$ **Step 4: form row 2 of the outer product.** $$ 3(4)=12,\qquad3(5)=15 $$ **Step 5: write both rows.** $$ ab^T=\begin{bmatrix}8&10\\12&15\end{bmatrix} $$ **Final answer:** Entry-by-entry result = [8,15]ᵀ. Outer product = $\begin{bmatrix}8 & 10\\12 & 15\end{bmatrix}$ . ## Drill 5 Matrix rows receive inputs ## Question **Question.** For the weight matrix W shown below, x = [2,1], b = [0.5,−0.5], find Wx+b and ReLU. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_ed1fcbfdf5.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W | 1 | 2 | −1 | 3 Given matrices for this question Tap the diagram to enlarge. ## Answer — step by step ### Given W = $\begin{bmatrix}1 & 2\\-1 & 3\end{bmatrix}$ . Input column x = [2,1]ᵀ; bias column b = [0.5,−0.5]ᵀ. Activation: ReLU on each result. ### To find Find Wx+b and the activated output. ### Formula — which one and why **Row–column multiplication.** Each receiving neuron uses one weight row. Multiply that row by the input column, add the products, then add that neuron’s bias. Wⱼᵢ = weight from input i to receiver j; xᵢ = input i; bⱼ = bias of receiver j; zⱼ = its preactivation. A superscript T means transpose: exchange rows and columns. $$ z_j=\sum_i W_{ji}x_i+b_j $$ **Yaad rakhne ka cue:** Ek row × poora input column = ek receiving neuron ka answer. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step ### Meaning and formula Each row supplies the weights of one receiving neuron. Multiply that row by the input column, then add that neuron's own bias. The two biases are separate; do not add both to every row. $$ z=Wx+b,\qquad a_j=\max(0,z_j) $$ $$ W=\begin{bmatrix}1&2\\-1&3\end{bmatrix},\quad x=\begin{bmatrix}2\\1\end{bmatrix},\quad b=\begin{bmatrix}0.5\\-0.5\end{bmatrix} $$ ### Step 1: calculate receiving neuron 1 from row 1 $$ z_1=(1\times2)+(2\times1)+0.5 $$ $$ z_1=2+2+0.5=4.5 $$ ### Step 2: calculate receiving neuron 2 from row 2 $$ z_2=((-1)\times2)+(3\times1)+(-0.5) $$ $$ z_2=-2+3-0.5=0.5 $$ ### Step 3: apply ReLU to each entry $$ a_1=\max(0,4.5)=4.5,\qquad a_2=\max(0,0.5)=0.5 $$ $$ z=\begin{bmatrix}4.5\\0.5\end{bmatrix},\qquad a=\begin{bmatrix}4.5\\0.5\end{bmatrix} $$ The shape is 2 rows by 1 column: one output for each receiving row.