Deep Learning · MT2 · formulas

Matrix multiplication

Matrix multiplication implements affine transformations in a dense layer. Dimensions determine which products are valid; each output component is the dot product of a weight row and the input vector, plus its bias.

$(AB)_{ij}=\sum_k A_{ik}B_{kj}$ LaTeX: (AB)_{ij}=\sum_k A_{ik}B_{kj}

Matrix multiplication: every row has a job

Row, column, vector and matrix — read these first

A row runs horizontally, from left to right. A column runs vertically, from top to bottom. A matrix is a rectangular arrangement of numbers in rows and columns. Its size is written rows × columns. A 2 × 3 matrix therefore has two rows and three columns, not six rows.

A vector is an ordered list of numbers. Written horizontally it is a row vector, such as [2, 1, 3]. Written vertically it is a column vector. Neural-network equations in these notes usually use column vectors.

Transpose: rows become columns

The superscript T means transpose, not a power. To obtain Aᵀ, copy row 1 of A as column 1, row 2 as column 2, and so on. Do not reverse the order within a row, change a sign or multiply anything. An m × n matrix becomes n × m.

$[A^T]_{ij}=A_{ji}$ LaTeX: [A^T]_{ij}=A_{ji}

The two indices are positions: i identifies a row and j a column. The rule says “swap the row and column addresses”. For example, the value at row 1, column 3 moves to row 3, column 1.

Yaad rakhne ka cue: transpose = row ko column bana; number aur sign ko haath mat laga.

Transpose example 1

Question. Find the transpose of the following matrix A and state the size of both matrices.

Given

$A=\begin{bmatrix}2&1&3\end{bmatrix}$ LaTeX: A=\begin{bmatrix}2&1&3\end{bmatrix}

To find

Aᵀ and both matrix sizes.

Formula — which one and why

A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication.

A is the original matrix; Aᵀ is its transpose; i and j are row/column positions.

$[A^T]_{ij}=A_{ji}$ LaTeX: [A^T]_{ij}=A_{ji}

Yaad rakhne ka cue: Row seedha column mein; double transpose se original wapas.

Solution — step by step

Step 1: write the original size.

A has 1 row(s) and 3 column(s), so its size is 1 × 3.

Step 2: move original row 1 into new column 1.

Copy [2, 1, 3] down column 1, in the same order.

Step 3: write the transposed matrix.

$A^T=\begin{bmatrix}2\\1\\3\end{bmatrix}$ LaTeX: A^T=\begin{bmatrix}2\\1\\3\end{bmatrix}

Step 4: state its new size.

The result has 3 row(s) and 1 column(s): 3 × 1.

Final answer: Aᵀ has size 3 × 1. Transposing it again gives the original A.

Transpose: follow the same numbers from a row into a column

A: 1 rows × 3 columns
2	1	3

Read each colored row left to right.

Aᵀ: 3 rows × 1 columns
2
1
3

Each original row is now the matching colored column.

The entry at row i, column j moves to row j, column i. Its value and sign do not change.

Transpose example 2

Question. Find the transpose of the following matrix A and state the size of both matrices.

Given

$A=\begin{bmatrix}1&2&3\\4&5&6\end{bmatrix}$ LaTeX: A=\begin{bmatrix}1&2&3\\4&5&6\end{bmatrix}

To find

Aᵀ and both matrix sizes.

Formula — which one and why

A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication.

A is the original matrix; Aᵀ is its transpose; i and j are row/column positions.

$[A^T]_{ij}=A_{ji}$ LaTeX: [A^T]_{ij}=A_{ji}

Yaad rakhne ka cue: Row seedha column mein; double transpose se original wapas.

Solution — step by step

Step 1: write the original size.

A has 2 row(s) and 3 column(s), so its size is 2 × 3.

Step 2: move original row 1 into new column 1.

Copy [1, 2, 3] down column 1, in the same order.

Step 3: move original row 2 into new column 2.

Copy [4, 5, 6] down column 2, in the same order.

Step 4: write the transposed matrix.

$A^T=\begin{bmatrix}1&4\\2&5\\3&6\end{bmatrix}$ LaTeX: A^T=\begin{bmatrix}1&4\\2&5\\3&6\end{bmatrix}

Step 5: state its new size.

The result has 3 row(s) and 2 column(s): 3 × 2.

Final answer: Aᵀ has size 3 × 2. Transposing it again gives the original A.

Transpose: follow the same numbers from a row into a column

A: 2 rows × 3 columns
1	2	3
4	5	6

Read each colored row left to right.

Aᵀ: 3 rows × 2 columns
1	4
2	5
3	6

Each original row is now the matching colored column.

The entry at row i, column j moves to row j, column i. Its value and sign do not change.

Transpose example 3

Question. Find the transpose of the following matrix A and state the size of both matrices.

Given

$A=\begin{bmatrix}1&-2\\0&3\end{bmatrix}$ LaTeX: A=\begin{bmatrix}1&-2\\0&3\end{bmatrix}

To find

Aᵀ and both matrix sizes.

Formula — which one and why

A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication.

A is the original matrix; Aᵀ is its transpose; i and j are row/column positions.

$[A^T]_{ij}=A_{ji}$ LaTeX: [A^T]_{ij}=A_{ji}

Yaad rakhne ka cue: Row seedha column mein; double transpose se original wapas.

Solution — step by step

Step 1: write the original size.

A has 2 row(s) and 2 column(s), so its size is 2 × 2.

Step 2: move original row 1 into new column 1.

Copy [1, -2] down column 1, in the same order.

Step 3: move original row 2 into new column 2.

Copy [0, 3] down column 2, in the same order.

Step 4: write the transposed matrix.

$A^T=\begin{bmatrix}1&0\\-2&3\end{bmatrix}$ LaTeX: A^T=\begin{bmatrix}1&0\\-2&3\end{bmatrix}

Step 5: state its new size.

The result has 2 row(s) and 2 column(s): 2 × 2.

Final answer: Aᵀ has size 2 × 2. Transposing it again gives the original A.

Transpose: follow the same numbers from a row into a column

A: 2 rows × 2 columns
1	−2
0	3

Read each colored row left to right.

Aᵀ: 2 rows × 2 columns
1	0
−2	3

Each original row is now the matching colored column.

The entry at row i, column j moves to row j, column i. Its value and sign do not change.

Transpose example 4

Question. Find the transpose of the following matrix A and state the size of both matrices.

Given

$A=\begin{bmatrix}-1\\4\\2\end{bmatrix}$ LaTeX: A=\begin{bmatrix}-1\\4\\2\end{bmatrix}

To find

Aᵀ and both matrix sizes.

Formula — which one and why

A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication.

A is the original matrix; Aᵀ is its transpose; i and j are row/column positions.

$[A^T]_{ij}=A_{ji}$ LaTeX: [A^T]_{ij}=A_{ji}

Yaad rakhne ka cue: Row seedha column mein; double transpose se original wapas.

Solution — step by step

Step 1: write the original size.

A has 3 row(s) and 1 column(s), so its size is 3 × 1.

Step 2: move original row 1 into new column 1.

Copy [-1] down column 1, in the same order.

Step 3: move original row 2 into new column 2.

Copy [4] down column 2, in the same order.

Step 4: move original row 3 into new column 3.

Copy [2] down column 3, in the same order.

Step 5: write the transposed matrix.

$A^T=\begin{bmatrix}-1&4&2\end{bmatrix}$ LaTeX: A^T=\begin{bmatrix}-1&4&2\end{bmatrix}

Step 6: state its new size.

The result has 1 row(s) and 3 column(s): 1 × 3.

Final answer: Aᵀ has size 1 × 3. Transposing it again gives the original A.

Transpose: follow the same numbers from a row into a column

A: 3 rows × 1 columns
−1
4
2

Read each colored row left to right.

Aᵀ: 1 rows × 3 columns
−1	4	2

Each original row is now the matching colored column.

The entry at row i, column j moves to row j, column i. Its value and sign do not change.

Dot product: multiply matching entries, then add

A dot product turns two equally long lists into one number. Match first with first, second with second, and so on. It is the weighted sum inside a neuron. If both lists are columns, wᵀ turns the weight column into a row so row × column is allowed.

This formula is the three-input case. With two inputs there are only two products; with d inputs there are d products. A bias is an additional term, added after these products.

Dot-product example 1

Question. For w = [2, 3] and x = [4, 1], calculate wᵀx.

Given

Weights w = [2, 3].

Matching inputs x = [4, 1].

Both lists contain 2 numbers.

To find

One scalar value wᵀx.

Formula — which one and why

A dot product is a sum of matching products. There is no bias in this question.

Σ means add; the index i visits each matching pair.

$w^Tx=\sum_i w_ix_i$ LaTeX: w^Tx=\sum_i w_ix_i

Yaad rakhne ka cue: Same position multiply; sab products add.

Solution — step by step

Step 1: multiply matching pair 1.

$(2)(4)=8$ LaTeX: (2)(4)=8

Step 2: multiply matching pair 2.

$(3)(1)=3$ LaTeX: (3)(1)=3

Step 3: add the products.

$w^Tx=(8)+(3)=11$ LaTeX: w^Tx=(8)+(3)=11

Final answer: wᵀx = 11.

Dot-product example 2

Question. For w = [1, -2] and x = [2, 3], calculate wᵀx.

Given

Weights w = [1, -2].

Matching inputs x = [2, 3].

Both lists contain 2 numbers.

To find

One scalar value wᵀx.

Formula — which one and why

A dot product is a sum of matching products. There is no bias in this question.

Σ means add; the index i visits each matching pair.

$w^Tx=\sum_i w_ix_i$ LaTeX: w^Tx=\sum_i w_ix_i

Yaad rakhne ka cue: Same position multiply; sab products add.

Solution — step by step

Step 1: multiply matching pair 1.

$(1)(2)=2$ LaTeX: (1)(2)=2

Step 2: multiply matching pair 2.

$(-2)(3)=-6$ LaTeX: (-2)(3)=-6

Step 3: add the products.

$w^Tx=(2)+(-6)=-4$ LaTeX: w^Tx=(2)+(-6)=-4

Final answer: wᵀx = -4.

Dot-product example 3

Question. For w = [0.5, 1, -1] and x = [4, 2, 3], calculate wᵀx.

Given

Weights w = [0.5, 1, -1].

Matching inputs x = [4, 2, 3].

Both lists contain 3 numbers.

To find

One scalar value wᵀx.

Formula — which one and why

A dot product is a sum of matching products. There is no bias in this question.

Σ means add; the index i visits each matching pair.

$w^Tx=\sum_i w_ix_i$ LaTeX: w^Tx=\sum_i w_ix_i

Yaad rakhne ka cue: Same position multiply; sab products add.

Solution — step by step

Step 1: multiply matching pair 1.

$(0.5)(4)=2$ LaTeX: (0.5)(4)=2

Step 2: multiply matching pair 2.

$(1)(2)=2$ LaTeX: (1)(2)=2

Step 3: multiply matching pair 3.

$(-1)(3)=-3$ LaTeX: (-1)(3)=-3

Step 4: add the products.

Final answer: wᵀx = 1.

Matrix multiplication: each row makes one output

To multiply A of size m × n by B of size n × p, the inner sizes n must match. The result has size m × p. Output entry (i,j) is row i of A dotted with column j of B.

$C_{ij}=\sum_{k=1}^{n}A_{ik}B_{kj}$ LaTeX: C_{ij}=\sum_{k=1}^{n}A_{ik}B_{kj}

For a layer receiving two inputs and producing two neuron outputs, W is 2 × 2, x is 2 × 1 and Wx is 2 × 1. Each row of W holds all weights entering one receiving neuron. Its matching bias is added once.

Yaad rakhne ka cue: multiplication mein andar ke sizes match; answer ka size bahar ke sizes. Har output = ek row × ek column.

Definition. A vector is an ordered collection of scalars; a matrix represents a linear transformation between vector spaces. A neural-network layer combines a linear transformation with an additive bias, giving an affine transformation.

$z=Wx+b$ LaTeX: z=Wx+b

For d input features and m receiving neurons, the dimensions are:

The inner dimensions must match. Output component j is the dot product of row j of W with x, plus bias bⱼ. Wᵀ reverses the matrix dimensions. The Hadamard product u⊙v multiplies corresponding components and is different from a matrix product.

Each row of the weight matrix calculates one receiving neuron's preactivation. Tap the diagram to enlarge.

Question. Given W = $\begin{bmatrix}1 & -2\\0.5 & 3\end{bmatrix}$ , x = [2, 1]ᵀ and b = [0.1, −0.2]ᵀ, calculate z = Wx + b.

Given

Weights W = $\begin{bmatrix}1 & -2\\0.5 & 3\end{bmatrix}$ .

Input column x = [2,1]ᵀ; bias column b = [0.1,−0.2]ᵀ.

To find

Find both components of z = Wx+b.

Formula — which one and why

Row–column multiplication. Each receiving neuron uses one weight row. Multiply that row by the input column, add the products, then add that neuron’s bias.

Wⱼᵢ = weight from input i to receiver j; xᵢ = input i; bⱼ = bias of receiver j; zⱼ = its preactivation. A superscript T means transpose: exchange rows and columns.

$z_j=\sum_i W_{ji}x_i+b_j$ LaTeX: z_j=\sum_i W_{ji}x_i+b_j

Yaad rakhne ka cue: Ek row × poora input column = ek receiving neuron ka answer.

Solution — step by step

Step 1: verify compatible dimensions.

W has 2 rows and 2 columns; x has 2 rows and 1 column. The inner dimensions match, and the result has 2 rows and 1 column.

Step 2: calculate the first receiving row products.

$1(2)=2$ LaTeX: 1(2)=2

$(-2)(1)=-2$ LaTeX: (-2)(1)=-2

Step 3: sum row 1 and add its bias.

$z_1=2+(-2)+0.1=0.1$ LaTeX: z_1=2+(-2)+0.1=0.1

Step 4: calculate the second receiving row products.

$0.5(2)=1$ LaTeX: 0.5(2)=1

$3(1)=3$ LaTeX: 3(1)=3

Step 5: sum row 2 and add its bias.

$z_2=1+3-0.2=3.8$ LaTeX: z_2=1+3-0.2=3.8

Row 1 produces output component z1

1. Select weight row 1
1	−2
0.5	3

2. Use the entire input column
2
1

3. Match and multiply
2	−2

First × first; second × second.

Sum the products0

Bias for this row0.1

z10.1

One receiving neuron corresponds to one row of W and one bias. The other weight row is not used for this component.

Row 2 produces output component z2

1. Select weight row 2
1	−2
0.5	3

2. Use the entire input column
2
1

3. Match and multiply
1	3

First × first; second × second.

Sum the products4

Bias for this row−0.2

z23.8

One receiving neuron corresponds to one row of W and one bias. The other weight row is not used for this component.

$z=\begin{bmatrix}0.1\\3.8\end{bmatrix}$ LaTeX: z=\begin{bmatrix}0.1\\3.8\end{bmatrix}

An activation is applied afterward if the question asks for the layer output. In backpropagation, the input gradient uses Wᵀ because it maps derivatives from m outputs back to d inputs. For a single example, an outer product δxᵀ produces a weight-gradient matrix with exactly the same dimensions as W.

Two output columns

Question. Multiply A = $\begin{bmatrix}1 & 2\\3 & 4\end{bmatrix}$ by B = $\begin{bmatrix}2 & 0\\1 & 3\end{bmatrix}$ . Show all four output entries.

Given

$A=\begin{bmatrix}1&2\\3&4\end{bmatrix}$ LaTeX: A=\begin{bmatrix}1&2\\3&4\end{bmatrix}

$B=\begin{bmatrix}2&0\\1&3\end{bmatrix}$ LaTeX: B=\begin{bmatrix}2&0\\1&3\end{bmatrix}

To find

The 2 × 2 matrix C = AB.

Formula — which one and why

Matrix multiplication pairs one row of A with one column of B for each output. It is not entry-by-entry multiplication.

Cᵢⱼ is the entry in output row i, column j.

$C_{ij}=\sum_k A_{ik}B_{kj}$ LaTeX: C_{ij}=\sum_k A_{ik}B_{kj}

Yaad rakhne ka cue: Output ki position dekho: wahi row, wahi column.

Solution — step by step

Step 1: check the dimensions.

(2 × 2)(2 × 2) is valid; output size = 2 × 2.

Step 2: row 1 with column 1.

Step 3: row 1 with column 2.

Step 4: row 2 with column 1.

Step 5: row 2 with column 2.

Step 6: assemble the output by position.

$C=\begin{bmatrix}4&6\\10&12\end{bmatrix}$ LaTeX: C=\begin{bmatrix}4&6\\10&12\end{bmatrix}

Final answer: C = $\begin{bmatrix}4 & 6\\10 & 12\end{bmatrix}$ , with size 2 × 2.

Two products that must not be confused

Question. Let a = [2,3]ᵀ and b = [4,5]ᵀ. Find the entry-by-entry product a ⊙ b and the outer product abᵀ.

Given

a and b are both 2 × 1 columns: a contains 2,3 and b contains 4,5.

To find

The 2 × 1 entry-by-entry product and the 2 × 2 outer product.

Formula — which one and why

⊙ multiplies corresponding positions; the outer product combines every value of a with every value of b. Backpropagation uses both operations for different reasons.

⊙ is called the Hadamard or elementwise product. bᵀ is a 1 × 2 row.

$(a\odot b)_i=a_i b_i$ LaTeX: (a\odot b)_i=a_i b_i

$(ab^T)_{ij}=a_i b_j$ LaTeX: (ab^T)_{ij}=a_i b_j

Yaad rakhne ka cue: ⊙ mein same jagah; outer product mein har pair.

Solution — step by step

Step 1: multiply matching entries.

$a\odot b=\begin{bmatrix}8\\15\end{bmatrix}$ LaTeX: a\odot b=\begin{bmatrix}8\\15\end{bmatrix}

Step 2: transpose b.

$b^T=\begin{bmatrix}4&5\end{bmatrix}$ LaTeX: b^T=\begin{bmatrix}4&5\end{bmatrix}

Step 3: form row 1 of the outer product.

$2(4)=8,\qquad2(5)=10$ LaTeX: 2(4)=8,\qquad2(5)=10

Step 4: form row 2 of the outer product.

$3(4)=12,\qquad3(5)=15$ LaTeX: 3(4)=12,\qquad3(5)=15

Step 5: write both rows.

$ab^T=\begin{bmatrix}8&10\\12&15\end{bmatrix}$ LaTeX: ab^T=\begin{bmatrix}8&10\\12&15\end{bmatrix}

Final answer: Entry-by-entry result = [8,15]ᵀ. Outer product = $\begin{bmatrix}8 & 10\\12 & 15\end{bmatrix}$ .

Drill 5 Matrix rows receive inputs

Question

Question. For the weight matrix W shown below, x = [2,1], b = [0.5,−0.5], find Wx+b and ReLU.

Answer — step by step

Given

W = $\begin{bmatrix}1 & 2\\-1 & 3\end{bmatrix}$ .

Input column x = [2,1]ᵀ; bias column b = [0.5,−0.5]ᵀ.

Activation: ReLU on each result.

To find

Find Wx+b and the activated output.

Formula — which one and why

Row–column multiplication. Each receiving neuron uses one weight row. Multiply that row by the input column, add the products, then add that neuron’s bias.

Wⱼᵢ = weight from input i to receiver j; xᵢ = input i; bⱼ = bias of receiver j; zⱼ = its preactivation. A superscript T means transpose: exchange rows and columns.

$z_j=\sum_i W_{ji}x_i+b_j$ LaTeX: z_j=\sum_i W_{ji}x_i+b_j

Yaad rakhne ka cue: Ek row × poora input column = ek receiving neuron ka answer.

ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.

z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.

$\operatorname{ReLU}(z)=\max(0,z)$ LaTeX: \operatorname{ReLU}(z)=\max(0,z)

$\operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}$ LaTeX: \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}

Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.

Solution — step by step

Meaning and formula

Each row supplies the weights of one receiving neuron. Multiply that row by the input column, then add that neuron's own bias. The two biases are separate; do not add both to every row.

$W=\begin{bmatrix}1&2\\-1&3\end{bmatrix},\quad x=\begin{bmatrix}2\\1\end{bmatrix},\quad b=\begin{bmatrix}0.5\\-0.5\end{bmatrix}$ LaTeX:

W=\begin{bmatrix}1&2\\-1&3\end{bmatrix},\quad x=\begin{bmatrix}2\\1\end{bmatrix},\quad b=\begin{bmatrix}0.5\\-0.5\end{bmatrix}

Step 1: calculate receiving neuron 1 from row 1

$z_1=2+2+0.5=4.5$ LaTeX: z_1=2+2+0.5=4.5

Step 2: calculate receiving neuron 2 from row 2

$z_2=-2+3-0.5=0.5$ LaTeX: z_2=-2+3-0.5=0.5

Step 3: apply ReLU to each entry

$z=\begin{bmatrix}4.5\\0.5\end{bmatrix},\qquad a=\begin{bmatrix}4.5\\0.5\end{bmatrix}$ LaTeX: z=\begin{bmatrix}4.5\\0.5\end{bmatrix},\qquad a=\begin{bmatrix}4.5\\0.5\end{bmatrix}

The shape is 2 rows by 1 column: one output for each receiving row.