Deep Learning · TEE · section

Matrix multiplication: every row has a job

The maths behind a network

Row, column, vector and matrix — read these first

A row runs horizontally, from left to right. A column runs vertically, from top to bottom. A matrix is a rectangular arrangement of numbers in rows and columns. Its size is written rows × columns. A 2 × 3 matrix therefore has two rows and three columns, not six rows.

A vector is an ordered list of numbers. Written horizontally it is a row vector, such as [2, 1, 3]. Written vertically it is a column vector. Neural-network equations in these notes usually use column vectors.

Transpose: rows become columns

The superscript T means transpose, not a power. To obtain Aᵀ, copy row 1 of A as column 1, row 2 as column 2, and so on. Do not reverse the order within a row, change a sign or multiply anything. An m × n matrix becomes n × m.

LaTeX: [A^T]_{ij}=A_{ji}

The two indices are positions: i identifies a row and j a column. The rule says “swap the row and column addresses”. For example, the value at row 1, column 3 moves to row 3, column 1.

Yaad rakhne ka cue: transpose = row ko column bana; number aur sign ko haath mat laga.

Transpose example 1

Question. Find the transpose of the following matrix A and state the size of both matrices.

Given

LaTeX: A=\begin{bmatrix}2&1&3\end{bmatrix}

To find

Aᵀ and both matrix sizes.

Formula — which one and why

A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication.

A is the original matrix; Aᵀ is its transpose; i and j are row/column positions.

LaTeX: [A^T]_{ij}=A_{ji}

Yaad rakhne ka cue: Row seedha column mein; double transpose se original wapas.

Solution — step by step

Step 1: write the original size.

A has 1 row(s) and 3 column(s), so its size is 1 × 3.

Step 2: move original row 1 into new column 1.

Copy [2, 1, 3] down column 1, in the same order.

Step 3: write the transposed matrix.

LaTeX: A^T=\begin{bmatrix}2\\1\\3\end{bmatrix}

Step 4: state its new size.

The result has 3 row(s) and 1 column(s): 3 × 1.

Final answer: Aᵀ has size 3 × 1. Transposing it again gives the original A.

Transpose example 2

Question. Find the transpose of the following matrix A and state the size of both matrices.

Given

LaTeX: A=\begin{bmatrix}1&2&3\\4&5&6\end{bmatrix}

To find

Aᵀ and both matrix sizes.

Formula — which one and why

A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication.

A is the original matrix; Aᵀ is its transpose; i and j are row/column positions.

LaTeX: [A^T]_{ij}=A_{ji}

Yaad rakhne ka cue: Row seedha column mein; double transpose se original wapas.

Solution — step by step

Step 1: write the original size.

A has 2 row(s) and 3 column(s), so its size is 2 × 3.

Step 2: move original row 1 into new column 1.

Copy [1, 2, 3] down column 1, in the same order.

Step 3: move original row 2 into new column 2.

Copy [4, 5, 6] down column 2, in the same order.

Step 4: write the transposed matrix.

LaTeX: A^T=\begin{bmatrix}1&4\\2&5\\3&6\end{bmatrix}

Step 5: state its new size.

The result has 3 row(s) and 2 column(s): 3 × 2.

Final answer: Aᵀ has size 3 × 2. Transposing it again gives the original A.

Transpose example 3

Question. Find the transpose of the following matrix A and state the size of both matrices.

Given

LaTeX: A=\begin{bmatrix}1&-2\\0&3\end{bmatrix}

To find

Aᵀ and both matrix sizes.

Formula — which one and why

A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication.

A is the original matrix; Aᵀ is its transpose; i and j are row/column positions.

LaTeX: [A^T]_{ij}=A_{ji}

Yaad rakhne ka cue: Row seedha column mein; double transpose se original wapas.

Solution — step by step

Step 1: write the original size.

A has 2 row(s) and 2 column(s), so its size is 2 × 2.

Step 2: move original row 1 into new column 1.

Copy [1, -2] down column 1, in the same order.

Step 3: move original row 2 into new column 2.

Copy [0, 3] down column 2, in the same order.

Step 4: write the transposed matrix.

LaTeX: A^T=\begin{bmatrix}1&0\\-2&3\end{bmatrix}

Step 5: state its new size.

The result has 2 row(s) and 2 column(s): 2 × 2.

Final answer: Aᵀ has size 2 × 2. Transposing it again gives the original A.

Transpose example 4

Question. Find the transpose of the following matrix A and state the size of both matrices.

Given

LaTeX: A=\begin{bmatrix}-1\\4\\2\end{bmatrix}

To find

Aᵀ and both matrix sizes.

Formula — which one and why

A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication.

A is the original matrix; Aᵀ is its transpose; i and j are row/column positions.

LaTeX: [A^T]_{ij}=A_{ji}

Yaad rakhne ka cue: Row seedha column mein; double transpose se original wapas.

Solution — step by step

Step 1: write the original size.

A has 3 row(s) and 1 column(s), so its size is 3 × 1.

Step 2: move original row 1 into new column 1.

Copy [-1] down column 1, in the same order.

Step 3: move original row 2 into new column 2.

Copy [4] down column 2, in the same order.

Step 4: move original row 3 into new column 3.

Copy [2] down column 3, in the same order.

Step 5: write the transposed matrix.

LaTeX: A^T=\begin{bmatrix}-1&4&2\end{bmatrix}

Step 6: state its new size.

The result has 1 row(s) and 3 column(s): 1 × 3.

Final answer: Aᵀ has size 1 × 3. Transposing it again gives the original A.

Dot product: multiply matching entries, then add

A dot product turns two equally long lists into one number. Match first with first, second with second, and so on. It is the weighted sum inside a neuron. If both lists are columns, wᵀ turns the weight column into a row so row × column is allowed.

LaTeX: w^Tx=w_1x_1+w_2x_2+w_3x_3

This formula is the three-input case. With two inputs there are only two products; with d inputs there are d products. A bias is an additional term, added after these products.

Dot-product example 1

Question. For w = [2, 3] and x = [4, 1], calculate wᵀx.

Given

Weights w = [2, 3].

Matching inputs x = [4, 1].

Both lists contain 2 numbers.

To find

One scalar value wᵀx.

Formula — which one and why

A dot product is a sum of matching products. There is no bias in this question.

Σ means add; the index i visits each matching pair.

LaTeX: w^Tx=\sum_i w_ix_i

Yaad rakhne ka cue: Same position multiply; sab products add.

Solution — step by step

Step 1: multiply matching pair 1.

LaTeX: (2)(4)=8

Step 2: multiply matching pair 2.

LaTeX: (3)(1)=3

Step 3: add the products.

LaTeX: w^Tx=(8)+(3)=11

Final answer: wᵀx = 11.

Dot-product example 2

Question. For w = [1, -2] and x = [2, 3], calculate wᵀx.

Given

Weights w = [1, -2].

Matching inputs x = [2, 3].

Both lists contain 2 numbers.

To find

One scalar value wᵀx.

Formula — which one and why

A dot product is a sum of matching products. There is no bias in this question.

Σ means add; the index i visits each matching pair.

LaTeX: w^Tx=\sum_i w_ix_i

Yaad rakhne ka cue: Same position multiply; sab products add.

Solution — step by step

Step 1: multiply matching pair 1.

LaTeX: (1)(2)=2

Step 2: multiply matching pair 2.

LaTeX: (-2)(3)=-6

Step 3: add the products.

LaTeX: w^Tx=(2)+(-6)=-4

Final answer: wᵀx = -4.

Dot-product example 3

Question. For w = [0.5, 1, -1] and x = [4, 2, 3], calculate wᵀx.

Given

Weights w = [0.5, 1, -1].

Matching inputs x = [4, 2, 3].

Both lists contain 3 numbers.

To find

One scalar value wᵀx.

Formula — which one and why

A dot product is a sum of matching products. There is no bias in this question.

Σ means add; the index i visits each matching pair.

LaTeX: w^Tx=\sum_i w_ix_i

Yaad rakhne ka cue: Same position multiply; sab products add.

Solution — step by step

Step 1: multiply matching pair 1.

LaTeX: (0.5)(4)=2

Step 2: multiply matching pair 2.

LaTeX: (1)(2)=2

Step 3: multiply matching pair 3.

LaTeX: (-1)(3)=-3

Step 4: add the products.

LaTeX: w^Tx=(2)+(2)+(-3)=1

Final answer: wᵀx = 1.

Matrix multiplication: each row makes one output

To multiply A of size m × n by B of size n × p, the inner sizes n must match. The result has size m × p. Output entry (i,j) is row i of A dotted with column j of B.

LaTeX: C_{ij}=\sum_{k=1}^{n}A_{ik}B_{kj}

For a layer receiving two inputs and producing two neuron outputs, W is 2 × 2, x is 2 × 1 and Wx is 2 × 1. Each row of W holds all weights entering one receiving neuron. Its matching bias is added once.

Yaad rakhne ka cue: multiplication mein andar ke sizes match; answer ka size bahar ke sizes. Har output = ek row × ek column.

Definition. A vector is an ordered collection of scalars; a matrix represents a linear transformation between vector spaces. A neural-network layer combines a linear transformation with an additive bias, giving an affine transformation.

LaTeX: z=Wx+b

For d input features and m receiving neurons, the dimensions are:

LaTeX: W\in\mathbb R^{m\times d},\quad x\in\mathbb R^{d\times1},\quad b,z\in\mathbb R^{m\times1}

The inner dimensions must match. Output component j is the dot product of row j of W with x, plus bias bⱼ. Wᵀ reverses the matrix dimensions. The Hadamard product u⊙v multiplies corresponding components and is different from a matrix product.

Diagram: Each row of the weight matrix calculates one receiving neuron's preactivation.. Labels: A vector is a list. A matrix has rows and columns. | A vector is a list. A matrix has rows and columns. | The shape tells you how the values are arranged. | 2 | 1 | 3 | Vector | 3 × 1 | 1 | 2 | 3 | 4 | 5 | 6 | Matrix | 2 rows × 3 columns | Row 1 | Row 2 | A 2 × 3 matrix has six entries. The shape is not the multiplication result.

Question. Given W = LaTeX: \begin{bmatrix}1 & -2\\0.5 & 3\end{bmatrix}, x = [2, 1]ᵀ and b = [0.1, −0.2]ᵀ, calculate z = Wx + b.

Given

Weights W = LaTeX: \begin{bmatrix}1 & -2\\0.5 & 3\end{bmatrix}.

Input column x = [2,1]ᵀ; bias column b = [0.1,−0.2]ᵀ.

To find

Find both components of z = Wx+b.

Formula — which one and why

Row–column multiplication. Each receiving neuron uses one weight row. Multiply that row by the input column, add the products, then add that neuron’s bias.

Wⱼᵢ = weight from input i to receiver j; xᵢ = input i; bⱼ = bias of receiver j; zⱼ = its preactivation. A superscript T means transpose: exchange rows and columns.

LaTeX: z_j=\sum_i W_{ji}x_i+b_j

Yaad rakhne ka cue: Ek row × poora input column = ek receiving neuron ka answer.

Solution — step by step

Step 1: verify compatible dimensions.

W has 2 rows and 2 columns; x has 2 rows and 1 column. The inner dimensions match, and the result has 2 rows and 1 column.

Step 2: calculate the first receiving row products.

LaTeX: 1(2)=2

LaTeX: (-2)(1)=-2

Step 3: sum row 1 and add its bias.

LaTeX: z_1=2+(-2)+0.1=0.1

Step 4: calculate the second receiving row products.

LaTeX: 0.5(2)=1

LaTeX: 3(1)=3

Step 5: sum row 2 and add its bias.

LaTeX: z_2=1+3-0.2=3.8

LaTeX: z_1=1(2)+(-2)(1)+0.1=0.1

LaTeX: z_2=0.5(2)+3(1)-0.2=3.8

LaTeX: z=\begin{bmatrix}0.1\\3.8\end{bmatrix}

An activation is applied afterward if the question asks for the layer output. In backpropagation, the input gradient uses Wᵀ because it maps derivatives from m outputs back to d inputs. For a single example, an outer product δxᵀ produces a weight-gradient matrix with exactly the same dimensions as W.

Two output columns

Question. Multiply A = LaTeX: \begin{bmatrix}1 & 2\\3 & 4\end{bmatrix} by B = LaTeX: \begin{bmatrix}2 & 0\\1 & 3\end{bmatrix}. Show all four output entries.

Given

LaTeX: A=\begin{bmatrix}1&2\\3&4\end{bmatrix}

LaTeX: B=\begin{bmatrix}2&0\\1&3\end{bmatrix}

To find

The 2 × 2 matrix C = AB.

Formula — which one and why

Matrix multiplication pairs one row of A with one column of B for each output. It is not entry-by-entry multiplication.

Cᵢⱼ is the entry in output row i, column j.

LaTeX: C_{ij}=\sum_k A_{ik}B_{kj}

Yaad rakhne ka cue: Output ki position dekho: wahi row, wahi column.

Solution — step by step

Step 1: check the dimensions.

(2 × 2)(2 × 2) is valid; output size = 2 × 2.

Step 2: row 1 with column 1.

LaTeX: C_{11}=1(2)+2(1)=2+2=4

Step 3: row 1 with column 2.

LaTeX: C_{12}=1(0)+2(3)=0+6=6

Step 4: row 2 with column 1.

LaTeX: C_{21}=3(2)+4(1)=6+4=10

Step 5: row 2 with column 2.

LaTeX: C_{22}=3(0)+4(3)=0+12=12

Step 6: assemble the output by position.

LaTeX: C=\begin{bmatrix}4&6\\10&12\end{bmatrix}

Final answer: C = LaTeX: \begin{bmatrix}4 & 6\\10 & 12\end{bmatrix}, with size 2 × 2.

Two products that must not be confused

Question. Let a = [2,3]ᵀ and b = [4,5]ᵀ. Find the entry-by-entry product a ⊙ b and the outer product abᵀ.

Given

a and b are both 2 × 1 columns: a contains 2,3 and b contains 4,5.

To find

The 2 × 1 entry-by-entry product and the 2 × 2 outer product.

Formula — which one and why

⊙ multiplies corresponding positions; the outer product combines every value of a with every value of b. Backpropagation uses both operations for different reasons.

⊙ is called the Hadamard or elementwise product. bᵀ is a 1 × 2 row.

LaTeX: (a\odot b)_i=a_i b_i

LaTeX: (ab^T)_{ij}=a_i b_j

Yaad rakhne ka cue: ⊙ mein same jagah; outer product mein har pair.

Solution — step by step

Step 1: multiply matching entries.

LaTeX: a\odot b=\begin{bmatrix}8\\15\end{bmatrix}

Step 2: transpose b.

LaTeX: b^T=\begin{bmatrix}4&5\end{bmatrix}

Step 3: form row 1 of the outer product.

LaTeX: 2(4)=8,\qquad2(5)=10

Step 4: form row 2 of the outer product.

LaTeX: 3(4)=12,\qquad3(5)=15

Step 5: write both rows.

LaTeX: ab^T=\begin{bmatrix}8&10\\12&15\end{bmatrix}

Final answer: Entry-by-entry result = [8,15]ᵀ. Outer product = LaTeX: \begin{bmatrix}8 & 10\\12 & 15\end{bmatrix}.