Deep Learning · MT2 · formulas

Dot product and a neuron

A neuron starts with matched input–weight pairs. Multiply inside each pair, add the products, add the bias once, and then apply the activation. The summation sign is simply a shorter way to write those additions.

LaTeX: z=\sum_i w_i x_i+b

LaTeX: a=f(z)

Model answer: explain an artificial neuron

Answer — step by step

An artificial neuron is a computational unit that applies an activation function to an affine combination of its inputs. Its trainable weights and bias determine the preactivation; its output is passed to subsequent layers.

Diagram: Follow each input through its weight, add one bias at the receiving neuron, then apply activation.. Labels: INPUTS | WEIGHTED SUM | ACTIVATION | x1 | w1 | x2 | w2 | x3 | w3 | z = Σwx + b | a = f(z) | Add one bias b | Multiply → add → activate

Formula and symbols

LaTeX: z=w_1x_1+w_2x_2+\cdots+w_nx_n+b,\qquad a=f(z)

Part	Meaning	What it does
x₁, x₂, …	Input features	Supply the information for this calculation
w₁, w₂, …	Weights	Control the contribution of each matching input
b	Bias	Shifts the weighted sum independently of the inputs
z	Net input	The number before activation
f	Activation rule	Transforms z, often nonlinearly
a	Activated output	The number sent onward

Worked ReLU neuron example

Question. An artificial neuron has x₁ = 2, x₂ = 1, w₁ = 0.5, w₂ = −0.2 and b = 0.1. Calculate its preactivation and ReLU output.

Given

Inputs = [2,1]; weights = [0.5,−0.2]; bias = 0.1; activation = ReLU.

To find

Preactivation z and activated output a.

Formula — which one and why

Each input contributes through its own weight. Add one bias after summing the products; ReLU then keeps a positive total.

z means before activation; a means after activation.

LaTeX: z=w_1x_1+w_2x_2+b

LaTeX: a=\max(0,z)

Yaad rakhne ka cue: Product, product, sum, bias, phir activation.

Solution — step by step

Step 1: multiply the first pair.

LaTeX: 0.5(2)=1

Step 2: multiply the second pair.

LaTeX: (-0.2)(1)=-0.2

Step 3: add both contributions.

LaTeX: 1+(-0.2)=0.8

Step 4: add the bias once.

LaTeX: z=0.8+0.1=0.9

Step 5: apply ReLU.

LaTeX: a=\max(0,0.9)=0.9

Final answer: z = 0.9 and a = 0.9. ReLU leaves this positive preactivation unchanged.

The weights and bias are learned parameters. During training, their values are adjusted to reduce an objective. The activation function itself is usually chosen as part of the architecture. Adding a bias does not mean adding one bias after every input product; add the receiving neuron's bias once.

What one artificial neuron does

Definition. An artificial neuron is a calculation unit that multiplies each input by a matching weight, adds the products and a bias, and applies an activation function to produce its output.

What goes into a neuron?

An input is a number supplied to the calculation. With three inputs, x = [2,1,3] simply lists those three numbers in order. The matching weight list tells the neuron which multiplier to use for each input. A negative weight contributes a negative amount when its input is positive.

The bias is one extra adjustable number added to the total. The total before activation is called z or preactivation. The activation function is the final rule applied to z: for example, ReLU replaces a negative z with zero, while sigmoid turns z into a number between 0 and 1. The resulting output is called a or the activation.

Yaad rakhne ka cue: inputs × apne weights → products ka sum → ek bias → activation → output.

Mathematical model and notation

LaTeX: z=\sum_{i=1}^{d}w_ix_i+b=w^Tx+b

LaTeX: a=f(z)

The dot product wᵀx is a compact notation for the sum of matching weight–input products. A positive weight and a negative weight can make opposite contributions for the same positive input. A larger weight magnitude means a larger contribution for a fixed input; it is not, by itself, a general measure of feature importance across differently scaled inputs.

Why both a bias and an activation are needed

The bias changes the preactivation independently of x. In a one-input affine function z = wx + b, it is the intercept. In a classifier with a decision boundary z = 0, a bias allows that boundary to be shifted instead of forcing it through the origin.

The activation determines how z is transformed. Sigmoid produces a value between 0 and 1; ReLU retains positive preactivations and sets negative ones to zero. Nonlinear activations let stacked layers represent nonlinear functions. The preactivation and activation are distinct quantities even when a particular ReLU calculation gives them equal values.

Diagram: Read the neuron in order: input contributions → affine sum z → activation function f → output a. Add one bias at the receiving neuron.. Labels: INPUTS | WEIGHTED SUM | ACTIVATION | x1 | w1 | x2 | w2 | x3 | w3 | z = Σwx + b | a = f(z) | Add one bias b | Multiply → add → activate

Worked numerical

Question. A neuron has x = [2, 1, 3]ᵀ, w = [0.4, −0.2, 0.5]ᵀ and b = 0.1. Calculate its preactivation and sigmoid output.

Given

Inputs = [2,1,3]; matching weights = [0.4,−0.2,0.5].

Bias = 0.1; activation = sigmoid.

To find

Find the preactivation z, then the sigmoid output.

Formula — which one and why

Weighted sum and activation. Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.

xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.

LaTeX: z=\sum_iw_ix_i+b

LaTeX: a=f(z)

Yaad rakhne ka cue: Input × weight, sabka sum, bias once, phir activation.

Sigmoid output. The question specifies sigmoid, so turn the complete weighted sum into an output between 0 and 1. Negate z, evaluate the exponential, add 1, then take the reciprocal.

z = sum before activation; a = sigmoid output; e ≈ 2.71828 is the exponential base.

LaTeX: a=\frac1{1+e^{-z}}

Yaad rakhne ka cue: Minus-z ki e power, plus one, phir one divided by whole denominator.

Solution — step by step

Step 1: multiply input 1 by weight 1.

LaTeX: x_1w_1=2(0.4)=0.8

Step 2: multiply input 2 by weight 2.

LaTeX: x_2w_2=1(-0.2)=-0.2

The minus sign belongs to the weight, so the contribution stays negative.

Step 3: multiply input 3 by weight 3.

LaTeX: x_3w_3=3(0.5)=1.5

Step 4: add the three contributions.

LaTeX: s=0.8+(-0.2)+1.5=0.6+1.5=2.1

Step 5: add the bias once.

LaTeX: z=s+b=2.1+0.1=2.2

This is the preactivation. It is not yet the requested sigmoid output.

Step 6: negate z for the exponential.

LaTeX: -z=-2.2

LaTeX: e^{-z}=e^{-2.2}\approx0.110803158

Step 7: calculate the complete denominator.

LaTeX: 1+e^{-z}=1+0.110803158=1.110803158

Step 8: divide one by that denominator.

LaTeX: a=\frac1{1+e^{-z}}=\frac1{1.110803158}\approx0.900249511

Final answer: sigmoid output ≈ 0.900249511. It lies between 0 and 1. The raw sum was 2.2, whereas the activated output is approximately 0.90025.

From one neuron to a layer

With m receiving neurons and d input components, W has shape m × d. Row j contains the incoming weights of neuron j. The bias vector has m entries, one for each receiving neuron:

LaTeX: z=Wx+b,\qquad a=f(z)

Sigmoid, tanh and ReLU apply independently to each component. Softmax is different: it transforms a whole vector of output logits together.

Neuron example 2: ReLU output

Question. A neuron receives inputs [2], weights [3] and bias 1. Calculate its preactivation and ReLU output.

Given

Inputs = [2].

Weights = [3]; bias = 1.

Activation = ReLU.

To find

z before activation and a after activation.

Formula — which one and why

The weighted sum combines the inputs. The output must then use the activation explicitly requested.

z names the sum; a names the activated result. They are not always equal.

LaTeX: z=\sum_i w_ix_i+b

LaTeX: a=\max(0,z)

Yaad rakhne ka cue: Sum alag, activation alag; activation lagana mat bhool.

Solution — step by step

Step 1: multiply input 1 by its own weight.

LaTeX: (2)(3)=6

Step 2: add the input contributions.

LaTeX: s=(6)=6

Step 3: add the bias once.

LaTeX: z=6+(1)=7

Step 4: apply the stated activation.

LaTeX: a=\max(0,7)=7

Final answer: z = 7; a = 7.

Neuron example 3: ReLU output

Question. A neuron receives inputs [2, 1], weights [-1, 0.5] and bias 0.2. Calculate its preactivation and ReLU output.

Given

Inputs = [2, 1].

Weights = [-1, 0.5]; bias = 0.2.

Activation = ReLU.

To find

z before activation and a after activation.

Formula — which one and why

The weighted sum combines the inputs. The output must then use the activation explicitly requested.

z names the sum; a names the activated result. They are not always equal.

LaTeX: z=\sum_i w_ix_i+b

LaTeX: a=\max(0,z)

Yaad rakhne ka cue: Sum alag, activation alag; activation lagana mat bhool.

Solution — step by step

Step 1: multiply input 1 by its own weight.

LaTeX: (2)(-1)=-2

Step 2: multiply input 2 by its own weight.

LaTeX: (1)(0.5)=0.5

Step 3: add the input contributions.

LaTeX: s=(-2)+(0.5)=-1.5

Step 4: add the bias once.

LaTeX: z=-1.5+(0.2)=-1.3

Step 5: apply the stated activation.

LaTeX: a=\max(0,-1.3)=0

Final answer: z = -1.3; a = 0.

Neuron example 4: sigmoid output

Question. A neuron receives inputs [1, 2], weights [0.2, -0.1] and bias 0. Calculate its preactivation and sigmoid output.

Given

Inputs = [1, 2].

Weights = [0.2, -0.1]; bias = 0.

Activation = sigmoid.

To find

z before activation and a after activation.

Formula — which one and why

The weighted sum combines the inputs. The output must then use the activation explicitly requested.

z names the sum; a names the activated result. They are not always equal.

LaTeX: z=\sum_i w_ix_i+b

LaTeX: a=\frac1{1+e^{-z}}

Yaad rakhne ka cue: Sum alag, activation alag; activation lagana mat bhool.

Solution — step by step

Step 1: multiply input 1 by its own weight.

LaTeX: (1)(0.2)=0.2

Step 2: multiply input 2 by its own weight.

LaTeX: (2)(-0.1)=-0.2

Step 3: add the input contributions.

LaTeX: s=(0.2)+(-0.2)=0

Step 4: add the bias once.

LaTeX: z=0+(0)=0

Step 5: apply the stated activation.

LaTeX: a=\frac1{1+e^{0}}=\frac1{1+1}=0.5

Final answer: z = 0; a = 0.5.

Drill 4 Dot product and bias

Answer — step by step

Given

Inputs x = [2, −1, 0.5].

Weights w = [0.3, 0.4, −0.2]; bias b = 0.1.

Activation: ReLU.

To find

Find the weighted sum z and ReLU output.

Formula — which one and why

Weighted sum and activation. Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.

xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.

LaTeX: z=\sum_iw_ix_i+b

LaTeX: a=f(z)

Yaad rakhne ka cue: Input × weight, sabka sum, bias once, phir activation.

ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.

z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.

LaTeX: \operatorname{ReLU}(z)=\max(0,z)

LaTeX: \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases}

Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.

Solution — step by step

Meaning and formulas

A dot product means multiply corresponding entries, then add the products. A bias is one extra value added after the sum. ReLU keeps a positive result and replaces a negative result with zero.

LaTeX: z=w_1x_1+w_2x_2+w_3x_3+b,\qquad a=\max(0,z)

Here x contains inputs, w contains weights, b is bias, z is the weighted sum, and a is the activated output.

Step 1: multiply the matching pairs

LaTeX: w_1x_1=0.3\times2=0.6

LaTeX: w_2x_2=0.4\times(-1)=-0.4

LaTeX: w_3x_3=(-0.2)\times0.5=-0.1

Step 2: add all products and add bias once

LaTeX: z=0.6+(-0.4)+(-0.1)+0.1

LaTeX: z=0.2-0.1+0.1=0.2

Step 3: apply the activation

LaTeX: a=\max(0,0.2)=0.2

Final answer: weighted sum z = 0.2; ReLU output a = 0.2. The two numbers happen to match because z is positive.