Deep Learning · MT2 · section

What one artificial neuron does

Neural-network foundations

Definition. An artificial neuron is a calculation unit that multiplies each input by a matching weight, adds the products and a bias, and applies an activation function to produce its output.

What goes into a neuron?

An input is a number supplied to the calculation. With three inputs, x = [2,1,3] simply lists those three numbers in order. The matching weight list tells the neuron which multiplier to use for each input. A negative weight contributes a negative amount when its input is positive.

The bias is one extra adjustable number added to the total. The total before activation is called z or preactivation. The activation function is the final rule applied to z: for example, ReLU replaces a negative z with zero, while sigmoid turns z into a number between 0 and 1. The resulting output is called a or the activation.

Yaad rakhne ka cue: inputs × apne weights → products ka sum → ek bias → activation → output.

Mathematical model and notation

$a=f(z)$ LaTeX: a=f(z)

Symbol	Meaning and role
xᵢ	Input component i; the neuron receives d such components
wᵢ	Trainable weight multiplying that input component
b	Trainable bias added once to this neuron's total
z	Preactivation: the result before applying the activation function
f	Chosen activation function, such as sigmoid or ReLU
a	Activation: the neuron's output after applying f

The dot product wᵀx is a compact notation for the sum of matching weight–input products. A positive weight and a negative weight can make opposite contributions for the same positive input. A larger weight magnitude means a larger contribution for a fixed input; it is not, by itself, a general measure of feature importance across differently scaled inputs.

Why both a bias and an activation are needed

The bias changes the preactivation independently of x. In a one-input affine function z = wx + b, it is the intercept. In a classifier with a decision boundary z = 0, a bias allows that boundary to be shifted instead of forcing it through the origin.

The activation determines how z is transformed. Sigmoid produces a value between 0 and 1; ReLU retains positive preactivations and sets negative ones to zero. Nonlinear activations let stacked layers represent nonlinear functions. The preactivation and activation are distinct quantities even when a particular ReLU calculation gives them equal values.

Read the neuron in order: input contributions → affine sum z → activation function f → output a. Add one bias at the receiving neuron. Tap the diagram to enlarge.

Worked numerical

Question. A neuron has x = [2, 1, 3]ᵀ, w = [0.4, −0.2, 0.5]ᵀ and b = 0.1. Calculate its preactivation and sigmoid output.

Given

Inputs = [2,1,3]; matching weights = [0.4,−0.2,0.5].

Bias = 0.1; activation = sigmoid.

To find

Find the preactivation z, then the sigmoid output.

Formula — which one and why

Weighted sum and activation. Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.

xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.

$z=\sum_iw_ix_i+b$ LaTeX: z=\sum_iw_ix_i+b

$a=f(z)$ LaTeX: a=f(z)

Yaad rakhne ka cue: Input × weight, sabka sum, bias once, phir activation.

Sigmoid output. The question specifies sigmoid, so turn the complete weighted sum into an output between 0 and 1. Negate z, evaluate the exponential, add 1, then take the reciprocal.

z = sum before activation; a = sigmoid output; e ≈ 2.71828 is the exponential base.

$a=\frac1{1+e^{-z}}$ LaTeX: a=\frac1{1+e^{-z}}

Yaad rakhne ka cue: Minus-z ki e power, plus one, phir one divided by whole denominator.

Solution — step by step

Step 1: multiply input 1 by weight 1.

$x_1w_1=2(0.4)=0.8$ LaTeX: x_1w_1=2(0.4)=0.8

Step 2: multiply input 2 by weight 2.

$x_2w_2=1(-0.2)=-0.2$ LaTeX: x_2w_2=1(-0.2)=-0.2

The minus sign belongs to the weight, so the contribution stays negative.

Step 3: multiply input 3 by weight 3.

$x_3w_3=3(0.5)=1.5$ LaTeX: x_3w_3=3(0.5)=1.5

Step 4: add the three contributions.

Step 5: add the bias once.

$z=s+b=2.1+0.1=2.2$ LaTeX: z=s+b=2.1+0.1=2.2

This is the preactivation. It is not yet the requested sigmoid output.

Step 6: negate z for the exponential.

$-z=-2.2$ LaTeX: -z=-2.2

$e^{-z}=e^{-2.2}\approx0.110803158$ LaTeX: e^{-z}=e^{-2.2}\approx0.110803158

Step 7: calculate the complete denominator.

Step 8: divide one by that denominator.

Final answer: sigmoid output ≈ 0.900249511. It lies between 0 and 1. The raw sum was 2.2, whereas the activated output is approximately 0.90025.

From one neuron to a layer

With m receiving neurons and d input components, W has shape m × d. Row j contains the incoming weights of neuron j. The bias vector has m entries, one for each receiving neuron:

$z=Wx+b,\qquad a=f(z)$ LaTeX: z=Wx+b,\qquad a=f(z)

Sigmoid, tanh and ReLU apply independently to each component. Softmax is different: it transforms a whole vector of output logits together.