# Q2(b) GMM versus k-means, log likelihood and parameter count [10]

Subject: Intelligent Systems II

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-2025-april/q2b

Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-2025-april--q2b/

Actual historical paper ·100 marks attempted /140 offered · Q1 compulsory + any four of Q2–Q7. Source defects and solution assumptions are explicitly identified.

**Question. Q2(b)** [10 marks · 2025-04-28]

With reference to Gaussian Mixture model (GMM), answer the following

- Compare GMM with K-means algorithm
- Derive the equation for Log likelihood function.
- What will be the no of parameters to be computed if K=5

### Answer

**Gaussian mixture model:** a probability density formed by a weighted sum of K Gaussian components. Each observation has a latent component label; its membership is generally represented by responsibilities rather than a single compulsory hard assignment.

| Aspect | K-means | GMM |
| --- | --- | --- |
| Assignment | Hard: choose one closest centroid | Soft: a posterior responsibility for each component |
| Parameters | K centroid vectors | Mixture weights, mean vectors and covariance matrices |
| Objective | Minimize within-cluster squared Euclidean distances | Maximize data likelihood or log likelihood |
| Shape | Nearest-centroid partitions suit roughly spherical/equal-scale groups | Full covariance allows tilted ellipsoids; restricted covariance gives simpler shapes |
| Updates | Alternate assignment and centroid averaging | EM: responsibility calculation and weighted parameter updates |
| Uncertainty | No probability from basic assignment alone | Responsibilities quantify ambiguity under the fitted model |

### Log-likelihood derivation

### Step 1. Write one observation's density



$$
p(x_i\mid\theta)=\sum_{k=1}^K\pi_k\,\mathcal N(x_i\mid\mu_k,\Sigma_k),\quad \pi_k\ge0,\quad\sum_k\pi_k=1
$$





$$
\mathcal N(x\mid\mu,\Sigma)=\frac{\exp[-\frac12(x-\mu)^T\Sigma^{-1}(x-\mu)]}{(2\pi)^{d/2}|\Sigma|^{1/2}}
$$



πk is the probability of component k; µk is its centre; Σk describes spread and correlation. Assume positive-definite full covariances for this density.

### Step 2. Multiply independent observations



$$
\mathcal L(\theta)=\prod_{i=1}^{N}\left[\sum_{k=1}^{K}\pi_k\mathcal N(x_i\mid\mu_k,\Sigma_k)\right]
$$



The product follows from the usual independent, identically distributed training-sample assumption. The sum inside handles the unobserved component choice.

### Step 3. Take the natural logarithm



$$
\ell(\theta)=\log\mathcal L(\theta)=\sum_{i=1}^{N}\log\left[\sum_{k=1}^{K}\pi_k\mathcal N(x_i\mid\mu_k,\Sigma_k)\right]
$$



Log turns the product across samples into a sum. It does **not** turn the mixture sum inside each logarithm into a sum of logarithms. The coupled terms motivate iterative optimization such as EM.

### Step 4. Count parameters and state the missing assumption

The question gives K=5 but does not give dimension d or covariance structure, so one universal integer cannot be determined.

| Component | Independent parameters |
| --- | --- |
| Means | Kd |
| Full symmetric covariances | K d (d+1)/2 |
| Mixture weights | K−1, because their sum is 1 |



$$
n_{free}=Kd+K\frac{d(d+1)}2+(K-1)
$$





$$
K=5:\quad n_{free}=5d+\frac{5d(d+1)}2+4
$$



**If one-dimensional data are intended:** five means+five variances+four independent weights=**14 free parameters**. One implementation stores 15 numbers (including all five weights), but the sum-to-one constraint removes one independent degree of freedom. For d-dimensional diagonal covariances the count is 10d+4; state the selected covariance assumption.