Intelligent Systems II · TEE · papers

Q2(b) GMM versus k-means, log likelihood and parameter count [10]

Final / re-examination · 28 Apr 2025

Actual historical paper ·100 marks attempted /140 offered · Q1 compulsory + any four of Q2–Q7. Source defects and solution assumptions are explicitly identified.

Question. Q2(b) [10 marks · 2025-04-28]

With reference to Gaussian Mixture model (GMM), answer the following

Compare GMM with K-means algorithm
Derive the equation for Log likelihood function.
What will be the no of parameters to be computed if K=5

Answer

Gaussian mixture model: a probability density formed by a weighted sum of K Gaussian components. Each observation has a latent component label; its membership is generally represented by responsibilities rather than a single compulsory hard assignment.

Aspect	K-means	GMM
Assignment	Hard: choose one closest centroid	Soft: a posterior responsibility for each component
Parameters	K centroid vectors	Mixture weights, mean vectors and covariance matrices
Objective	Minimize within-cluster squared Euclidean distances	Maximize data likelihood or log likelihood
Shape	Nearest-centroid partitions suit roughly spherical/equal-scale groups	Full covariance allows tilted ellipsoids; restricted covariance gives simpler shapes
Updates	Alternate assignment and centroid averaging	EM: responsibility calculation and weighted parameter updates
Uncertainty	No probability from basic assignment alone	Responsibilities quantify ambiguity under the fitted model

Log-likelihood derivation

Step 1. Write one observation's density

p(x_i\mid\theta)=\sum_{k=1}^K\pi_k\,\mathcal N(x_i\mid\mu_k,\Sigma_k),\quad \pi_k\ge0,\quad\sum_k\pi_k=1

\mathcal N(x\mid\mu,\Sigma)=\frac{\exp[-\frac12(x-\mu)^T\Sigma^{-1}(x-\mu)]}{(2\pi)^{d/2}|\Sigma|^{1/2}}

πk is the probability of component k; µk is its centre; Σk describes spread and correlation. Assume positive-definite full covariances for this density.

Step 2. Multiply independent observations

\mathcal L(\theta)=\prod_{i=1}^{N}\left[\sum_{k=1}^{K}\pi_k\mathcal N(x_i\mid\mu_k,\Sigma_k)\right]

The product follows from the usual independent, identically distributed training-sample assumption. The sum inside handles the unobserved component choice.

Step 3. Take the natural logarithm

\ell(\theta)=\log\mathcal L(\theta)=\sum_{i=1}^{N}\log\left[\sum_{k=1}^{K}\pi_k\mathcal N(x_i\mid\mu_k,\Sigma_k)\right]

Log turns the product across samples into a sum. It does not turn the mixture sum inside each logarithm into a sum of logarithms. The coupled terms motivate iterative optimization such as EM.

Step 4. Count parameters and state the missing assumption

The question gives K=5 but does not give dimension d or covariance structure, so one universal integer cannot be determined.

Component	Independent parameters
Means	Kd
Full symmetric covariances	K d (d+1)/2
Mixture weights	K−1, because their sum is 1

n_{free}=Kd+K\frac{d(d+1)}2+(K-1)

K=5:\quad n_{free}=5d+\frac{5d(d+1)}2+4

If one-dimensional data are intended: five means+five variances+four independent weights=14 free parameters. One implementation stores 15 numbers (including all five weights), but the sum-to-one constraint removes one independent degree of freedom. For d-dimensional diagonal covariances the count is 10d+4; state the selected covariance assumption.