Intelligent Systems II · TEE · papers
Q3(a) GMM versus K-means [10]
Final examination · 17 Apr 2024
Actual historical paper ·100 marks attempted /140 offered · Q1 compulsory + any four of Q2–Q7. Source defects and solution assumptions are explicitly identified.
Discuss the significance of using Gaussian Mixture Models (GMMs) over K-means clustering in capturing complex data distributions.
Answer
GMM models the density as a weighted sum of Gaussian components. K-means minimizes within-cluster squared Euclidean distance and assigns each point to one nearest centroid.
LaTeX: p(x)=\sum_{k=1}^K\pi_k\mathcal N(x\mid\mu_k,\Sigma_k)| Aspect | K-means | Gaussian mixture |
|---|---|---|
| Assignment | Hard: one cluster label | Soft: posterior responsibility for each component |
| Cluster shape | Voronoi regions from Euclidean distance; suited to compact roughly spherical groups | Full covariance allows elongated and differently oriented ellipses |
| Spread | No explicit cluster-specific covariance in the standard objective | Separate covariance estimates model unequal spreads |
| Overlap | Boundary points assigned completely to one side | Uncertainty is represented by partial memberships |
| Output | Centroids and assignments | Density, mixing weights, means, covariances and responsibilities |
| Training | Alternate assignment and centroid mean | EM responsibility and parameter updates |
| Cost | Usually simpler and faster | More parameters and cost; can overfit or develop singular covariance |
| Initialization | Can reach local minima | Can reach local likelihood maxima; multiple restarts help |
Example. Two customer groups overlap in annual income/spending: one is a narrow diagonal band and one a broad ellipse. A spherical distance-only partition may split the diagonal group badly. Separate covariance matrices can align GMM ellipses with each group. A boundary customer might have responsibilities [0.55,0.45], showing uncertainty instead of an unjustifiably certain label.
Why this matters. The mixture can approximate multimodal distributions; unequal mixing weights represent different group sizes. Its density supports likelihood-based anomaly detection and probabilistic downstream decisions. However a Gaussian component is not automatically a real-world class, and arbitrary non-Gaussian shapes may require several components.
Selection and limitations. Choose K and covariance restrictions using validation/BIC/AIC as appropriate, scale features, regularize covariance and inspect convergence. K-means remains attractive for large, roughly spherical, similarly scaled clusters when hard assignments are enough. GMM is more flexible, not universally superior.