The E-M Algorithm

Steps
1. 2. 3. 4. 5. Calculate , for all xn, Ci pairs Calculate , for all Ci, xn pairs Calculate new means Calculate new variances Calculate new weights xn is some sample/observation given in the question Ci refers to the ith component of a Gaussian Mixture Model (defined by its mean and variance parameters)

Step 1
Calculating is just a case of plugging the values into this badboy: ( )

So, do this for every sample value (xn), for every component (substituting m and v with Cis mean and variance). Dont worry about component weights for now they come later.

Step 2
Here were calculating how much each sample belongs to each component. The numerator is the previously calculated multiplied by the weight associated with GMM component Ci The denominator is the sum of products of weight and probability of xn for all components in the GMM

Step 3: Calculating New Means

Once the values from the previous steps have been computed, from here on its just a bit of multiplication and addition. No biggy! The numerator is the sum how much the xns belong to the GMM component, Ci, multiplied by the xns themselves. The denominator is the sum of the probabilities that each xn belongs to the GMM component, Ci

Step 4: Calculating New Variances

As in the previous step, weve already worked out all the values well need to compute the new variances. Its done like so: The numerator is the sum of how much the xns belong to the GMM component, Ci, multiplied by the xns (after theyve had the newly-calculated means subtracted and been squared to remove negative values) The denominator is the sum of the probabilities that each xn belongs to the GMM component, Ci

Step 5: Calculating New Weights

The new weights are the mean average of how much the samples belong to a given component of the GMM:

Worked Example
Consider modelling of 1-dimensional data using a Gaussian Mixture Model with 2 mixture components. The parameters of the GMM components at the current iteration of the E-M training procedure are: C1 = [m1 = 6; v1 = 1; w1 = 0.5] C2 = [m2 = 8; v2 = 1; w2 = 0.5]

Consider the sequence of data y = (7, 5, 10, 9, 4). Calculate the new values for the means and variances of the GMM components after one iteration of the E-M training procedure.

Step 1
Component 1 p(x|C1) 0.242 0.242 1.34 x 10-4 4.43 x 10-3 0.054 Component 2 p(x|C2) 0.242 4.43 x 10-3 0.054 0.242 1.34 x 10-4

x 7 5 10 9 4

Step 2
Component 1 p(x|C1) p(C1|x) 0.242 0.5 0.242 0.982 1.34 x 10-4 0.002 4.43 x 10-3 0.018 0.054 0.998 Component 2 p(x|C2) p(C2|x) 0.242 0.5 4.43 x 10-3 0.018 0.054 0.998 0.242 0.982 -4 1.34 x 10 0.002

x 7 5 10 9 4

Step 3

Step 4