disc13_gsi

Data 100, Spring 2024 Discussion #13 Solutions Note: Your TA will probably not cover all the problems. This is fine; the discussion worksheets are not designed to be finished within an hour. They are deliberately made slightly longer so they can serve as resources you can use to practice, reinforce, and build upon concepts discussed in lectures, labs, and homework. PCA Basics 1. Consider the following dataset, where X is the corresponding 4 × 3 design matrix. The mean and variance for each of the features are also provided. Observations Feature 1 Feature 2 Feature 3 1 -3.59 7.39 -0.78 2 -8.37 -5.32 0.90 3 1.75 -0.61 -0.62 4 10.21 -1.46 0.50 Mean 0 0 0 Variance 47.56 21.35 0.51 Suppose we perform a singular value decomposition (SVD) on this data to obtain X = USV T : ( Note: U and V T are not perfectly orthonormal due to rounding to 2 decimal places. ) U =     − 0 . 25 0 . 81 0 . 20 − 0 . 61 − 0 . 56 0 . 24 0 . 13 − 0 . 06 − 0 . 85 0 . 74 − 0 . 18 0 . 41     , S =   13 . 79 0 0 0 9 . 32 0 0 0 0 . 81   , V T =   1 . 00 0 . 02 0 . 00 − 0 . 02 0 . 99 − 0 . 13 0 . 00 0 . 13 0 . 99   (a) Recall that XV contains the principal components of dataset X . and that we can alterna- tively calculate it using US . Prove, using the definition from lecture, that XV = US . Solution: Since V is orthonormal, we know that V T V = I . Starting with X = USV T , we right multiply by V on both sides: XV = USV T V = USI = US This completes the proof. Staff Notes: This is a great time to stress the properties of the U, Σ , V T . For example, V T being orthonormal means V − 1 = V T , what S being a diagonal matrix means, etc. 1

Discussion #13 2 (b) Compute the vector for the first principal component (round to 2 decimal places). Solution: We compute the first principal component by multiplying X by the first row of V T to get ≈ − 3 . 44 − 8 . 47 1 . 74 10 . 18 T (your values may differ slightly due to rounding). You can also compute the first PC by observing that XV = US . Therefore, the first principal component is also the first column of US . (c) What is the component score of the first principal component? In other words, how much variance does it capture of the original data X ? Solution: The variance captured by i -th principal component of the original data X is equal to ( i -th singular value ) 2 number of observations n In this case, n = 4 , and σ 1 = 13 . 79 . Therefore, the component score can be computed as follows: 13 . 79 2 4 = 47 . 54 (d) (Bonus) Given the results of (a), how can we interpret the rows of V T ? What do the values in these rows represent? Solution: Each principal component of X is a linear combination of X ’s features. The rows of V T correspond to the weights of each feature in the linear combinations that make up their respective principal components.