1. Introduction to Whitening

Whitening (or sphering) is a preprocessing transformation that converts a vector of random variables with a known covariance matrix into a new vector whose covariance is the Identity Matrix ($I$).

Two goals of whitening:

  • Decorrelation — all features become linearly independent.
  • Unit Variance — every feature is scaled to have a variance of 1.

The Whitening Pipeline: from correlated data through rotation and scaling to spherical (whitened) data

2. The Core Math: Eigenvalue Decomposition (EVD)

The foundation of whitening is the decomposition of the Covariance Matrix ($\Sigma$):

$$\Sigma = U \Lambda U^{\top}$$
Symbol Meaning Geometric Role
$U$ Eigenvector matrix Rotation
$\Lambda$ Diagonal eigenvalue matrix Variance along principal axes
$U^{\top}$ Transpose of $U$ Inverse rotation

Think of it as: rotate, scale, rotate back. The covariance matrix encodes both the shape and the orientation of your data cloud.

3. The PCA Whitening Process

PCA Whitening rotates the data into the principal component space and then rescales it.

$$X_{PCA\text{-}white} = \Lambda^{-1/2} U^{\top} X_{centered}$$

Steps for PCA Whitening

Given a dataset $X$ (where each column is a feature/dimension and each row is a sample), here are the steps to compute the PCA whitened data $X_{PCA\text{-}white}$:

Step 1: Zero-Mean the Data

Subtract the mean of each feature (column) to center the data around the origin.

\[X_{centered} = X - \mu\]

where $\mu$ is the mean vector of the features in $X$.

Step 2: Compute the Covariance Matrix ($\Sigma$)

Calculate the empirical covariance matrix of the centered data:

\[\Sigma = \frac{1}{N} X_{centered}^{\top} X_{centered}\]

where $N$ is the number of samples (rows).

Step 3: Eigendecomposition of $\Sigma$

Compute the eigendecomposition of the symmetric covariance matrix $\Sigma$:

\[\Sigma = U \Lambda U^{\top}\]
  • $U$: The orthogonal matrix whose columns are the eigenvectors (principal components) of $\Sigma$. This matrix represents the rotation.
  • $\Lambda$: A diagonal matrix whose diagonal entries are the eigenvalues ($\lambda_i$) of $\Sigma$. These values represent the variance along each principal component direction.

Step 4: Compute the PCA Whitening Matrix ($W_{PCA}$)

The whitening transformation matrix is formed by:

\[W_{PCA} = \Lambda^{-1/2} U^{\top}\]

where $\Lambda^{-1/2}$ is a diagonal matrix containing the reciprocal of the square root of each eigenvalue (i.e., $\frac{1}{\sqrt{\lambda_i}}$).

Numerical stability: A small constant $\epsilon$ is often added to the eigenvalues before taking the reciprocal square root — $\frac{1}{\sqrt{\lambda_i + \epsilon}}$ — to prevent division by zero when an eigenvalue is close to zero.

Step 5: Apply the Transformation

Apply the whitening matrix to the centered data to get the PCA-whitened data:

\[X_{PCA\text{-}white} = W_{PCA} \cdot X_{centered} = \Lambda^{-1/2} U^{\top} X_{centered}\]

Why the scaling factor $\Lambda^{-1/2}$?

  1. After rotation ($U^{\top} X$), the variance along each axis is given by $\Lambda$.
  2. To normalize each axis to unit variance, we multiply by the inverse square root of the eigenvalues.
  3. This gives us: $\text{Cov}(\Lambda^{-1/2} U^{\top} X) = \Lambda^{-1/2} \Lambda \Lambda^{-1/2} = I$.

4. ZCA Whitening (Zero-Phase Component Analysis)

ZCA goes one step further by rotating the data back to its original orientation after whitening.

$$X_{ZCA\text{-}white} = U \Lambda^{-1/2} U^{\top} X_{centered}$$

Why ZCA? PCA whitening “scrambles” the original feature space into abstract components. ZCA whitens the data while staying as close as possible to the original representation — making it ideal for images.

Property Description
Identity Covariance Like PCA whitening, the final covariance is $I$.
Minimal Distortion It stays as close to the original data as possible.
Structure Preservation Excellent for images — doesn’t scramble pixels into abstract components.

5. Summary Table

Operation Formula Effect
Centering $X - \mu$ Originates data at zero.
Rotation $U^{\top} X$ Decorrelates data (PCA Scores).
PCA Whitening $\Lambda^{-1/2} U^{\top} X$ Decorrelates + Unit Variance.
ZCA Whitening $U \Lambda^{-1/2} U^{\top} X$ Decorrelates + Unit Variance + Original Orientation.

6. Frequently Asked Questions

Q1: Why are we rescaling in PCA Whitening?

The rescaling step is essential to achieve the second part of the “whitening” goal: unit variance.

The full PCA whitening process involves two main linear transformation steps:

  1. Rotation (Decorrelation): The data is rotated into the principal component basis. This step uses the matrix of eigenvectors ($U$) to align the data with its principal axes, which removes all linear correlation between the new features (components). The covariance matrix of the rotated data, $U^{\top} X$, is now a diagonal matrix $\Lambda$, where the diagonal elements ($\lambda_i$) are the eigenvalues (variances) of the data along each component axis.
  2. Rescaling (Normalization): The data is rescaled by dividing each component by the square root of its own variance (the corresponding eigenvalue).

This division ensures that the variance of every component is equal to $1$. The transformation matrix includes $\Lambda^{-1/2}$, which is the diagonal matrix of these $\frac{1}{\sqrt{\lambda_i}}$ factors.

Purpose of Rescaling:

  • Achieve Identity Covariance: The ultimate goal of whitening is to transform the data so that its final covariance matrix is the Identity Matrix ($I$). The rotation step gives you a diagonal covariance matrix ($\Lambda$), but the diagonal entries are the original variances ($\lambda_i$). The rescaling step changes these entries from $\lambda_i$ to $1$. The final covariance matrix of the whitened data, $\text{Cov}(X_{white})$, is $I$.
  • Equalize Feature Importance: By ensuring all features have unit variance, the rescaling prevents features with large natural scales (large $\lambda_i$) from numerically dominating the downstream machine learning algorithm. It makes all features equally important in terms of their variance, which can stabilize training and improve the performance of models that assume spherical data (like certain clustering or neural network algorithms).
Q2: Can we do dimensionality reduction in whitening?

Yes, you can, and in practice, you often do.

PCA whitening inherently contains the mechanism for dimensionality reduction: PCA (Principal Component Analysis).

How Dimensionality Reduction is Performed:

The eigenvalues ($\lambda_i$) computed during the PCA step represent the amount of variance captured by their corresponding principal components.

  1. Identify Informative Components: The eigenvalues are typically sorted in descending order. You choose to keep only the top $k$ principal components that capture a sufficient amount of the total variance (e.g., 95% or 99%).
  2. Truncate the Matrices: Instead of using the full eigenvector matrix $U$ and eigenvalue matrix $\Lambda$ (both $d \times d$ for $d$ dimensions), you only keep the first $k$ columns of $U$ and the first $k$ diagonal elements of $\Lambda$.
  3. Apply the Transformation: You then apply the whitening transformation using these reduced $k$-dimensional matrices.

The resulting whitened data will have $k$ dimensions (features), where $k < d$, and will still be decorrelated with unit variance.

Therefore, PCA whitening is frequently used as a pre-processing step to both decorrelate and reduce the dimensionality of the data before feeding it into a subsequent algorithm like a neural network or a classifier.

Q3: What is the meaning of the decomposition formula $A = U \Lambda U^{\top}$?

The expression $A = U \Lambda U^{\top}$ is the most common form of the Eigenvalue Decomposition (EVD) for real symmetric matrices (like a covariance matrix, $\Sigma$).

This formula states that a square, symmetric matrix $A$ can be factored into a product of three matrices:

  • $U$ (The Eigenvector Matrix):
    • This is an orthogonal matrix whose columns are the eigenvectors of $A$.
    • Since it is orthogonal, its transpose is its inverse: $U^{\top} = U^{-1}$. This is why the decomposition is often written as $A = P D P^{-1}$ for non-symmetric matrices, but simplifies to $A = U \Lambda U^{\top}$ for symmetric matrices.
    • Geometric Meaning: The matrix $U$ represents a rotation (or reflection) that aligns the coordinate axes with the principal axes of the data defined by $A$.
  • $\Lambda$ (Lambda - The Eigenvalue Matrix):
    • This is a diagonal matrix whose diagonal entries are the eigenvalues ($\lambda_i$) of $A$.
    • Geometric Meaning: The eigenvalues represent the scaling factors (the variance) along each new coordinate axis defined by the eigenvectors in $U$.
  • $U^{\top}$ (U Transpose):
    • This is the transpose of the eigenvector matrix, which is also its inverse ($U^{-1}$).
    • Geometric Meaning: This represents the inverse rotation needed to return to the original coordinate system.
Q4: What is the significance of EVD in data analysis (like PCA)?

The decomposition $A = U \Lambda U^{\top}$ is incredibly important because it allows you to interpret the complex action of the matrix $A$ as a simple sequence of geometric transformations:

  1. $U^{\top}$: Rotate the vector $x$ into the basis of the eigenvectors.
  2. $\Lambda$: Scale the rotated vector independently along each new axis (by the eigenvalues).
  3. $U$: Rotate the result back to the original coordinate system.

In the context of Principal Component Analysis (PCA), where $A$ is the covariance matrix ($\Sigma$), this decomposition is the foundation:

  1. The columns of $U$ are the Principal Components (PCs).
  2. The diagonal entries of $\Lambda$ are the variances along those PCs.
Q5: Why is centering (zero-meaning) required in PCA?

Centering is the process of subtracting the mean of each feature from the dataset. In PCA, this is a non-negotiable preprocessing step because it ensures that the algorithm focuses on the internal structure (variance) of the data rather than its absolute position in space.

1. The Mathematical Requirement of Covariance

PCA is based on the Covariance Matrix ($\Sigma$). The statistical formula for covariance between two variables $X$ and $Y$ is:

\[\text{Cov}(X, Y) = \frac{1}{N} \sum_{i=1}^{N} (x_i - \bar{x})(y_i - \bar{y})\]

Notice that the formula explicitly subtracts the means ($\bar{x}$ and $\bar{y}$). In matrix notation, we often simplify the covariance calculation to:

\[\Sigma = \frac{1}{N} X^{\top} X\]

This simplified matrix multiplication only works if the data is already centered (where $\bar{x} = 0$). If you skip centering, $X^{\top} X$ calculates the “raw moments” (distance from the origin) rather than the variance (distance from the mean), leading to an incorrect covariance matrix.

2. Geometric Alignment of Principal Components

PCA seeks to find the directions (eigenvectors) of maximum spread. These vectors must originate from the origin $(0, 0)$.

  • With Centering: The data cloud is shifted so its center of mass is at $(0, 0)$. The first principal component (PC1) then aligns perfectly with the direction of the greatest spread within the cluster.
  • Without Centering: If the data cluster is far from the origin, the first principal component will likely point from the origin directly toward the “center” of the data cluster. Instead of capturing the variance (the shape of the cloud), it captures the offset (the position of the cloud).

3. Orthogonality and Basis Transformation

PCA creates a new coordinate system. For the new axes to be valid rotations of the original axes, they must share the same origin. If the data isn’t centered, the “rotation” performed by the eigenvector matrix $U$ will be performed around the coordinate $(0, 0)$ rather than the center of the data, which “scrambles” the relationship between the features and makes the resulting components uninterpretable.

4. Impact on Reconstruction

If you use PCA for dimensionality reduction and then try to reconstruct the original data, the reconstruction will be fundamentally broken if you didn’t center. PCA assumes that the data is a linear combination of components plus the mean. If the mean isn’t handled correctly at the start, you lose the baseline needed to map the compressed data back into its original space.