Understanding t-SNE: A Tool for High-Dimensional Data Visualization

High-dimensional data can be difficult to understand and visualize because of its complexity. However, techniques like t-SNE (t-Distributed Stochastic Neighbor Embedding) help by transforming this data into a simpler, lower-dimensional form that’s easier to work with. In this post, We’ll explain how t-SNE functions works and why it's such a valuable tool for tasks like clustering, exploring data, and creating visualizations.

What is t-SNE?

t-SNE is a technique used to simplify and visualize complex, high-dimensional data. It focuses on keeping the local structure of the data intact, meaning that points that are close to each other in the original high-dimensional space will stay close in the lower-dimensional version. This makes t-SNE great for spotting clusters or groups within the data.

The process of t-SNE involves two main steps:

• Define pairwise similarities between data points in the high-dimensional space.
• Minimize the divergence between high-dimensional and low-dimensional similarity distributions, generating the low-dimensional visualization.

Step-by-Step Breakdown of t-SNE

Step 1: Building Probabilities in High-Dimensional Space

t-SNE begins by measuring how similar each pair of data points is in the original high-dimensional space. It uses a Gaussian distribution, with the similarity between two points determined by how far apart they are using Euclidean distance.

For two data points \( \mathbf{x}_i \) and \( \mathbf{x}_j \), the probability \( p_{j|i} \) that \( \mathbf{x}_j \) is a neighbor of \( \mathbf{x}_i \) is defined as:

\[ p_{j|i} = \frac{\exp \left( -\frac{\|\mathbf{x}_i - \mathbf{x}_j\|^2}{2\sigma_i^2} \right)}{\sum_{k \ne i} \exp \left( -\frac{\|\mathbf{x}_i - \mathbf{x}_k\|^2}{2\sigma_i^2} \right)} \]

Here:

• \( \|\mathbf{x}_i - \mathbf{x}_j\|^2 \) is the squared Euclidean distance between \( \mathbf{x}_i \) and \( \mathbf{x}_j \).
• \( \sigma_i \) is a parameter that controls the "spread" of the Gaussian distribution, influencing the neighborhood size.

To ensure the similarities are symmetric, the joint probability \( p_{ij} \) is defined as:

\[ p_{ij} = \frac{p_{j|i} + p_{i|j}}{2} \]

This symmetrization ensures that the relationship between points \( \mathbf{x}_i \) and \( \mathbf{x}_j \) reflects mutual similarity.

Step 2: Mapping to Low-Dimensional Space

Once we’ve calculated the probabilities in the high-dimensional space, we then move to a lower-dimensional space (usually 2D or 3D for visualization). In this space, we measure similarities between points using a Student’s t-distribution with 1 degree of freedom, also called the Cauchy distribution. This helps prevent the crowding problem, where too many points get packed together in the lower dimensions.

The similarity in the low-dimensional space is given by:

\[ q_{ij} = \frac{(1 + \|\mathbf{y}_i - \mathbf{y}_j\|^2)^{-1}}{\sum_{k \ne l} (1 + \|\mathbf{y}_k - \mathbf{y}_l\|^2)^{-1}} \]

Where:

• \( \mathbf{y}_i \) and \( \mathbf{y}_j \) are the low-dimensional representations of the points \( \mathbf{x}_i \) and \( \mathbf{x}_j \).
• \( \|\mathbf{y}_i - \mathbf{y}_j\|^2 \) is the squared Euclidean distance in the low-dimensional space.

Step 3: Minimizing Divergence

The final goal of t-SNE is to ensure that the pairwise similarities in the high-dimensional and low-dimensional spaces are as close as possible. This is achieved by minimizing the Kullback-Leibler (KL) divergence between the two distributions:

\[ \text{KL}(P \| Q) = \sum_{i \ne j} p_{ij} \log \frac{p_{ij}}{q_{ij}} \]

Here:

• \( P \) represents the distribution of similarities in the high-dimensional space.
• \( Q \) represents the distribution of similarities in the low-dimensional space.

By using gradient descent, t-SNE adjusts the positions of the points in the low-dimensional space to minimize this divergence, thus preserving local structures.

Example: Visualizing t-SNE

Let’s say you have the following set of 5 data points:

\( \mathbf{X} = \begin{bmatrix} 1 & 2 \\ 2 & 3 \\ 3 & 3 \\ 4 & 2 \\ 5 & 1 \end{bmatrix} \)

1. Compute Pairwise Similarities: First, t-SNE calculates the Euclidean distances between each pair of points.

2. Convert to Probabilities: These distances are then converted into probabilities using the Gaussian distribution.

3. Symmetrize: The probabilities are symmetrized to ensure mutual similarity.

4. Minimize Divergence: The algorithm adjusts the positions of these points in a 2D space to ensure that similar points remain close to each other.

Why t-SNE Works Well

t-SNE is particularly effective for the following reasons:

• Preservation of Local Structure: It maintains the local neighborhoods of the data, making it excellent for tasks like clustering.
• Non-Linear Dimensionality Reduction: Unlike linear methods like PCA, t-SNE can capture non-linear relationships between data points.
• Effective for Visualization: It produces visually interpretable 2D or 3D maps, making it easier to spot patterns and clusters.

When to Use t-SNE

While t-SNE is powerful, it may not be the best choice for all situations:

• High Computational Cost: t-SNE can be computationally expensive and may not scale well with very large datasets.
• Parameter Sensitivity: The results can be sensitive to the choice of parameters, such as perplexity and learning rate.
• Not Ideal for All Data Types: It works best for data with a clear cluster structure and may not be suitable for all types of data.

Conclusion

t-SNE is a great tool for visualizing complex data by keeping local patterns and uncovering hidden structures. Although it can be expensive to compute and requires careful tuning of its settings, its ability to create clear, meaningful visualizations makes it highly useful for data analysis and exploration.