Dimensionality Reduction in Machine Learning
In order to get valuable intelligence from sizable datasets, it is vital to collect, analyze, and display data in the right ways. Data is now an integral part of corporations across all sectors. That said, it does not always mean that more quantity leads to higher accuracy or productivity. More the data generated, the harder it is to interpret and understand it in order to make reliable judgements.
The dimensionality of a dataset is the number of factors or attributes in it. Dimensionality reduction methods aim to minimize the number of input variables in a dataset.
Curse of dimensionality
The “curse of dimensionality”—the difficulty in handling high-dimensional data—is a well-known phenomenon. Any machine learning technique and model get increasingly sophisticated as the dimensionality of the input data rises. More the variables or components (also known as parameters) in a set of features, the more challenging it is to see and operate with the training dataset. The fact that the vast majority of variables are often connected seems to be another crucial thing to keep in mind. Therefore, if you incorporate every item in the feature set in your training dataset, many of them will be superfluous.
As the number of characteristics rises, the number of samples rises correspondingly as well, raising the possibility of overfitting.
Dimensionality Reduction
Dimensionality reduction, put simply, is the process of lowering the dimension of a data feature collection. In a three-dimensional domain, a dataset typically consists of an array of points or hundreds of columns, or features. You may reduce the number of columns to measurable quantities by using dimensionality reduction, which will reduce the three-dimensional sphere to a two-dimensional circle.
Dimensionality Reduction Techniques
All the techniques used in dimensionality reduction can be classified into 2 major categories, explained below:
1. Feature selection:
A small sample of the input characteristics from the entire data (those that are most pertinent) is sought after using the feature selection approach. There are three ways for feature selection, namely:
- Filtering approach
- Wrapper approach
- Embedded approach
2. Feature extraction:
Data is transformed from a high-dimensional domain into one that has less dimensions via feature extraction, also often known as feature projection. This data processing might be linear or nonlinear, or both. With the use of this methodology, a smaller number of new variables that individually combine input variables are discovered with no information loss.
Advantages of Dimensionality Reduction
The following are some advantages of using the dimensionality reduction approach on the provided dataset:
1. Requires less space: The space needed to store data is significantly decreased by lowering the dimensionality of the features.
2. Requires less time: Reduced feature dimensions call for shorter computation training times.
3. Requires less effort: The dataset’s characteristics with reduced dimensions make the data easier to see rapidly.
4. Reduced number of variables: By taking care of the multicollinearity, it eliminates the redundant features, if any.
Disadvantages of Dimensionality Reduction:
The following list of drawbacks of using the dimensionality reduction includes:
1. Data loss: The decrease in dimensionality may result in some data loss.
2. Unknown factors: Sometimes, the primary components that are needed to be considered in the PCA dimensionality reduction approach are unknown.
Dimensionality Reduction Approaches:
1. Principal Component Analysis (PCA):
Among the most popular linear dimensionality reduction approaches is principal component analysis. In order to optimize the variation of the information in the low-dimensional format, this approach directly maps the data to a space with fewer dimensions.
In essence, it is a statistical process that two orthogonal transforms a dataset’s ‘n’ dimensions into a fresh set of n dimensions, referred to as the main constituents. The first main component that is produced as a result of this conversion has the most variance. If a main component is orthogonal (i.e., not correlated) to those that came before it, it will have the largest variance imaginable.
2. Backward Feature Elimination:
It is necessary to start with all ‘n’ dimensions when using the backward feature removal strategy. As a result, you may train a specific classification algorithm on n input characteristics at a given repetition. Now you must build the very same algorithm on n-1 variables n times while removing one input feature at a time. Then you eliminate the input variable that results in the least gain in error rate, leaving n-1 input features behind. Once no additional variable can be eliminated, you restart the classification process with n-2 characteristics.
A model trained on n-k features and having an error rate of e is produced after each iteration (k) (k). In order to identify the least number of parameters required to achieve the performance of the classifier using the provided algorithm, you must first choose the highest tolerable error rate.
3. Random Forest:
An effective feature selection approach for machine learning is called Random Forest. It is unnecessary to build a separate feature significance module because this algorithm already has one. With the aid of using data for each variable, we may identify the set of features in this strategy by creating a huge collection of trees against the attribute value.
The incoming data must be hot encoded into numeric data since the random forest method only accepts numeric attributes.
4. Factor Analysis:
By keeping each parameter inside a category based on its correlation with these other factors, a method known as factor analysis allows for the possibility of strong correlations among category members but low correlations with members of other categories.
An illustration of it would be if we had two variables, Vegetables and recipes. Because of the strong association between these two factors, more money is spent by those with higher earnings and vice versa. As a result, these variables are grouped together to form the component. In comparison to the dataset’s initial dimension, there will be fewer of these components.
5. Auto-encoders:
The auto-encoder, a form of ANN or artificial neural network, is a prominent technique for dimensionality reduction. Its primary goal is to replicate the input to the results. This process compresses the input into a latent space before utilizing that representation to produce the output. This is primarily in two parts:
- Encoder: The encoder’s job is to condense the input so that it may be represented in latent space.
- Decoders: Reproducing the output from the latent-space representation is the decoder’s main duty.
6. Linear discriminant analysis (LDA):
Widely used in statistics, pattern recognition, and machine learning, linear discriminant analysis is an extension of Fisher’s linear discriminant approach. Finding a linear arrangement of characteristics that can distinguish between two or more classes of objects is the goal of the LDA approach. With LDA, class separability is maximized in the representation of data. While projection allows for the juxtaposition of items of the same class, it places objects of different classes far apart from one another.
7. Forward feature elimination:
The backward feature deletion approach is the reverse of the forward feature development. When using the forward feature creation approach, you start with one feature and add others, one at a point to advance. It takes a lot of time and calculation to do backward feature deletion and forward feature building. These techniques work well with data that just have a few input variables.
8. Generalized discriminant analysis (GDA):
Generalized discriminant analysis can be described as a form of nonlinear discriminant analysis. It is only different because it makes use of the kernel function operators. Because of how strongly its basic principle resembles that of support vector machines (SVM), the GDA approach aids in the mapping of input variables into higher dimensional space. GDA maximizes the proportion of between-class scatter to within-class scatter in order to discover projections for data in a lower-dimensional domain, just like the LDA method does.
Conclusion
There is no optimal method for dimensionality reduction, as well as no technique-to-problem mapping. Therefore, the ideal strategy is to conduct systematic, well-controlled trials to determine which dimensionality reduction methods, when combined with your preferred model, produce the greatest results on your data.