Dimensionality Reduction - Part 5 - Practical Approach to Dimensionality Reduction Using PCA, LDA and Kernel PCA

Dimensionality reduction is an important approach in machine learning. A large number of features available in the dataset may result in overfitting of the learning model. To identify the set of significant features and to reduce the dimension of the dataset, there are three popular dimensionality reduction techniques that are used. 

In this article, we will discuss the practical implementation of these three dimensionality reduction techniques:

• Principal Component Analysis (PCA)
• Linear Discriminant Analysis (LDA), and
• Kernel PCA (KPCA)
• Comparison of PCA, LDA and Kernel PCA

Dimensionality Reduction - Part 3 - How to Perform PCA Dimensionality Reduction

The most popular technique for dimensionality reduction in machine learning is Principal Component Analysis (PCA). This is a technique that comes from the field of linear algebra and can be used as a data preparation technique to create a projection of a dataset prior to fitting a model. 

In this tutorial, you will discover how to use PCA for dimensionality reduction when developing predictive models. After completing this tutorial, you will know:

• PCA is a technique from linear algebra that can be used to automatically perform dimensionality reduction.
• How to evaluate predictive models that use a PCA projection as input and make predictions with new raw data.

Dimensionality Reduction - Part 2 - How to Perform LDA Dimensionality Reduction

Linear Discriminant Analysis (LDA) is a predictive modeling algorithm for multiclass classification.

It can also be used as a dimensionality reduction technique, providing a projection of a training dataset that best separates the examples by their assigned class.

The ability to use Linear Discriminant Analysis for dimensionality reduction often surprises most practitioners. 

Dimensionality Reduction - Part 1 - What is Dimensionality Reduction?

The number of input variables or features for a dataset is referred to as its dimensionality. Dimensionality reduction refers to techniques that reduce the number of input variables in a dataset. More input features often make a predictive modeling task more challenging to model, more generally referred to as the curse of dimensionality.

High-dimensionality statistics and dimensionality reduction techniques are often used for data visualization. Nevertheless these techniques can be used in applied machine learning to simplify a classification or regression dataset in order to better fit a predictive model.

In this tutorial, you will discover a gentle introduction to dimensionality reduction for machine learning.

Advanced Transform - Part 3 - How to Save and Load Data Transforms

It is critical that any data preparation performed on a training dataset is also performed on a new dataset in the future. This may include a test dataset when evaluating a model or new data from the domain when using a model to make predictions. 

Typically, the model fit on the training dataset is saved for later use. The correct solution to preparing new data for the model in the future is to also save any data preparation objects, like data scaling methods, to file along with the model.

In this tutorial, you will discover how to save a model and data preparation object to file for later use. After completing this tutorial, you will know:

• The challenge of correctly preparing test data and new data for a machine learning model.
• The solution of saving the model and data preparation objects to file for later use.
• How to save and later load and use a machine learning model and data preparation model on new data.

Advanced Transform - Part 2 - How to Transform the Target in Regression

On regression predictive modeling problems where a numerical value must be predicted, it can also be critical to scale and perform other data transformations on the target variable. This can be achieved in Python using the TransformedTargetRegressor class.

In this tutorial, you will discover how to use the TransformedTargetRegressor to scale and transform target variables for regression using the scikit-learn Python machine learning library.

After completing this tutorial, you will know:

• The importance of scaling input and target data for machine learning.
• The two approaches to applying data transforms to target variables.
• How to use the TransformedTargetRegressor on a real regression dataset

Advanced Transform - Part 1 - How to Transform Numerical and Categorical Data

Applying data transforms like scaling or encoding categorical variables is straightforward when all input variables are the same type. It can be challenging when you have a dataset with mixed types and you want to selectively apply data transforms to some, but not all, input features.

The scikit-learn Python machine learning library provides the ColumnTransformer that allows you to selectively apply data transforms to different columns in your dataset. In this tutorial, you will discover how to use the ColumnTransformer to selectively apply data transforms to columns in a dataset with mixed data types. 

After completing this tutorial, you will know:

• The challenge of using data transformations with datasets that have mixed data types
• How to define, fit, and use the ColumnTransformer to selectively apply data transforms to columns
• How to work through a real dataset with mixed data types and use the ColumnTransformer to apply different transforms to categorical and numerical data columns

Data Transform - Part 7 - How to Derive New Input Variables

Often, the input features for a predictive modeling task interact in unexpected and often nonlinear ways. These interactions can be identified and modeled by a learning algorithm. Another approach is to engineer new features that expose these interactions and see if they improve model performance.

Transforms like raising input variables to a power can help to better expose the important relationships between input variables and the target variable.

Data Transform - Part 6 - How to Transform Numerical to Categorical Data

Many machine learning algorithms prefer or perform better when numerical input variables have a standard probability distribution. 

The discretization transform provides an automatic way to change a numeric input variable to have a different data distribution, which in turn can be used as input to a predictive model. In this tutorial, you will discover how to use discretization transforms to map numerical values to discrete categories for machine learning. After completing this tutorial, you will know:

• Many machine learning algorithms prefer or perform better when numerical features with non-standard probability distributions are made discrete.
• Discretization transforms are a technique for transforming numerical input or output variables to have discrete ordinal labels.
• How to use the KBinsDiscretizer to change the structure and distribution of numeric variables to improve the performance of predictive models.

Data Transform - Part 5 - How to Change Numerical Data Distributions

Many machine learning algorithms prefer or perform better when numerical input variables and even output variables in the case of regression have a standard probability distribution, such as a Gaussian (normal) or a uniform distribution. 

The quantile transform provides an automatic way to transform a numeric input variable to have a different data distribution, which in turn, can be used as input to a predictive model.

Data Transform - Part 4 - How to Make Distributions More Gaussian

Machine learning algorithms like Linear Regression and Gaussian Naive Bayes assume the numerical variables have a Gaussian probability distribution. Your data may not have a Gaussian distribution and instead may have a Gaussian-like distribution (e.g. nearly Gaussian but with outliers or a skew) or a totally different distribution (e.g. exponential).

As such, you may be able to achieve better performance on a wide range of machine learning algorithms by transforming input and/or output variables to have a Gaussian or more Gaussian distribution.

Power transforms like the Box-Cox transform and the Yeo-Johnson transform provide an automatic way of performing these transforms on your data and are provided in the scikit-learn Python machine learning library.

Data Transform - Part 3 - How to encode Categorical Data

Machine learning models require all input and output variables to be numeric. This means that if your data contains categorical data, you must encode it to numbers before you can fit and evaluate a model. The two most popular techniques are an Ordinal encoding and a One Hot encoding. 

In this tutorial, you will discover how to use encoding schemes for categorical machine learning data. After completing this tutorial, you will know:

• Encoding is a required pre-processing step when working with categorical data for machine learning algorithms.
• How to use ordinal encoding for categorical variables that have a natural rank ordering.
• How to use one hot encoding for categorical variables that do not have a natural rank ordering.

Data Transform - Part 2 - How to scale Data with Outliers

Many machine learning algorithms perform better when numerical input variables are scaled to a standard range. This includes algorithms that use a weighted sum of the input, like linear regression, and algorithms that use distance measures, like k-nearest neighbors. 

Standardizing is a popular scaling technique that subtracts the mean from values and divides by the standard deviation, transforming the probability distribution for an input variable to a standard Gaussian (zero mean and unit variance). Standardization can become skewed or biased if the input variable contains outlier values.

Data Transforms - Part 1 - How to Scale Numerical Data

Many machine learning algorithms perform better when numerical input variables are scaled to a standard range. 

This includes algorithms that use a weighted sum of the input, like linear regression, and algorithms that use distance measures, like k-nearest neighbors.

The two most popular techniques for scaling numerical data prior to modeling are normalization and standardization.

Normalization scales each input variable separately to the range 0-1, which is the range for floating-point values where we have the most precision. Standardization scales each input variable separately by subtracting the mean (called centering) and dividing by the standard deviation to shift the distribution to have a mean of zero and a standard deviation of one.

Feature Selection - Part 6 - How to Use Feature Importance

Feature importance refers to techniques that assign a score to input features based on how useful they are at predicting a target variable.

There are many types and sources of feature importance scores, although popular examples include statistical correlation scores, coefficients calculated as part of linear models, decision trees, and permutation importance scores.

Feature importance scores play an important role in a predictive modeling project, including providing insight into the data, insight into the model, and the basis for dimensionality reduction and feature selection that can improve the efficiency and effectiveness of a predictive model on the problem. In this tutorial, you will discover feature importance scores for machine learning in Python. After completing this tutorial, you will know:

• The role of feature importance in a predictive modeling problem.
• How to calculate and review feature importance from linear models and decision trees.
• How to calculate and review permutation feature importance scores.

Feature Selection - Part 5 - How to Use RFE for Feature Selection

Recursive Feature Elimination, or RFE for short, is a popular feature selection algorithm. RFE is popular because it is easy to configure and use, and because it is effective at selecting those features (columns) in a training dataset that are more or most relevant in predicting the target variable.

In this tutorial, you will discover how to use Recursive Feature Elimination (RFE) for feature selection in Python. After completing this tutorial, you will know:

• RFE is an efficient approach for eliminating features from a training dataset for feature selection.
• How to use RFE for feature selection for classification and regression predictive modeling problems.
• How to explore the number of selected features and wrapped algorithm used by the RFE procedure

Feature Selection - Part 4 - How to Select Features for Numerical Output

The simplest case of feature selection is the case where there are numerical input variables and a numerical target for regression predictive modeling. 

This is because the strength of the relationship between each input variable and the target can be calculated, called correlation, and compared relative to each other. 

In this tutorial, you will discover how to perform feature selection with numerical input data for regression predictive modeling. After completing this tutorial, you will know:

• How to evaluate the importance of numerical input data using the correlation and mutual information statistics.
• How to perform feature selection for numerical input data when fitting and evaluating a regression model.
• How to tune the number of features selected in a modeling pipeline using a grid search.

Feature Selection - Part 3 - How to Select Numerical Input Features

The two most commonly used feature selection methods for numerical input data when the target variable is categorical (e.g. classification predictive modeling) are the ANOVA F-test statistic and the mutual information statistic. 

In this tutorial, you will discover how to perform feature selection with numerical input data for classification. After completing this tutorial, you will know:

• The diabetes predictive modeling problem with numerical inputs and binary classification target variables.
• How to evaluate the importance of numerical features using the ANOVA F-test and mutual information statistics.
• How to perform feature selection for numerical data when fitting and evaluating a classification model.

Feature Selection - Part 2 - How to Select Categorical Input Features

Feature selection is the process of identifying and selecting a subset of input features that are most relevant to the target variable.

Feature selection is often straightforward when working with real-valued data, such as using the Pearson’s correlation coefficient, but can be challenging when working with categorical data.

The two most commonly used feature selection methods for categorical input data when the target variable is also categorical (e.g. classification predictive modeling) are the chi-squared statistic and the mutual information statistic. 

In this tutorial, you will discover how to perform feature selection with categorical input data. 

Feature Selection - Part 1 - What is Feature Selection

Feature selection is a process of reducing the number of input variables when developing a predictive model.

It is desirable to reduce the number of input variables to both reduce the computational cost of modeling and, in many cases, to improve the performance of the model.

Statistical-based feature selection methods involve evaluating the relationship between each input variable and the target variable using statistics and selecting those input variables that have the strongest relationship with the target variable.