Advanced Transform - Part 2 - How to Transform the Target in Regression

On regression predictive modeling problems where a numerical value must be predicted, it can also be critical to scale and perform other data transformations on the target variable. This can be achieved in Python using the TransformedTargetRegressor class.

In this tutorial, you will discover how to use the TransformedTargetRegressor to scale and transform target variables for regression using the scikit-learn Python machine learning library.

After completing this tutorial, you will know:

• The importance of scaling input and target data for machine learning.
• The two approaches to applying data transforms to target variables.
• How to use the TransformedTargetRegressor on a real regression dataset

This tutorial is divided into three parts; they are:

• Importance of Data Scaling
• How to Scale Target Variables
• Example of Using the TransformedTargetRegressor

A. Importance of Data Scaling

It is common to have data where the scale of values differs from variable to variable.  For example, one variable may be in feet, another in meters, and so on. 

Some machine learning algorithms perform much better if all of the variables are scaled to the same range, such as scaling all variables to values between 0 and 1, called normalization. 

This effects algorithms that use a weighted sum of the input, like linear models and neural networks, as well as models that use distance measures such as support vector machines and k-nearest neighbors.

As such, it is a good practice to scale input data, and perhaps even try other data transforms such as making the data more normal (better fit a Gaussian probability distribution) using a power transform.

This also applies to output variables, called target variables, such as numerical values that are predicted when modeling regression predictive modeling problems. For regression problems, it is often desirable to scale or transform both the input and the target variables.

...
# prepare the model with input scaling
pipeline = Pipeline(steps=[('normalize', MinMaxScaler()), ('model', LinearRegression())])
# fit pipeline
pipeline.fit(train_x, train_y)
# make predictions
yhat = pipeline.predict(test_x)

B. How to Scale Target Variables

There are two ways that you can scale target variables. The first is to manually manage the transform, and the second is to use a new automatic way for managing the transform.

• Manually transform the target variable
• Automatically transform the target variable

1. Manual transform of the Target Variable

Manually managing the scaling of the target variable involves creating and applying the scaling object to the data manually. It involves the following steps:

• Create the transform object, e.g. a MinMaxScaler.
• Fit the transform on the training dataset.
• Apply the transform to the train and test datasets
• Invert the transform on any predictions made.

...
# create target scaler object
target_scaler = MinMaxScaler()
target_scaler.fit(train_y)

...
# transform target variables
train_y = target_scaler.transform(train_y)
test_y = target_scaler.transform(test_y)

...
# invert transform on predictions
yhat = model.predict(test_X)
yhat = target_scaler.inverse_transform(yhat)

2. Automatic Transform of the Target Variable

An alternate approach is to automatically manage the transform and inverse transform. This can be achieved by using the TransformedTargetRegressor object that wraps a given model and a scaling object.

It will prepare the transform of the target variable using the same training data used to fit the model, then apply that inverse transform on any new data provided when calling fit(), returning predictions in the correct scale.

To use the TransformedTargetRegressor, it is defined by specifying the model and the transform object to use on the target; for example:

...
# define the target transform wrapper
wrapped_model = TransformedTargetRegressor(regressor=model, transformer=MinMaxScaler())

Later, the TransformedTargetRegressor instance can be fit like any other model by calling the fit() function and used to make predictions by calling the predict() function.

...
# use the target transform wrapper
wrapped_model.fit(train_X, train_y)
yhat = wrapped_model.predict(test_X)

This is much easier and allows you to use helpful functions like cross val score() to evaluate a model.

C. Example of Using the TransformedTargetRegressor

In this section, we will demonstrate how to use the TransformedTargetRegressor on a real dataset. We will use the Boston housing regression problem that has 13 inputs and one numerical target and requires learning the relationship between suburb characteristics and house prices.

We can now prepare an example of using the TransformedTargetRegressor. A naive regression model that predicts the mean value of the target on this problem can achieve a mean absolute error (MAE) of about 6.659. We will aim to do better. In this example, we will fit a HuberRegressor class (a type of linear regression robust to outliers) and normalize the input variables using a Pipeline.

# example of normalizing input and output variables for regression.
from numpy import mean
from numpy import absolute
from numpy import loadtxt
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedKFold
from sklearn.pipeline import Pipeline
from sklearn.linear_model import HuberRegressor
from sklearn.preprocessing import MinMaxScaler
from sklearn.compose import TransformedTargetRegressor
# load data
dataset = loadtxt('housing.csv', delimiter=",")
# split into inputs and outputs
X, y = dataset[:, :-1], dataset[:, -1]
# prepare the model with input scaling
pipeline = Pipeline(steps=[('normalize', MinMaxScaler()), ('model', HuberRegressor())])
# prepare the model with target scaling
model = TransformedTargetRegressor(regressor=pipeline, transformer=MinMaxScaler())
# evaluate model
cv = RepeatedKFold(n_splits=10, n_repeats=3, random_state=1)
scores = cross_val_score(model, X, y, scoring='neg_mean_absolute_error', cv=cv, n_jobs=-1)
# convert scores to positive
scores = absolute(scores)
# summarize the result
s_mean = mean(scores)
print('Mean MAE: %.3f' % (s_mean))

-----Result-----

Mean MAE: 3.203

In this case, we achieve a MAE of about 3.2, much better than a naive model that achieved about 6.6.

We are not restricted to using scaling objects; for example, we can also explore using other data transforms on the target variable, such as the PowerTransformer, that can make each variable more-Gaussian-like (using the Yeo-Johnson transform) and improve the performance of linear models. 

By default, the PowerTransformer also performs a standardization of each variable after performing the transform.

# example of power transform input and output variables for regression.
from numpy import mean
from numpy import absolute
from numpy import loadtxt
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedKFold
from sklearn.pipeline import Pipeline
from sklearn.linear_model import HuberRegressor
from sklearn.preprocessing import PowerTransformer
from sklearn.preprocessing import MinMaxScaler
from sklearn.compose import TransformedTargetRegressor
# load data
dataset = loadtxt('housing.csv', delimiter=",")
# split into inputs and outputs
X, y = dataset[:, :-1], dataset[:, -1]
# prepare the model with input scaling and power transform
steps = list()
steps.append(('scale', MinMaxScaler(feature_range=(1e-5,1))))
steps.append(('power', PowerTransformer()))
steps.append(('model', HuberRegressor()))
pipeline = Pipeline(steps=steps)
# prepare the model with target scaling
model = TransformedTargetRegressor(regressor=pipeline, transformer=PowerTransformer())
# evaluate model
cv = RepeatedKFold(n_splits=10, n_repeats=3, random_state=1)
scores = cross_val_score(model, X, y, scoring='neg_mean_absolute_error', cv=cv, n_jobs=-1)
# convert scores to positive
scores = absolute(scores)
# summarize the result
s_mean = mean(scores)
print('Mean MAE: %.3f' % (s_mean))

-----Result-----

Mean MAE: 2.972

In this case, we see further improvement to a MAE of about 2.9.