Model Evaluation - Tour of Model Evaluation Metrics

A classifier is only as good as the metric used to evaluate it. If you choose the wrong metric to evaluate your models, you are likely to choose a poor model.

In this tutorial, you will discover metrics that you can use for imbalanced classification. After completing this tutorial, you will know:

1. About the challenge of choosing metrics for classification, and how it is particularly difficult when there is a skewed class distribution.
2. How there are three main types of metrics for evaluating classifier models, referred to as rank, threshold, and probability.
3. How to choose a metric for imbalanced classification if you don’t know where to start.

Foundation - Challenge of Imbalanced Classification

In this tutorial, you will discover data characteristics that compound the challenge of imbalanced classification. After completing this tutorial, you will know:

1. Imbalanced classification is specifically hard because of the severely skewed class distribution and the unequal misclassification costs.
2. The difficulty of imbalanced classification is compounded by properties such as dataset size, label noise, and data distribution.
3. How to develop an intuition for the compounding effects on modeling difficulty posed by different dataset properties.

Foundation - Intuition for Imbalanced Classification

In this tutorial, you will discover how to develop a practical intuition for imbalanced and highly skewed class distributions.

After completing this tutorial, you will know:

1. How to create a synthetic dataset for binary classification and plot the examples by class.
2. How to create synthetic classification datasets with any given class distribution.
3. How different skewed class distributions actually look in practice.

Foundation - What is Imbalanced Classification?

Imbalanced classification is the problem of classification when there is an unequal distribution of classes in the training dataset.

The imbalance in the class distribution may vary, but a severe imbalance is more challenging to model and may require specialized techniques.

Many real-world classification problems have an imbalanced class distribution, such as fraud detection, spam detection, and churn prediction.

Project - Regression Machine Learning Case Study

How do you work through a predictive modeling machine learning problem end-to-end? In this lesson you will work through a case study regression predictive modeling problem in Python including each step of the applied machine learning process. After completing this project, you will know:

• How to work through a regression predictive modeling problem end-to-end
• How to use data transforms to improve model performance 
• How to use algorithm tuning to improve model performance
• How to use ensemble methods and tuning of ensemble methods to improve model performance

Project - Predictive Modeling Project Template

Applied machine learning is an empirical skill. You cannot get better at it by reading books and articles. You have to practice. In this lesson you will discover the simple six-step machine learning project template that you can use to jump-start your project in Python. After completing this lesson you will know:

1. How to structure an end-to-end predictive modeling project.

2. How to best use the structured project template to ensure an accurate result for your dataset.

Project 1 - Monthly Sales of French Champagne

We will work through a time series forecasting project from end-to-end, from downloading the dataset and defining the problem to training a final model and making predictions. This project is not exhaustive, but shows how you can get good results quickly by working through a time series forecasting problem systematically.

The steps of this project that we will through are as follows.

1. Problem Description.
2. Test Harness.
3. Persistence.
4. Data Analysis.
5. ARIMA Models.
6. Model Validation.

Guide For Time Series Forecast Projects

A time series forecast process is a set of steps or a recipe that leads you from defining your problem through to the outcome of having a time series forecast model or set of predictions.

In this lesson, you will discover time series forecast processes that you can use to guide you through your forecast project. After reading this lesson, you will know:

• The 5-Step forecasting task by Hyndman and Athanasopoulos to guide you from problem definition to using and evaluating your forecast model.
• The iterative forecast development process by Shmueli and Lichtendahl to guide you from defining your goal to implementing forecasts.
• Suggestions and tips for working through your own time series forecasting project.

Forecast Models - Part 8 - Forecast Confidence Intervals

Confidence intervals provide an upper and lower expectation for the real

observation. These can be useful for assessing the range of real possible outcomes for a prediction and for better understanding the skill of the model.
In this tutorial, you will discover how to calculate and interpret confidence intervals for time series forecasts with Python.
Specifically, you will learn:

• How to make a forecast with an ARIMA model and gather forecast diagnostic information.
• How to interpret a confidence interval for a forecast and configure different intervals.
• How to plot the confidence interval in the context of recent observations.

Forecast Models - Part 7 - Save Models and Make Predictions

Selecting a time series forecasting model is just the beginning. Using the chosen model in practice can pose challenges, including data transformations and storing the model parameters on disk. 

In this tutorial, you will discover how to finalize a time series forecasting model and use it to make predictions in Python. 

After completing this tutorial, you will know:

• How to finalize a model and save it and required data to file.
• How to load a finalized model from file and use it to make a prediction.
• How to update data associated with a finalized model in order to make subsequent predictions.