How decision trees are used in Machine Learning?

In machine learning, decision trees are used for both classification and regression tasks. They are part of a class of algorithms called supervised learning algorithms, which means they learn from labeled training data to make predictions or decisions without being explicitly programmed to perform the task.

Here's a brief overview of how decision trees are used in machine learning:

Classification: Decision trees are commonly used for classification problems, where the goal is to predict which category a new observation belongs to base on its features. The decision tree algorithm builds a tree-like model of decisions based on the features in the training data. Each node in the tree represents a feature, each branch represents a decision rule, and each leaf represents an outcome, or class label. When a new observation is made, the algorithm starts at the root of the tree and moves down the tree following the decision rules that match the observation's features until it reaches a leaf, which gives the predicted class label.

Regression: Decision trees can also be used for regression problems, where the goal is to predict a continuous value (like the price of a house) rather than a category. The decision tree algorithm works in much the same way as for classification, but instead of predicting a class label at each leaf, it predicts a numerical value.

Feature Selection: Decision trees are also useful for selecting relevant features in your data. Features used at the top of the tree are typically the most important, as they contribute to the majority of the decision-making process. This can be useful for understanding which features are driving the predictions of your model, or for reducing the dimensionality of your data.

Ensemble Methods: Decision trees are often used as building blocks for more powerful machine learning models. For example, the Random Forest and Gradient Boosting algorithms work by training many decision trees on different subsets of the data and combining their predictions.

In Python, the sklearn library provides the DecisionTreeClassifier and DecisionTreeRegressor classes for building decision tree models.

Download the Jupyter Notebook

Example of DecisionTreeClassifier

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score

# Load the iris dataset
iris = load_iris()
X = iris.data
y = iris.target

# Split the data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Initialize the Decision Tree Classifier
dtc = DecisionTreeClassifier()

# Fit the model on the training data
dtc.fit(X_train, y_train)

# Make predictions on the test data
predictions = dtc.predict(X_test)

# Calculate the accuracy of the model
accuracy = accuracy_score(y_test, predictions)

 # Print the accuracy
print(f'Accuracy: {accuracy}')

 

Download the Jupyter Notebook

Example of DecisionTreeRegressor

from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import mean_squared_error

 

# Load the Boston housing dataset

boston = load_boston()
X = boston.data
y = boston.target

# Split the data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Initialize the Decision Tree Regressor
dtg = DecisionTreeRegressor()

# Fit the model on the training data
dtg.fit(X_train, y_train)

# Make predictions on the test data
predictions = dtg.predict(X_test)

# Calculate the mean squared error of the model
mse = mean_squared_error(y_test, predictions)

# Print the mean squared error
print(f'Mean Squared Error: {mse}')

 

Important Points about DecisionTreeRegressor

In regression tasks, accuracy is not a suitable metric because the output is a continuous value, not a binary or multi-class label. Instead, we use other metrics such as Mean Squared Error (MSE), Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), or R-squared (R²).

However, if you want to evaluate the performance of a DecisionTreeRegressor, you can use the R-squared (R²) metric, which provides a measure of how well future samples are likely to be predicted by the model. The best possible score is 1.0.

 

from sklearn.metrics import r2_score 

# Calculate R-squared

r2 = r2_score(y_test, predictions)

print(f'R-squared: {r2}')

 

What is Mean Squared Error?

Mean Squared Error (MSE) is a common metric used to evaluate the performance of regression models. It measures the average squared difference between the actual and predicted values, hence the name "Mean Squared Error".

Mathematically, it can be expressed as:

MSE = (1/n) * Σ(actual - prediction)²

where:

ü  n is the total number of observations

ü  actual is the actual value

ü  prediction is the predicted value

ü  Σ is the sum over all observations

The squaring is done so negative values do not cancel positive values. The smaller the Mean Squared Error, the closer the fit is to the data.