# Hyper-parameters Tuning#

This is a supplement material for the Machine Learning Simplified book. It sheds light on Python implementations of the topics discussed while all detailed explanations can be found in the book.

I also assume you know Python syntax and how it works. If you don’t, I highly recommend you to take a break and get introduced to the language before going forward with my code.

This material can be downloaded as a Jupyter notebook (Download button in the upper-right corner ->

`.ipynb`

) to reproduce the code and play around with it.

This notebook is a supplement for *Chapter 12. Model Tuning and Selection* of **Machine Learning For Everyone** book.

## 1. Required Libraries#

This block imports all necessary libraries. numpy is used for array manipulations, sklearn provides tools for data mining and data analysis, and skopt is used for Bayesian optimization.

```
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, RandomizedSearchCV, GridSearchCV
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score
from skopt import BayesSearchCV
from skopt.space import Integer, Categorical, Real
```

## 2. Generate Synthetic Data#

Here, we generate a synthetic dataset with 1000 samples and 20 features, split into training and test sets. This dataset will be used to train and evaluate our Decision Tree models.

```
# Generate a binary classification dataset.
X, y = make_classification(n_samples=1000, n_features=20, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
```

## 3. Hyperparameter Tuning with GridSearchCV#

GridSearchCV exhaustively searches through the defined parameter grid, evaluating model performance for each combination using cross-validation. The best parameters and their corresponding performance are then displayed.

```
# Define the parameter grid
param_grid = {
'max_depth': [None, 10, 20, 30, 40, 50],
'min_samples_split': [2, 5, 10, 20],
'min_samples_leaf': [1, 2, 4, 6]
}
# Initialize the GridSearchCV object
grid_search = GridSearchCV(estimator=DecisionTreeClassifier(random_state=42), param_grid=param_grid, cv=5, verbose=1, scoring='accuracy')
grid_search.fit(X_train, y_train)
# Best parameters and best score
print("Best parameters found: ", grid_search.best_params_)
print("Best score: ", grid_search.best_score_)
# Evaluate on the test set
y_pred = grid_search.predict(X_test)
print("Accuracy on test set: ", accuracy_score(y_test, y_pred))
```

```
Fitting 5 folds for each of 96 candidates, totalling 480 fits
```

```
Best parameters found: {'max_depth': None, 'min_samples_leaf': 6, 'min_samples_split': 20}
Best score: 0.89625
Accuracy on test set: 0.865
```

## 4. Hyperparameter Tuning with RandomizedSearchCV#

RandomizedSearchCV offers a probabilistic approach, randomly selecting combinations from the parameter distribution. It’s typically faster than GridSearchCV, especially when dealing with a large hyperparameter space or when every incremental improvement is not critical.

```
# Define the parameter distribution
param_dist = {
'max_depth': [None, 10, 20, 30, 40, 50],
'min_samples_split': np.arange(2, 21),
'min_samples_leaf': np.arange(1, 7)
}
# Initialize the RandomizedSearchCV object
random_search = RandomizedSearchCV(estimator=DecisionTreeClassifier(random_state=42), param_distributions=param_dist, n_iter=100, cv=5, verbose=1, random_state=42, scoring='accuracy')
random_search.fit(X_train, y_train)
# Best parameters and best score
print("Best parameters found: ", random_search.best_params_)
print("Best score: ", random_search.best_score_)
# Evaluate on the test set
y_pred = random_search.predict(X_test)
print("Accuracy on test set: ", accuracy_score(y_test, y_pred))
```

```
Fitting 5 folds for each of 100 candidates, totalling 500 fits
```

```
Best parameters found: {'min_samples_split': 18, 'min_samples_leaf': 6, 'max_depth': 30}
Best score: 0.89375
Accuracy on test set: 0.87
```

## 5. Hyperparameter Tuning with Bayesian Optimization#

BayesSearchCV utilizes Bayesian optimization to search for optimal parameters. This method builds a probability model of the objective function and uses it to select the most promising parameters to evaluate in the true objective function.

```
# Define the parameter space
param_space = {
'max_depth': Integer(10, 50),
'min_samples_split': Integer(2, 20),
'min_samples_leaf': Integer(1, 6)
}
# Initialize the BayesSearchCV object
bayes_search = BayesSearchCV(estimator=DecisionTreeClassifier(random_state=42), search_spaces=param_space, n_iter=32, cv=5, verbose=1, scoring='accuracy')
bayes_search.fit(X_train, y_train)
# Best parameters and best score
print("Best parameters found: ", bayes_search.best_params_)
print("Best score: ", bayes_search.best_score_)
# Evaluate on the test set
y_pred = bayes_search.predict(X_test)
print("Accuracy on test set: ", accuracy_score(y_test, y_pred))
```

```
Fitting 5 folds for each of 1 candidates, totalling 5 fits
Fitting 5 folds for each of 1 candidates, totalling 5 fits
Fitting 5 folds for each of 1 candidates, totalling 5 fits
Fitting 5 folds for each of 1 candidates, totalling 5 fits
Fitting 5 folds for each of 1 candidates, totalling 5 fits
Fitting 5 folds for each of 1 candidates, totalling 5 fits
Fitting 5 folds for each of 1 candidates, totalling 5 fits
Fitting 5 folds for each of 1 candidates, totalling 5 fits
Fitting 5 folds for each of 1 candidates, totalling 5 fits
```

```
Fitting 5 folds for each of 1 candidates, totalling 5 fits
```

```
Fitting 5 folds for each of 1 candidates, totalling 5 fits
```

```
Fitting 5 folds for each of 1 candidates, totalling 5 fits
```

```
Fitting 5 folds for each of 1 candidates, totalling 5 fits
```

```
Fitting 5 folds for each of 1 candidates, totalling 5 fits
```

```
Fitting 5 folds for each of 1 candidates, totalling 5 fits
```

```
Fitting 5 folds for each of 1 candidates, totalling 5 fits
```

```
Fitting 5 folds for each of 1 candidates, totalling 5 fits
```

```
Fitting 5 folds for each of 1 candidates, totalling 5 fits
```

```
Fitting 5 folds for each of 1 candidates, totalling 5 fits
```

```
Fitting 5 folds for each of 1 candidates, totalling 5 fits
```

```
Fitting 5 folds for each of 1 candidates, totalling 5 fits
```

```
Fitting 5 folds for each of 1 candidates, totalling 5 fits
```

```
Fitting 5 folds for each of 1 candidates, totalling 5 fits
```

```
Fitting 5 folds for each of 1 candidates, totalling 5 fits
```

```
Fitting 5 folds for each of 1 candidates, totalling 5 fits
```

```
Fitting 5 folds for each of 1 candidates, totalling 5 fits
```

```
Fitting 5 folds for each of 1 candidates, totalling 5 fits
```

```
Fitting 5 folds for each of 1 candidates, totalling 5 fits
```

```
Fitting 5 folds for each of 1 candidates, totalling 5 fits
```

```
Fitting 5 folds for each of 1 candidates, totalling 5 fits
```

```
Fitting 5 folds for each of 1 candidates, totalling 5 fits
```

```
Fitting 5 folds for each of 1 candidates, totalling 5 fits
```

```
Best parameters found: OrderedDict([('max_depth', 13), ('min_samples_leaf', 6), ('min_samples_split', 20)])
Best score: 0.89625
Accuracy on test set: 0.865
```