Writing a Custom Cross Validation Function with StratifiedKFold

In the cell below, we have set up a function custom_cross_val_score that has an interface that resembles the cross_val_score function from scikit-learn.

Most of it is set up for you already, all you need to do is add the SMOTE and StandardScaler steps described above.

# Import relevant sklearn and imblearn classes
from sklearn.preprocessing import StandardScaler
from imblearn.over_sampling import SMOTE

def custom_cross_val_score(estimator, X, y):
    # Create a list to hold the scores from each fold
    kfold_train_scores = np.ndarray(5)
    kfold_val_scores = np.ndarray(5)

    # Instantiate a splitter object and loop over its result
    kfold = StratifiedKFold(n_splits=5)
    for fold, (train_index, val_index) in enumerate(kfold.split(X, y)):
        # Extract train and validation subsets using the provided indices
        X_t, X_val = X.iloc[train_index], X.iloc[val_index]
        y_t, y_val = y.iloc[train_index], y.iloc[val_index]
        
        # Instantiate StandardScaler
        scaler = StandardScaler()
        # Fit and transform X_t
        X_t_scaled = scaler.fit_transform(X_t)
        # Transform X_val
        X_val_scaled = scaler.transform(X_val)
        
        # Instantiate SMOTE with random_state=42 and sampling_strategy=0.28
        sm = SMOTE(random_state=42, sampling_strategy=0.28)
        # Fit and transform X_t_scaled and y_t using sm
        X_t_oversampled, y_t_oversampled = sm.fit_resample(X_t_scaled, y_t)
        
        # Clone the provided model and fit it on the train subset
        temp_model = clone(estimator)
        temp_model.fit(X_t_oversampled, y_t_oversampled)
        
        # Evaluate the provided model on the train and validation subsets
        neg_log_loss_score_train = neg_log_loss(temp_model, X_t_oversampled, y_t_oversampled)
        neg_log_loss_score_val = neg_log_loss(temp_model, X_val_scaled, y_val)
        kfold_train_scores[fold] = neg_log_loss_score_train
        kfold_val_scores[fold] = neg_log_loss_score_val
        
    return kfold_train_scores, kfold_val_scores

model_with_preprocessing = LogisticRegression(random_state=42, class_weight={1: 0.28})
preprocessed_train_scores, preprocessed_neg_log_loss_cv = custom_cross_val_score(model_with_preprocessing, X_train, y_train)
- (preprocessed_neg_log_loss_cv.mean())

0.132358995512103

Let’s compare that to the baseline log loss:

# Run this cell without changes
print(-baseline_neg_log_loss_cv.mean())
print(-preprocessed_neg_log_loss_cv.mean())

0.1723165500699139
0.132358995512103

Looks like the preprocessing with StandardScaler and SMOTE has provided some improvement over the baseline!

Reduce Regularization

Instantiate a LogisticRegression model with lower regularization (i.e. higher C), along with random_state=42 and class_weight={1: 0.28}.

model_less_regularization = LogisticRegression(random_state = 42, class_weight = {1: 0.28}, C = 1e5)

Now, evaluate that model using custom_cross_val_score. Recall that custom_cross_val_score takes 3 arguments: estimator, X, and y. In this case, estimator should be model_less_regularization, X should be X_train, and y should be y_train.

less_regularization_train_scores, less_regularization_val_scores = custom_cross_val_score(
    model_less_regularization,
    X_train,
    y_train
)

print("Previous Model")
print("Train average:     ", -preprocessed_train_scores.mean())
print("Validation average:", -preprocessed_neg_log_loss_cv.mean())
print("Current Model")
print("Train average:     ", -less_regularization_train_scores.mean())
print("Validation average:", -less_regularization_val_scores.mean())

Previous Model
Train average:      0.28974560105233593
Validation average: 0.132358995512103

Current Model
Train average:      0.2895752558726792
Validation average: 0.1323443569205182

We can also try different regularization penalties.

From the docs:

• ‘newton-cg’, ‘lbfgs’, ‘sag’ and ‘saga’ handle L2 or no penalty
• ‘liblinear’ and ‘saga’ also handle L1 penalty
• ‘saga’ also supports ‘elasticnet’ penalty

In other words, the only models that support L1 or elastic net penalties are liblinear and saga. liblinear is going to be quite slow with the size of the dataset, so this will use saga.

model_alternative_solver = LogisticRegression(
    random_state = 42,
    class_weight = {1: 0.28},
    C = 1e5,
    solver = 'saga',
    penalty = 'elasticnet',
    l1_ratio = .5
)

alternative_solver_train_scores, alternative_solver_val_scores = custom_cross_val_score(
    model_alternative_solver,
    X_train,
    y_train
)

print("Previous Model (Less Regularization)")
print("Train average:     ", -less_regularization_train_scores.mean())
print("Validation average:", -less_regularization_val_scores.mean())
print("Current Model")
print("Train average:     ", -alternative_solver_train_scores.mean())
print("Validation average:", -alternative_solver_val_scores.mean())

Previous Model (Less Regularization)
Train average:      0.2895752558726792
Validation average: 0.1323443569205182
Current Model
Train average:      0.29297684099860494
Validation average: 0.13604493761082842

Adjusting Gradient Descent Parameters

In this case, the model is getting slightly worse metrics on both the train and the validation data (compared to a different solver strategy), so increasing the number of iterations (max_iter) seems like a better strategy. Essentially this is allowing the gradient descent algorithm to take more steps as it tries to find an optimal solution.

model_more_iterations = LogisticRegression(
    random_state = 42,
    class_weight = {1: 0.28},
    C = 1e5,
    solver = 'saga',
    penalty = 'elasticnet',
    l1_ratio = .5,
    max_iter = 1000
)

more_iterations_train_scores, more_iterations_val_scores = custom_cross_val_score(
    model_more_iterations,
    X_train,
    y_train
)

print("Previous Best Model (Less Regularization)")
print("Train average:     ", -less_regularization_train_scores.mean())
print("Validation average:", -less_regularization_val_scores.mean())
print("Previous Model with This Solver")
print("Train average:     ", -alternative_solver_train_scores.mean())
print("Validation average:", -alternative_solver_val_scores.mean())
print("Current Model")
print("Train average:     ", -more_iterations_train_scores.mean())
print("Validation average:", -more_iterations_val_scores.mean())

At this point, the model_less_regularization the best model, and I will move on to the final evaluation phase.

Preprocessing the Full Dataset

# Instantiate StandardScaler
scaler = StandardScaler()

# Fit and transform X_train
X_train_scaled = scaler.fit_transform(X_train)

# Transform X_test
X_test_scaled = scaler.transform(X_test)

# Instantiate SMOTE with random_state=42 and sampling_strategy=0.28
sm = SMOTE(random_state = 42, sampling_strategy = 0.28)

# Fit and transform X_train_scaled and y_train using sm
X_train_oversampled, y_train_oversampled = sm.fit_resample(X_train_scaled, y_train)

Fitting the Model on the Full Training Data

final_model.fit(X_train_oversampled, y_train_oversampled)

LogisticRegression(C=100000.0, class_weight={1: 0.28}, random_state=42)

Evaluating the Model on the Test Data

log_loss(y_test, final_model.predict_proba(X_test_scaled))

0.13031294393913376

from sklearn.metrics import accuracy_score

accuracy_score(y_test, final_model.predict(X_test_scaled))

0.9456679825472678

# Import the relevant function
from sklearn.metrics import precision_score

# Display the precision score
precision_score(y_test, final_model.predict(X_test_scaled))

0.6659919028340081

# Import the relevant function
from sklearn.metrics import recall_score

# Display the recall score
recall_score(y_test, final_model.predict(X_test_scaled))

0.47889374090247455