Machine Learning-Cross Validation & ROC curve

Another post starts with you beautiful people!
Hope you enjoyed my previous post about improving your model performance by confusion metrix.
Today we will continue our performance improvement journey and will learn about Cross Validation (k-fold cross validation) & ROC in Machine Learning.

A common practice in data science competitions is to iterate over various models to find a better performing model. However, it becomes difficult to distinguish whether this improvement in score is coming because we are capturing the relationship better or we are just over-fitting the data. To find the right answer of this question, we use cross validation technique. This method helps us to achieve more generalized relationships.

What is Cross Validation?
Cross Validation is a technique which involves reserving a particular sample of a data set on which we do not train the model. Later, we test the model on this sample before finalizing the model.
Here are the steps involved in cross validation:

• We reserve a sample data set.
• Train the model using the remaining part of the data set.
• Use the reserve sample of the data set test (validation) set. This will help us to know the effectiveness of model performance. It our model delivers a positive result on validation data, go ahead with current model.

The k-fold cross validation method of cross validation technique takes care of below three requirements-

• We should train model on large portion of data set. Else, we’d fail every time to read the underlying trend of data sets. Eventually, resulting in higher bias.
• We also need a good ratio testing data points. As, we have seen that lower data points can lead to variance error while testing the effectiveness of model.
• We should iterate on training and testing process multiple times. We should change the train and test data set distribution. This helps to validate the model effectiveness well.

Here are the steps to implement k-fold validation method:-

• Randomly split our entire dataset into k”folds”.
• For each k folds in our dataset, build our model on k – 1 folds of the data set. Then, test the model to check the effectiveness for kth fold.
• Record the error we see on each of the predictions.
• Repeat this until each of the k folds has served as the test set.
• The average of our k recorded errors is called the cross-validation error and will serve as our performance metric for the model.

In this exercise will train our model with the same dataset and continue our step after random forest step as we did in last post of Confusion Metrix. Please revise those steps from previous post.

Comparing above result with Random forest:-

We can see that the accuracy has been increased when performed Cross-Validation in random forest classifier as well as for logistic regression.

Now train the model on whole data and predict the future data points:-

From above results it is quite clear that-

The accuracy scores for

        Random Forest on train/test split : 75                   

        Logistic Regression on train/test split: 75.5  

        Random Forest on Cross Validation : 77.09         

        Logistic Regression on Cross Validation : 76.8

So with cross-validataion there is high probability of increasing model accuracy.

Adjusting the classification threshold:-

From the above graph we find following result for our dataset:-

• Decrease the threshold for predicting diabetes in order to increase the sensitivity of the classifier
• Threshold of 0.5 is used by default (for binary problems) to convert predicted probabilities into class predictions.
• Threshold can be adjusted to increase sensitivity or specificity Sensitivity and specificity have an inverse relationship.

Wouldn't it be nice if we could see how sensitivity and specificity are affected by various thresholds, without actually changing the threshold?

Yes, we can and answer is by plotting ROC curve.

For more details about this curve please visit here- what is ROC?

• ROC curve tries to evaluate how well the model has achieved the seperation between the classes at all threshold values.
• ROC curve can help us to choose a threshold that balances sensitivity and specificity in a way that makes sense for our particular context.

Result:-

Define a function that accepts a threshold and prints sensitivity and specificity:-

Conclusion of this exercise:

In this way business can understand where should the threshold be set so as to maximize Sensitivity or Specificity.

In my next post we will learn about Principal component analysis or PCA.