Template Credit: Adapted from a template made available by Dr. Jason Brownlee of Machine Learning Mastery.
INTRODUCTION: Mining activity has always been connected with the occurrence of dangers which are commonly called mining hazards. A special case of such a threat is a seismic hazard which frequently occurs in many underground mines. Seismic hazard is the hardest detectable and predictable of natural hazards, and it is comparable to an earthquake. The complexity of seismic processes and big disproportion between the number of low-energy seismic events and the number of high-energy phenomena causes the statistical techniques to be insufficient to predict seismic hazard. Therefore, it is essential to search for new opportunities for better hazard prediction, also using machine learning methods.
In iteration Take1, we had three algorithms with high accuracy results but with dismal Kappa scores. For this iteration, we will examine the viability of using the ROC scores to rank and choose the models.
CONCLUSION: From the previous Take1 iteration, the baseline performance of the eight algorithms achieved an average accuracy of 93.11%. Three algorithms (Random Forest, Support Vector Machine, and Adaboost) achieved the top three accuracy scores after the first round of modeling. After a series of tuning trials, all three algorithms turned in the identical accuracy result of 93.42%, with an identical Kappa score of 0.0. With an imbalanced dataset we have on-hand, we will need to look for another metric or another approach to evaluate the models.
From the current iteration, the baseline performance of the eight algorithms achieved an average ROC score of 71.99%. Three algorithms (Random Forest, Adaboost, and Stochastic Gradient Boosting) achieved the top three ROC scores after the first round of modeling. After a series of tuning trials, Stochastic Gradient Boosting turned in the best ROC result of 78.59, but with a dismal sensitivity score of 0.88%.
The ROC metric has given us a more viable way to evaluate the models, other than using the accuracy scores. However, with an imbalanced dataset that we have on-hand, we still need to look for another approach to further validate our modeling effort.
The HTML formatted report can be found here on GitHub.