Automation in Cybersecurity using Machine Learning: A CaseStudy on Anomaly Detection with Isolation Forest

Noorul Hassan S.; Sandhiya L.; Kavya S.; Priyadharshini E.; Vanmathi T.

doi:10.51903/jtie.v4i3.478

Authors

Noorul Hassan S. Arunai Engineering College, Tiruvannamalai, Tamil Nadu, India https://orcid.org/0009-0009-1095-5621
Sandhiya L. Arunai Engineering College, Tiruvannamalai, Tamil Nadu, India
Kavya S. Arunai Engineering College, Tiruvannamalai, Tamil Nadu, India
Priyadharshini E. Arunai Engineering College, Tiruvannamalai, Tamil Nadu, India
Vanmathi T. Arunai Engineering College, Tiruvannamalai, Tamil Nadu, India

DOI:

https://doi.org/10.51903/jtie.v4i3.478

Keywords:

Cybersecurity, Anomaly Detection, Isolation Forest, Intrusion Detection System, Machine Learning

Abstract

The escalating sophistication of cyber threats necessitates advanced anomaly detection techniques that transcend traditional signature-based methods. This paper presents an automated cybersecurity framework leveraging the Isolation Forest algorithm for unsupervised anomaly detection in network traffic. Using the NSL-KDD dataset, we demonstrate that Isolation Forest achieves 95.2% detection accuracy with a 4.7% false-positive rate, outperforming conventional methods such as One-Class SVM (88.1% accuracy) and Local Outlier Factor (82.3% accuracy) in both computational efficiency and precision. Key advantages include: (1) real-time processing capability (8.2s training time, 4× faster than density-based approaches), (2) effective identification of rare attack types (U2R/R2L), and (3) elimination of dependency on labeled training data. The proposed system integrates dynamic threshold tuning and SHAP-based feature weighting to enhance detection stability and reduce false alarms. The results validate Isolation Forest as a scalable and reliable solution for modern intrusion detection systems, with strong implications for SIEM integration and real-time cybersecurity automation. Challenges in parameter tuning and encrypted traffic analysis are discussed, alongside future directions involving hybrid deep learning architectures.

References

Al-Amri, R., Murugesan, R. K., Man, M., Abdulateef, A. F., Al-Sharafi, M. A., & Alkahtani, A. A. (2021). A Review of Machine Learning and Deep Learning Techniques for Anomaly Detection in IoT Data. Applied Sciences, 11(12), 5320. https://doi.org/10.3390/app11125320

Ansari, M. S., Ashy, V. G., & Gupta, R. K. (2025). Robust IoT Security Using Isolation Forest and One-Class SVM Algorithms. Scientific Reports, 15, 36586. https://doi.org/10.1038/s41598-025-20445-4

Carletti, M., Terzi, M., & Susto, G. A. (2023). Interpretable Anomaly Detection with DIFFI: Depth-Based Feature Importance of Isolation Forest. Engineering Applications of Artificial Intelligence, 119, 105730. https://doi.org/10.1016/j.engappai.2022.105730

Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly Detection: A Survey. ACM Computing Surveys, 41(3), 1–58. https://doi.org/10.1145/1541880.1541882

Chua, W., Pajas, A. L., Castro, S. C., Panganiban, P. S., Pasuquin, J. A., Purganan, J. M., et al. (2024). Web Traffic Anomaly Detection Using Isolation Forest. Informatics, 11(4), 83. https://doi.org/10.3390/informatics11040083

Fuhnwi, G. S., Adedoyin, V., & Agbaje, J. O. (2023). An Empirical Internet Protocol Network Intrusion Detection Using Isolation Forest and One-Class SVM. International Journal of Advanced Computer Science and Applications, 14(8), 123–132. https://scholarworks.montana.edu/handle/1/18188

García-Teodoro, E., Díaz-Verdejo, J., Macía-Fernández, G., & Vázquez, E. (2009). Anomaly-Based Network Intrusion Detection: Techniques, Systems and Challenges. Computers & Security, 28(1), 18–28. https://doi.org/10.1016/j.cose.2008.08.003

Hartono, B., Silalahi, F. D., & Muthohir, M. (2024). Transformers in Cybersecurity: Advancing Threat Detection and Response Through Machine Learning Architectures. Journal of Technology Informatics and Engineering, 3(3), 382–396. https://doi.org/10.51903/jtie.v3i3.211

Khaledian, E., Pandey, S., Kundu, P., & Srivastava, A. K. (2021). Real-Time Synchrophasor Data Anomaly Detection and Classification Using Isolation Forest, K-Means, and LOOP. IEEE Transactions on Smart Grid, 12(3), 2378–2388. https://doi.org/10.1109/tsg.2020.3046602

Khraisat, A., Gondal, I., Vamplew, P., & Kamruzzaman, J. (2019). Survey of Intrusion Detection Systems: Techniques, Datasets and Challenges. Cybersecurity, 2(1), 1–22. https://doi.org/10.1186/s42400-019-0038-7

Kumar, S., & Sangwan, P. (2022). Performance Evaluation of Signature-Based and Anomaly-Based Intrusion Detection Systems. International Journal of Cyber Security and Digital Forensics, 11(2), 123–131. https://doi.org/10.1007/978-3-031-35510-3_47

Lesouple, J., Baudoin, C., Spigai, M., & Tourneret, J.-Y. (2021). Generalized Isolation Forest for Anomaly Detection. Pattern Recognition Letters, 149, 109–119. https://doi.org/10.1016/j.patrec.2021.05.022

Liu, F. T., Ting, K. M., & Zhou, Z. (2008). Isolation Forest. IEEE Transactions on Knowledge and Data Engineering, 22(1), 1–12. https://doi.org/10.1109/tkde.2008.190

Lubis, H. T., Roslina, R., & Tanti, L. (2025). Anomaly Detection in Computer Networks Using Isolation Forest in Data Mining. Jurnal Teknik Informatika, 18(1), 45–56. https://doi.org/10.15408/jti.v18i1.44285

Mahajan, A., et al. (2024). A Novel Hybrid Model Merging LOF and IForest Algorithms for Insider Threats Detection. In 2024 4th Asian Conference on Innovation in Technology (ASIANCON), 1–6. https://doi.org/10.1109/asiancon62057.2024.10837763

Mai, N. T., & Khalid, I. (2025). Human Error vs. System Security: Evaluating the Weakest Link in Digital Business Information Systems. Journal of Management and Informatics, 4(3), 981–997. https://doi.org/10.51903/jmi.v4i3.305

Meira, J. (2018). Comparative Results with Unsupervised Techniques in Cyber Attack Novelty Detection. Proceedings, 2(18), 1191. https://doi.org/10.3390/proceedings2181191

Moustafa, N., & Slay, J. (2015). UNSW-NB15: A Comprehensive Data Set for Network Intrusion Detection Systems. In 2015 Military Communications and Information Systems Conference (MilCIS), 1–6. https://doi.org/10.1109/milcis.2015.7348942

Nalini, M., Yamini, B., Ambhika, C., & Siva Subramanian, R. (2024). Enhancing Early Attack Detection: Novel Hybrid Density-Based Isolation Forest for Improved Anomaly Detection. International Journal of Machine Learning and Cybernetics, 15, 4801–4814. https://doi.org/10.1007/s13042-024-02193-5

Salsabila, A. F., Wulandari, A. D., Zahro, I. K., & Hamdani, A. (2026). Design of a Monitoring System for Detecting ARP Spoofing on a Rule-Based Wi-Fi Network. Jurnal Ilmiah Sistem Informasi, 5(1), 257–274. https://doi.org/10.51903/4cykf888

Sharafaldin, I., Lashkari, A. H., & Ghorbani, A. A. (2018). Toward Generating a New Intrusion Detection Dataset and Intrusion Traffic Characterization. In ICISSP 2018, 108–116. https://doi.org/10.5220/0006639801080116

Stefanov, M., Burton, S. L., Akbas, I. M., & Crouse, S. (2025). Exploring the Potential of Artificial Intelligence to Predict Cyber Attacks: Creation, Evaluation and Comparative Analysis of Effective Models of Ensemble Methods, Isolation Forest, and ARIMA. Scientific Bulletin, 30(1), 162–174. https://doi.org/10.2478/raft-2025-0016

Tavallaee, M., Bagheri, E., Lu, W., & Ghorbani, A. (2009). A Detailed Analysis of the KDD CUP 99 Data Set. In Proceedings of the IEEE Symposium on Computational Intelligence for Security and Defense Applications, 1–6. https://doi.org/10.1109/cisda.2009.5356528

Verizon. (2023). Data Breach Investigations Report. Verizon. https://www.verizon.com/business/resources/reports/dbir

Wang, S., Balarezo, J. F., Kandeepan, S., Al-Hourani, A., Chavez, K. G., & Rubinstein, B. (2021). Machine Learning in Network Anomaly Detection: A Survey. IEEE Access, 9, 152379–152396. https://doi.org/10.1109/access.2021.3126834

Wang, X., Li, J., & Zhang, Y. (2021). Improving Network Intrusion Detection Using LSTM and Isolation Forest. In Proceedings of the 2021 International Conference on Cyber Security Intelligence and Analytics (CSIA), 105–112. https://doi.org/10.1007/978-3-030-91421-9_15

Zhang, Z., Lin, Q., & Wu, H. (2023). Concept Drift Adaptation in Intrusion Detection Using Online Learning. Journal of Network and Computer Applications, 196, 103251. https://doi.org/10.1016/j.jnca.2021.103251

Zhong, Y., Liu, W., & Yang, M. (2019). A Comparative Study of Anomaly Detection Algorithms for Intrusion Detection. IEEE Access, 7, 167276–167285. https://doi.org/10.1109/access.2019.2956751