A Malware Detection Model in Transport Layer Security Traffic Using the Half-Space Tree Algorithm

Abstract

This study explores the use of the Half-Space Tree (HSTree) algorithm for unsupervised malware detection in TLS traffic, addressing the limitations of traditional supervised learning methods that rely on labelled data. Utilising a comprehensive dataset from [1], the research involved data cleaning and preparation, handshake feature extraction, extraction of streaming test data, HSTree model streaming, testing data extraction and streaming, malware identification, and model evaluation using precision, accuracy, recall, and F1 score metrics. The results showed that the unsupervised HSTree algorithm effectively detected anomalies in TLS traffic, identifying deviations indicative of malicious activities without prior labelled data. Comparative evaluations demonstrated its potential to adapt to new threats, performing comparably to supervised models in accuracy and excelling in detecting new anomalies. This study highlights the algorithm's suitability for real-time malware detection, particularly in environments where labelled data is scarce or outdated and recommends further refinement and preprocessing for improved adaptability and accuracy in cybersecurity defences.