Dissecting Malicious Behaviours of Mobile Applications

Abstract

This thesis dissects the behaviours of malicious software and unveils the internal mechanism of evading the detection by malware detectors. With the popularity of smartphones, malicious software has been one of the most severe risks to the public. Therefore, mobile security has been a critical and hot topic in security research. Various malware detection and antivirus methodologies have been proposed to defend the rapid evaluation and variation. These malware variants can camouflage themselves with complicated techniques, such as obfuscation and feature perturbation, to evade the detection by antivirus products. Machine learning-based malware detection techniques are introduced into these products to address this problem. Machine learning-based malware detectors leverage features extracted from malicious and benign software to train detection models to identify malware and its variants effectively. In this thesis, I will first conduct a literature review of state-of-the-art malware detection techniques to unveil how these techniques contribute to anti-malware research. The literature review covers state-of-the-art methodologies, including software static and dynamic analysis, malware detection and machine learning. Then, an explainability-guided measurement approach is proposed to measure malware detectors’ functionalities and guide adversarial sample generation. In this approach, we introduce a novel measurement concept, Accrued Malicious Magnitude (AMM) to identify which malware features should be manipulated to maximize the likelihood of evading detection. The AMM is defined as the product of the magnitude of SHAP values in each feature and the number of samples that have malicious-oriented values in the corresponding feature. Compared with SHAP values representing features’ importance to the prediction results, AMM values reflect how much the specific features can contribute to flipping the prediction result. Finally, I will conclude the thesis and discuss the future work.