A novel framework for APT attack detection based on network traffic

Document Type : Research Paper


1 Department of Information Technology, University of Economics and Technical Industries, Ha Noi, Vietnam

2 Information Technology Department, LienVietPostBank, Ha Noi, Vietnam

3 Department of Information Technology, Posts and Telecommunications Institute of Technology, Ha Noi, Vietnam


APT (Advanced Persistent Threat) attack is a dangerous, targeted attack form with clear targets. APT attack campaigns have huge consequences. Therefore, the problem of researching and developing the APT attack detection solution is very urgent and necessary nowadays. On the other hand, no matter how advanced the APT attack, it has clear processes and lifecycles. Taking advantage of this point, security experts recommend that could develop APT attack detection solutions for each of their life cycles and processes. In APT attacks, hackers often use phishing techniques to perform attacks and steal data. If this attack and phishing phase is detected, the entire APT attack campaign will be crash. Therefore, it is necessary to research and deploy technology and solutions that could detect early the APT attack when it is in the stages of attacking and stealing data. This paper proposes an APT attack detection framework based on the Network traffic analysis technique using open-source tools and deep learning models. This research focuses on analyzing Network traffic into different components, then finds ways to extract abnormal behaviors on those components, and finally uses deep learning algorithms to classify Network traffic based on the extracted abnormal behaviors. The abnormal behavior analysis process is presented in detail in section III.A of the paper. The APT attack detection method based on Network traffic is presented in section III.B of this paper. Finally, the experimental process of the proposal is performed in section IV of the paper


[1] Z. Aaron, C.H. Song, W. Zhaoshun and C. Mumbi, Modeling and detection of the multi-stages of Advanced
Persistent Threats attacks based on semi-supervised learning and complex networks characteristics, Future Gen.
Comput. Syst. 106 (2020), 501–517.
[2] A. Alshamrani, A. Chowdhary, O. Mjihil, S. Myneni and D. Huang, Combining dynamic and static attack information for attack tracing and event correlation, 2018 IEEE Glob. Commun. Conf. (GLOBECOM), 2018, pp.
[3] A. Alshamrani, A. Chowdhary, S. Myneni and D. Huang, A survey on advanced persistent threats: Techniques,
solutions, challenges, and research opportunities, IEEE Commun. Surv. Tutor. 1 (2019), 1–29.
[4] D.X. Cho and H.H. Nam, Method of monitoring and detecting APT attacks based on unknown domains, Procedia
Comput. Sci. 150 (2019), 316–323.
[5] X.C. Do, D. Duc and D.H. Xuan, A multi-layer approach for advanced persistent threat detection using machine
learning based on network traffic, J. Intell.Fuzzy Syst. 40 (2021), no. 6, 11311–11329.
[6] I. Ghafir, M. Hammoudeh, V. Prenosil, L. Han, R. Hegarty, K. Rabie and F.J. Aparicio-Navarro, Detection of
advanced persistent threat using machine-learning correlation analysis, Future Gen. Comput. Syst. 89 (2018),
[7] I. Ghafir, K.G. Kyriakopoulos, S. Lambotharan, F.J. Aparicio-Navarro, B. AsSadhan, H. Binsalleeh, D.M. Diab,
Hidden Markov models and alert correlations for the prediction of advanced persistent threats, IEEE Access 7
(2019), 99508–99520.
[8] S. Hochreiter and J. Schmidhuber, Long short-term memory, Neural Comput. 9 (1997), no. 8, 1735—1780.
[9] H. Huang, H. Deng, Y. Sheng and X. Ye, Accelerating convolutional neural network-based malware traffic detection
through ant-colony clustering, J. Intell. Fuzzy Syst. 37 (2019), 409–423.[10] Y. Ji, S. Lee, E. Downing, W. Wang, M. Fazzini, T. Kim, A. Orso and W. Lee, Rain: Refinable attack investigation
with on-demand inter-process information flow tracking, ACM SIGSAC Conf. Comput. Commun. Security, 2017,
[11] A. Lajevardi and M. Amini, A semantic-based correlation approach for detecting hybrid and low-level APTs,
Future Gen. Comput. Syst. 96 (2019), 64–88.
[12] S. Ma, J. Zhai, F. Wang, K.H. Lee, X. Zhang and D. Xu, MPI: Multiple perspective attack investigation with
semantic aware execution partitioning, 26th USENIX Conf. Security Symp., 2017, pp. 1111-–1128.
[13] Malware Capture Facility Project, Available online: https://www.stratosphereips.org/datasets-malware. (Accessed on 8 June 2021).
[14] M. Marchetti, F. Pierazzi, M. Colajanni and A. Guido, Analysis of high volumes of network traffic for Advanced
Persistent Threat detection, Comput. Networks 109 (2016), 127–141.
[15] H. Peng, L. Liu, J. Liu and J.R. Lewis, Network traffic anomaly detection algorithm using mahout classifier, J.
Intell. Fuzzy Syst. 37 (2019), 137–144.
[16] M. Shen, P. Ju and F. Shumin, Event-triggered nonfragile H∞H∞ filtering of Markov jump systems with imperfect
transmisions, Signal Process. 149 (2018).
[17] A. Sherstinsky, Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network,
Phys. D: Nonlinear Phenomena 404 (2020).
[18] Suricata, Available online: https://suricata-ids.org/. (Accessed Feb 14, 2020).
[19] N. Van Can, D.N. Tu, T.A. Tuan, H.V. Long, L.H. Son and N.T.K. Son, A new method to classify malicious
domain name using neutrosophic sets in DGA botnet detection, J. Intell. Fuzzy Syst. 36 (2020), 4223–4236.
[20] R. Vinayakumara, K.P. Somana and P. Poornachandranb, Detecting malicious domain names using deep learning
approaches at scale, J. Intell. Fuzzy Syst. 34 (2018), 1355–1367.
[21] H. Wang, Z. Cao and B Hong, A network intrusion detection system based on convolutional neural network, J.
Intell. Fuzzy Syst. 38 (2020), 7623–7637.
[22] F. Wang, Y. Kwon, S. Ma and X. Zhang, Lprov: Practical library-aware provenance tracing, 34th Ann. Comput.
Security Appl. Conf., 2018, pp.605–617.
[23] L.C. Wen, J.L. Chih and N.C. Ke, Detection and classification of advanced persistent threats and attacks using
the support vector machine, Appl. Sci. 9 (2019), 45–79.
[24] Z. Xiang, D. Guo and Q. Li, Detecting mobile advanced persistent threats based on large-scale DNS logs, Comput.
Secur. 96 (2020).
[25] W. Xianming, Q. Wen, P. Ju and Mo. Shen, Event-triggered data-driven control of discrete-time nonlinear systems
with unknown disturbance, ISA Transactions (2021) doi:10.1016/j.isatra.2021.11.026.
[26] C.D. Xuan, Detecting APT attacks based on network traffic using machine learning, J. Web Engin. 20 (2021),
no. 1, 171–190.
[27] C.D. Xuan and H.M. Dao, A novel approach for APT attack detection based on combined deep learning model,
Neural Comput. Appl. 33 (2021), no. 20, 13251–13264.
[28] C.D. Xuan, H.D. Nguyen and H.M. Dao, APT attack detection based on flow network analysis techniques using
deep learning, J. Intell. Fuzzy Syst. 290 (2020), no. 3, 4785–4801.
[29] S. Yan, Z. Gu, J. H. Park, X. Xie and C. Dou, Probability-density-dependent load frequency control of
power systems with random delays and cyber-attacks via circuital implementation, IEEE Trans. Smart Grid
[30] S. Yan, Z. Gu and J. H. Park, Memory-event-triggered H∞ load frequency control of multi-area power systems
with cyber-attacks and communication delays, IEEE Trans. Network Sci. Engin. 8 (2021), no. 2, 1571–1583.
[31] S. Yan, Z. Gu, S.K. Nguang, F. Yang and L. Zhang, Co-design of event-triggered scheme and H∞ output control
for Markov jump systems against deception attacks, IEEE Access 8 (2020), 106554–106563.[32] G. Yan, Q. Li, D. Guo and X. Meng, Discovering suspicious APT behaviors by analyzing DNS activities, Sensors
20 (2020), 1–17.
[33] S. Yan, S.K. Nguang and L. Zhang, Nonfragile integral-based event-triggered control of uncertain cyber-physical
systems under cyber-attacks, Complexity 2019 (2019).
Volume 13, Issue 2
July 2022
Pages 2933-2945
  • Receive Date: 25 April 2022
  • Revise Date: 07 July 2022
  • Accept Date: 17 July 2022