International Journal of Nonlinear Analysis and Applications
A landscape view of deepfake techniques and detection methods
Document Type : Review articles
Authors
Faculty of Computer Science and Mathematics, University of Kufa, Iraq
Abstract
Deep fakes is the process of changing the information of the image or video with different techniques and methods that start with humor and fun and sometimes reach economic, political and social goals such as counterfeiting, financial fraud or impersonation. The data for this field is still increasing at a very high rate. And therefore. The process of combating and exploring them is a very difficult task. In this paper, we conducted a review of previous studies and what researchers dealt with on the subject of deep fakes. Explain the concepts of deepfakes. Counterfeiting methods and techniques and patterns through the techniques and algorithms used in counterfeiting. Some deepfake detection algorithms.
Keywords
[1] D. Afchar, V. Nozick, J. Yamagishi and I. Echizen, MesoNet: A compact facial video forgery detection network, 10th IEEE Int. Work Inf. Forensics Secur. WIFS 2018, (2019).
[2] I. Amerini, L. Galteri, R. Caldelli and A. Del Bimbo, Deepfake video detection through optical flow based CNN, Proc. IEEE/CVF Int. Conf. on Comput. Vis. (ICCV) Workshops, 2019.
[3] A. Bharati, R. Singh, M. Vatsa and K. Bowyer, Detecting facial retouching using supervised deep learning, IEEE Transactions on Inf. Forens. Secur. 11(9) (2016) 1903–1913.
[4] BBC Bitesize, Deepfakes: What Are They and Why Would I Make One?, www.bbc.co.uk, 2019.
[5] C. Bregler, M. Covell and M. Slaney, Video rewrite: driving visual speech with audio, Proc. 24th Annu. Conf. Comput. Graph Interact Tech SIGGRAPH 1997 (1997) 353-360.
[6] W. Cho, S. Choi, D.K. Park, I. Shin and J. Choo, Image-to-image translation via group-wise deep whitening-andcoloring transformation, Proceedings-2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019, IEEE Computer Society, 2019-June (2019) 10639–10647.
[7] Y. Choi, M. Choi, M. Kim, J.W. Ha, S. Kim and J. Choo, StarGAN: unified generative adversarial networks for multi-domain image-to-image translation, Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (2018) 8789–8797.
[8] H. Dang, F. Liu, J. Stehouwer, X. Liu and A.K. Jain, On the detection of digital face manipulation, Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (2020) 5780–5789.
[9] B. Dolhansky, J. Bitton, B. Pflaum, J. Lu, R. Howes, M. Wang and C.C. Ferrer, The deepfake detection challenge (dfdc) dataset, arXiv preprint arXiv:2006.07397, 2020.
[10] H. Farid, Image forgery detection, IEEE Signal Processing Magazine 26(2) (2009) 16–25.
[11] J. Galbally, S. Marcel and J. Fierrez, Biometric antispoofing methods: A survey in face recognition, IEEE Access 2 (2014) 1530–1552.
[12] H. Guan, M. Kozak, E. Robertson, Y. Lee, A. Yates, A. Delgado, D. Zhou, T. Kheyrkhah, J. Smith and J. Fiscus, MFC datasets: large-scale benchmark datasets for media forensic challenge evaluation, in Proc. IEEE Winter Applications of Computer Vision Workshops, 2019.
[13] L. Guarnera, O. Giudice and S. Battiato, DeepFake detection by analyzing convolutional traces, IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. Work, 2020 (2020) 2841–2850.
[14] D. Guera and E.J. Delp, Deepfake video detection using recurrent neural networks, Proc. AVSS 2018 - 2018 15th IEEE Int. Conf. Adv. Video Signal-Based Surveill, 2019.
[15] Z. He, W. Zuo, M. Kan, S. Shan and X. Chen, AttGAN: Facial attribute editing by only changing what you want, IEEE Trans. Image Process, 28(11) (2019) 5464–5478.
[16] N. Hulzebosch, S. Ibrahimi and M. Worring, Detecting CNN-generated facial images in real-world scenarios, IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit Work, 2020 (2020) 2729–2738.
[17] A. Jain, R. Singh and M. Vatsa, On detecting GANs and retouching based synthetic alterations, 2018 IEEE 9th Int. Conf. Biometrics Theory, Appl. Syst. BTAS 2018 (2018).
[18] T. Jung, S. Kim and K. Kim, Deepvision: deepfakes detection using human eye blinking pattern, IEEE Access. 8 (2020) 83144–83154.
[19] T. Karras, T. Aila, S. Laine and J. Lehtinen, Progressive growing of GANs for improved quality, stability, and variation, 6th Int. Conf. Learn Represent ICLR 2018 – Conf. Track Proc. (2018) 1–26.
[2] I. Amerini, L. Galteri, R. Caldelli and A. Del Bimbo, Deepfake video detection through optical flow based CNN, Proc. IEEE/CVF Int. Conf. on Comput. Vis. (ICCV) Workshops, 2019.
[3] A. Bharati, R. Singh, M. Vatsa and K. Bowyer, Detecting facial retouching using supervised deep learning, IEEE Transactions on Inf. Forens. Secur. 11(9) (2016) 1903–1913.
[4] BBC Bitesize, Deepfakes: What Are They and Why Would I Make One?, www.bbc.co.uk, 2019.
[5] C. Bregler, M. Covell and M. Slaney, Video rewrite: driving visual speech with audio, Proc. 24th Annu. Conf. Comput. Graph Interact Tech SIGGRAPH 1997 (1997) 353-360.
[6] W. Cho, S. Choi, D.K. Park, I. Shin and J. Choo, Image-to-image translation via group-wise deep whitening-andcoloring transformation, Proceedings-2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019, IEEE Computer Society, 2019-June (2019) 10639–10647.
[7] Y. Choi, M. Choi, M. Kim, J.W. Ha, S. Kim and J. Choo, StarGAN: unified generative adversarial networks for multi-domain image-to-image translation, Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (2018) 8789–8797.
[8] H. Dang, F. Liu, J. Stehouwer, X. Liu and A.K. Jain, On the detection of digital face manipulation, Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (2020) 5780–5789.
[9] B. Dolhansky, J. Bitton, B. Pflaum, J. Lu, R. Howes, M. Wang and C.C. Ferrer, The deepfake detection challenge (dfdc) dataset, arXiv preprint arXiv:2006.07397, 2020.
[10] H. Farid, Image forgery detection, IEEE Signal Processing Magazine 26(2) (2009) 16–25.
[11] J. Galbally, S. Marcel and J. Fierrez, Biometric antispoofing methods: A survey in face recognition, IEEE Access 2 (2014) 1530–1552.
[12] H. Guan, M. Kozak, E. Robertson, Y. Lee, A. Yates, A. Delgado, D. Zhou, T. Kheyrkhah, J. Smith and J. Fiscus, MFC datasets: large-scale benchmark datasets for media forensic challenge evaluation, in Proc. IEEE Winter Applications of Computer Vision Workshops, 2019.
[13] L. Guarnera, O. Giudice and S. Battiato, DeepFake detection by analyzing convolutional traces, IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. Work, 2020 (2020) 2841–2850.
[14] D. Guera and E.J. Delp, Deepfake video detection using recurrent neural networks, Proc. AVSS 2018 - 2018 15th IEEE Int. Conf. Adv. Video Signal-Based Surveill, 2019.
[15] Z. He, W. Zuo, M. Kan, S. Shan and X. Chen, AttGAN: Facial attribute editing by only changing what you want, IEEE Trans. Image Process, 28(11) (2019) 5464–5478.
[16] N. Hulzebosch, S. Ibrahimi and M. Worring, Detecting CNN-generated facial images in real-world scenarios, IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit Work, 2020 (2020) 2729–2738.
[17] A. Jain, R. Singh and M. Vatsa, On detecting GANs and retouching based synthetic alterations, 2018 IEEE 9th Int. Conf. Biometrics Theory, Appl. Syst. BTAS 2018 (2018).
[18] T. Jung, S. Kim and K. Kim, Deepvision: deepfakes detection using human eye blinking pattern, IEEE Access. 8 (2020) 83144–83154.
[19] T. Karras, T. Aila, S. Laine and J. Lehtinen, Progressive growing of GANs for improved quality, stability, and variation, 6th Int. Conf. Learn Represent ICLR 2018 – Conf. Track Proc. (2018) 1–26.
[20] T. Karras, S. Laine and T. Aila, A style-based generator architecture for generative adversarial networks, Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 2019 (2019) 4396–4405.
[21] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen and T. Aila, Analyzing and improving the image quality of stylegan, Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 2020 (2020) 8107–8116.
[22] D.P. Kingma and M. Welling, Auto-encoding variational bayes, 2nd Int. Conf. Learn Represent ICLR 2014 – Conf. Track Proc. 2014 (2014) 1–14.
[23] P. Korshunov and S. Marcel, Deepfakes: A new threat to face recognition? Assessment and detection, arXiv, 2018.
[24] P. Korus, Digital image integrity – a survey of protection and verification techniques, Digit. Signal Process, Rev. J. 71 (2017) 1–26.
[25] Y. Li and S. Lyu, Exposing deepfake videos by detecting face warping artifacts, arXiv, 2018.
[26] C. Li, K. Xu, J. Zhu and B. Zhang, Triple generative adversarial nets, Adv. Neural Inf. Process Syst. 2017 (2017) 4089–4099.
[27] W.S. Lin, S.K. Tjoa, H.V. Zhao and K.J.R. Liu, Digital image source coder forensics via intrinsic fingerprints, IEEE Trans. Inf. Forensics Secur. 4(3) (2009) 460–475.
[28] M. Liu, Y. Ding, M. Xia, X. Liu, E. Ding, W. Zuo and S. Wen, STGAN: A unified selective transfer network for arbitrary image attribute editing, Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 2019 (2019) 3668–3677.
[29] S. Marcel, M. Nixon, J. Fierrez and N. Evans, Handbook of Biometric Anti-Spoofing (2nd Edition), Springer, 2019.
[30] F. Marra, C. Saltori, G. Boato and L. Verdoliva, Incremental learning for the detection and classification of GAN-generated images, 2019 IEEE Int. Work Inf. Forensics Secur. WIFS 2019 (2019).
[31] F. Matern, C. Riess and M. Stamminger, Exploiting visual artifacts to expose deepfakes and face manipulations, in Proc. IEEE Winter Applications of Computer Vision Workshops, 2019.
[32] S. McCloskey and M. Albright, Detecting GAN-generated imagery using color cues, arXiv, 2018.
[33] L. Nataraj, T.M. Mohammed, B.S. Manjunath, S. Chandrasekaran, A. Flenner, J.H. Bappy and A.K. RoyChowdhury, Detecting GAN generated fake images using co-occurrence matrices, IS& T Int. Symp. Electron Imaging Sci. Technol. 2019(5) (2019) 1–7.
[34] J.C. Neves, R. Tolosana, R. Vera-Rodriguez, V. Lopes, H. Proen¸ca and J. Fierrez, GANprintR: Improved fakes and evaluation of the state of the art in face manipulation detection, IEEE J. Sel. Top Signal Process 14(5) (2020) 1038–1048.
[35] H.M. Nguyen and R. Derakhshani, Eyebrow recognition for identifying deepfake videos, BIOSIG 2020 – Proc. 19th Int. Conf. Biometrics Spec. Interes. Gr. (2020) 1–6.
[36] H.H. Nguyen, F. Fang, J. Yamagishi and I. Echizen, Multi-task learning for detecting and segmenting manipulated facial images and videos, 2019 IEEE 10th Int. Conf. Biometrics Theory, Appl. Syst. BTAS 2019 (2019).
[37] H.H. Nguyen, J. Yamagishi and I. Echizen, Use of a capsule network to detect fake images and videos, arXiv, 2019.
[38] O.M. Parkhi, A. Vedaldi and A. Zisserman, Deep face recognition, Visual Geometry Group Department of Engineering Science University of Oxford, (2015) 1–12.
[39] C. Rathgeb, A. Botaljov, F. Stockhardt, S. Isadskiy, L. Debiasi, A. Uhl and C. Busch, PRNU-based detection of facial retouching, IET Biomet. 9(4) (2020) 154–164.
[40] A. Rossler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies and M. Niessner, FaceForensics++: Learning to detect manipulated facial images, Proc. IEEE Int. Conf. Comput. Vis. 2019 (2019) 1–11.
[41] E. Sabir, J. Cheng, A. Jaiswal, W. AbdAlmageed, I. Masi and P. Natarajan, Recurrent convolutional strategies for face manipulation detection in videos, arXiv, 2019.
[42] F. Schroff, D. Kalenichenko and J. Philbin, FaceNet: A unified embedding for face recognition and clustering, Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (2015) 815–823.
[43] S. Tariq, S. Lee, H. Kim, Y. Shin and S. Woo, Detecting both machine and human created fake face images in the wild, Proc. Int. Workshop Multimedia Priv. Secur.(2018) 81–87.
[44] J. Thies, M. Zollh¨ofer, M. Stamminger, C. Theobalt and M. Nießner, Face2face, Commun. ACM. 62(1) (2018) 96–104.
[45] J. Thies, M. Zollh¨ofer, M. Stamminger, C. Theobalt and M. Nießner, Deferred neural rendering: image synthesis using neural textures, ACM. Trans. Graph. 38(4) (2019).
[46] R. Tolosana, S. Romero-Tapiador, J. Fierrez and R. Vera-Rodriguez, Deepfakes evolution: analysis of facial regions and fake detection performance, Lect. Notes Comput. Sci. (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics), (2021) 12665.
[21] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen and T. Aila, Analyzing and improving the image quality of stylegan, Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 2020 (2020) 8107–8116.
[22] D.P. Kingma and M. Welling, Auto-encoding variational bayes, 2nd Int. Conf. Learn Represent ICLR 2014 – Conf. Track Proc. 2014 (2014) 1–14.
[23] P. Korshunov and S. Marcel, Deepfakes: A new threat to face recognition? Assessment and detection, arXiv, 2018.
[24] P. Korus, Digital image integrity – a survey of protection and verification techniques, Digit. Signal Process, Rev. J. 71 (2017) 1–26.
[25] Y. Li and S. Lyu, Exposing deepfake videos by detecting face warping artifacts, arXiv, 2018.
[26] C. Li, K. Xu, J. Zhu and B. Zhang, Triple generative adversarial nets, Adv. Neural Inf. Process Syst. 2017 (2017) 4089–4099.
[27] W.S. Lin, S.K. Tjoa, H.V. Zhao and K.J.R. Liu, Digital image source coder forensics via intrinsic fingerprints, IEEE Trans. Inf. Forensics Secur. 4(3) (2009) 460–475.
[28] M. Liu, Y. Ding, M. Xia, X. Liu, E. Ding, W. Zuo and S. Wen, STGAN: A unified selective transfer network for arbitrary image attribute editing, Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 2019 (2019) 3668–3677.
[29] S. Marcel, M. Nixon, J. Fierrez and N. Evans, Handbook of Biometric Anti-Spoofing (2nd Edition), Springer, 2019.
[30] F. Marra, C. Saltori, G. Boato and L. Verdoliva, Incremental learning for the detection and classification of GAN-generated images, 2019 IEEE Int. Work Inf. Forensics Secur. WIFS 2019 (2019).
[31] F. Matern, C. Riess and M. Stamminger, Exploiting visual artifacts to expose deepfakes and face manipulations, in Proc. IEEE Winter Applications of Computer Vision Workshops, 2019.
[32] S. McCloskey and M. Albright, Detecting GAN-generated imagery using color cues, arXiv, 2018.
[33] L. Nataraj, T.M. Mohammed, B.S. Manjunath, S. Chandrasekaran, A. Flenner, J.H. Bappy and A.K. RoyChowdhury, Detecting GAN generated fake images using co-occurrence matrices, IS& T Int. Symp. Electron Imaging Sci. Technol. 2019(5) (2019) 1–7.
[34] J.C. Neves, R. Tolosana, R. Vera-Rodriguez, V. Lopes, H. Proen¸ca and J. Fierrez, GANprintR: Improved fakes and evaluation of the state of the art in face manipulation detection, IEEE J. Sel. Top Signal Process 14(5) (2020) 1038–1048.
[35] H.M. Nguyen and R. Derakhshani, Eyebrow recognition for identifying deepfake videos, BIOSIG 2020 – Proc. 19th Int. Conf. Biometrics Spec. Interes. Gr. (2020) 1–6.
[36] H.H. Nguyen, F. Fang, J. Yamagishi and I. Echizen, Multi-task learning for detecting and segmenting manipulated facial images and videos, 2019 IEEE 10th Int. Conf. Biometrics Theory, Appl. Syst. BTAS 2019 (2019).
[37] H.H. Nguyen, J. Yamagishi and I. Echizen, Use of a capsule network to detect fake images and videos, arXiv, 2019.
[38] O.M. Parkhi, A. Vedaldi and A. Zisserman, Deep face recognition, Visual Geometry Group Department of Engineering Science University of Oxford, (2015) 1–12.
[39] C. Rathgeb, A. Botaljov, F. Stockhardt, S. Isadskiy, L. Debiasi, A. Uhl and C. Busch, PRNU-based detection of facial retouching, IET Biomet. 9(4) (2020) 154–164.
[40] A. Rossler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies and M. Niessner, FaceForensics++: Learning to detect manipulated facial images, Proc. IEEE Int. Conf. Comput. Vis. 2019 (2019) 1–11.
[41] E. Sabir, J. Cheng, A. Jaiswal, W. AbdAlmageed, I. Masi and P. Natarajan, Recurrent convolutional strategies for face manipulation detection in videos, arXiv, 2019.
[42] F. Schroff, D. Kalenichenko and J. Philbin, FaceNet: A unified embedding for face recognition and clustering, Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (2015) 815–823.
[43] S. Tariq, S. Lee, H. Kim, Y. Shin and S. Woo, Detecting both machine and human created fake face images in the wild, Proc. Int. Workshop Multimedia Priv. Secur.(2018) 81–87.
[44] J. Thies, M. Zollh¨ofer, M. Stamminger, C. Theobalt and M. Nießner, Face2face, Commun. ACM. 62(1) (2018) 96–104.
[45] J. Thies, M. Zollh¨ofer, M. Stamminger, C. Theobalt and M. Nießner, Deferred neural rendering: image synthesis using neural textures, ACM. Trans. Graph. 38(4) (2019).
[46] R. Tolosana, S. Romero-Tapiador, J. Fierrez and R. Vera-Rodriguez, Deepfakes evolution: analysis of facial regions and fake detection performance, Lect. Notes Comput. Sci. (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics), (2021) 12665.
[47] R. Wang, F. Juefei-Xu, L. Ma, X. Xie, Y. Huang, J. Wang and Y. Liu, Fakespotter: a simple yet robust baseline for spotting ai-synthesized fake faces, IJCAI Int. J. T. Conf. Artif. Intell. 2020 (2021) 3444–3451.
[48] S.Y. Wang, O. Wang, R. Zhang, A. Owens and A. Efros, Detecting photoshopped faces by scripting photoshop, Proc. IEEE Int. Conf. Comput. Vis. 2019 (2019) 10071–10080.
[49] X. Yang, Y. Li and S. Lyu, Exposing deep fakes using inconsistent head poses, ICASSP, IEEE Int. Conf. Acoust. Speech Signal Process-Proc. 2019 (2019) 8261–8265.
[50] N. Yu, L. Davis and M. Fritz, Attributing fake images to GANs: learning and analyzing GAN fingerprints, Proc. IEEE Int. Conf. Comput. Vis. 2019 (2019) 7555–7565.
[51] X. Zhang, S. Karaman and S.F. Chang, Detecting and simulating artifacts in GAN fake images, 2019 IEEE Int. Work Inf. Forensics Secur. WIFS 2019 (2019).
[52] P. Zhou, X. Han, V.I. Morariu and L.S. Davis, Two-stream neural networks for tampered face detection, IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit Work 2017 (2017) 1831–1839.
[48] S.Y. Wang, O. Wang, R. Zhang, A. Owens and A. Efros, Detecting photoshopped faces by scripting photoshop, Proc. IEEE Int. Conf. Comput. Vis. 2019 (2019) 10071–10080.
[49] X. Yang, Y. Li and S. Lyu, Exposing deep fakes using inconsistent head poses, ICASSP, IEEE Int. Conf. Acoust. Speech Signal Process-Proc. 2019 (2019) 8261–8265.
[50] N. Yu, L. Davis and M. Fritz, Attributing fake images to GANs: learning and analyzing GAN fingerprints, Proc. IEEE Int. Conf. Comput. Vis. 2019 (2019) 7555–7565.
[51] X. Zhang, S. Karaman and S.F. Chang, Detecting and simulating artifacts in GAN fake images, 2019 IEEE Int. Work Inf. Forensics Secur. WIFS 2019 (2019).
[52] P. Zhou, X. Han, V.I. Morariu and L.S. Davis, Two-stream neural networks for tampered face detection, IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit Work 2017 (2017) 1831–1839.
)
Volume 13, Issue 1
March 2022Pages 745-755
- Receive Date: 11 August 2021
- Revise Date: 20 September 2021
- Accept Date: 28 September 2021