In this work, we will first provide a brief overview of Machine Learning, then describe a new and vitally important area of Machine Learning called Adversarial Machine Learning, before focusing on Adversarial Training as one of the best solutions to the problems in the field of Adversarial Machine Learning. We will examine a recently proposed Adversarial Training algorithm and present our observations. Then provide a comprehensive study on a new concept for activation functions which helps in robustness of the ML models. Finally, propose a new research topic for further development of the new Adversarial Training method.
Right after the advent of the first Adversarial Machine Learning discussions, many researchers and engineers have started developing new algorithms for new attacks as well as new defenses on ML models. These attacks could lead to catastrophic results, for example by attacking autonomous driving cars they can cause misinterpretation of road signs and as a result heavy accidents will occur. All these new algorithms and methods are in need of powerful computational resources and time; almost all of the useful algorithms need hours and even days to compile (train the ML model) and neither everyone has access to such powerful resources, nor these resources are free to use. As a result, understanding the optimum methods is critical in order to reduce the amount of time and hardware required for training.
The relevance of the research topic lies in researching on new methods to make the Machine Learning models robust against different attacks. The object of the master's thesis is to find the most optimal algorithms in order to advance the robustness of Machine Learning models. The subject of research is to develop Adversarial Training techniques which are the most effective methods in increasing robustness of ML models.
The purpose of this research is to provide a study on Adversarial Machine Learning, or more precisely Adversarial Training in Computer Vision, and to find the most practical Adversarial Training algorithm in terms of being fast and robust at the time of writing this dissertation, as well as to suggest new research directions for future developments.
The scope of the research is to provide the results of conducted attacks on famous image classification datasets (CIFAR10, CIFAR100, Tiny ImageNet, ImageNet) using different attack methods.
To achieve the goals of this work, the following tasks were performed:
- Study theoretical resources to become familiar with concepts
- Identifying the existing researches on the subject.
- Understanding the problems in the subject.
- Identifying the outstanding solutions.
- Implementing the algorithms.
- Identifying the best available algorithm.
- Testing and tweaking the algorithm.
- Providing the results of the successful implementations.
- Identifying the potentials for further development.
- Suggesting new research directions for further development.
In this research first we investigated a proposed sample-dependent adversarial initialization to boost Fast AT. To produce an effective initialization, a generative network conditioned on a benign image and its gradient information from the target network were used. The generative network and the target network are optimized together and play a game during the training phase. The former learns to generate stronger adversarial instances depending on the current target network by producing a dynamic sample-dependent initialization. To improve model robustness, the latter uses the generated adversarial cases for training. The proposed initialization eliminates catastrophic overfitting, therefore improving model robustness, when compared to widely used random initialization methods in Fast AT. Extensive experimental results back up the proposed method's advantages.
In the second part of the research, we looked into a new concept called smooth adversarial training, which enforces architectural smoothness by using adversarial training to replace nonsmooth activation functions with their smooth approximations. Without compromising accuracy or incurring additional calculation costs, SAT improves adversarial robustness. Extensive research has shown that SAT is generally effective. SAT reports the state-of-the-art adversarial robustness on ImageNet with EfficientNetL1, which exceeds the previous art by 9.5 percent in accuracy and 11.6 percent in robustness.
