Autonomous Weapon System and Artificial Intelligence: The Problems of Arms Control
|
Introduction 4
Chapter 1. Applying artificial intelligence for the development of autonomous weapon system 9
1.1. The development of the technologies 9
1.1.1. The development of the autonomy in weapon system 9
1.1.2. Changes bought by AWSs to combatants 11
1.2 The application of autonomous weapon system in military domain 12
1.2.1. Evaluation of existing system and experience 13
1.2.2. Autonomous weapon system and artificial intelligence as temptation to states 16
1.3. Problems that arise from the development of autonomous weapon system 18
1.3.1. Technical shortage of autonomous weapon system 18
1.3.1. Who should care responsibility for consequence? 20
1.3.2 Analysis of Authorities 21
1.3.2. Meaningful Human Control 23
Chapter 2. Bitter truth behind the surface - threats and challenges posting by autonomous weapon system 26
2.1. Autonomous weapon system into reality, the devil or the hero? 26
2.1.1. Debates of autonomous weapon system and artificial intelligence under the
international humanitarian law 26
2.1.2. Ethical challenges brought by autonomous weapon system and artificial intelligence ... 31
2.2. Autonomous weapon system and artificial intelligence as an impact on strategic stability
and international security 33
2.2.1. Artificial intelligence and the rise of nuclear war 34
2.2.2. Strategic Stability under the ear the autonomous weapon system and artificial
intelligence 36
2.2.3. Merging of autonomous weapon system with artificial intelligence as a challenge for
international security 40
Chapter 3. International control is needed and there is a long way to go 44
3.1. Proliferation of autonomous weapon system is possible and needed to be wary 44
3.1.1. Necessity of setting control 46
3.1.2. The problem of setting control 51
3.2. Mapping the approaches of setting control 57
3.2.1. Steps to autonomous weapon system arms control 57
3.2.2. Arguments under the frame of CCW 62
3.2.3. Possible solution to the problem of setting control 65
Conclusion 70
List of references 74
Annexes
Chapter 1. Applying artificial intelligence for the development of autonomous weapon system 9
1.1. The development of the technologies 9
1.1.1. The development of the autonomy in weapon system 9
1.1.2. Changes bought by AWSs to combatants 11
1.2 The application of autonomous weapon system in military domain 12
1.2.1. Evaluation of existing system and experience 13
1.2.2. Autonomous weapon system and artificial intelligence as temptation to states 16
1.3. Problems that arise from the development of autonomous weapon system 18
1.3.1. Technical shortage of autonomous weapon system 18
1.3.1. Who should care responsibility for consequence? 20
1.3.2 Analysis of Authorities 21
1.3.2. Meaningful Human Control 23
Chapter 2. Bitter truth behind the surface - threats and challenges posting by autonomous weapon system 26
2.1. Autonomous weapon system into reality, the devil or the hero? 26
2.1.1. Debates of autonomous weapon system and artificial intelligence under the
international humanitarian law 26
2.1.2. Ethical challenges brought by autonomous weapon system and artificial intelligence ... 31
2.2. Autonomous weapon system and artificial intelligence as an impact on strategic stability
and international security 33
2.2.1. Artificial intelligence and the rise of nuclear war 34
2.2.2. Strategic Stability under the ear the autonomous weapon system and artificial
intelligence 36
2.2.3. Merging of autonomous weapon system with artificial intelligence as a challenge for
international security 40
Chapter 3. International control is needed and there is a long way to go 44
3.1. Proliferation of autonomous weapon system is possible and needed to be wary 44
3.1.1. Necessity of setting control 46
3.1.2. The problem of setting control 51
3.2. Mapping the approaches of setting control 57
3.2.1. Steps to autonomous weapon system arms control 57
3.2.2. Arguments under the frame of CCW 62
3.2.3. Possible solution to the problem of setting control 65
Conclusion 70
List of references 74
Annexes
Arms control of AWS is an emerging issue in the field of international arms control in recent years. One of the reasons is that with the rapid development of AWS and extensive militarization of AI technology, more and more potential risks that will eventually post threats on human beings have appeared. Although AWS brings many advantages that may greatly increase military power, risk and challenges at the ethic and humanitarian level and the impact on international security cannot be neglected.
Since 2013, attention of the international community has been attracted to the emergence of AWS and problems of its arms control. In 2014, the parties held dialogues at the United Nations, and the whole dialogue has been conducted within the framework of the Convention on Conventional Weapons (CCW). The informal dialogue evolved into formal negotiations in 2017, involving more than 70 countries and many scientists and non-governmental organizations. Even though there are issues that slow down the process of setting control during the years, more and more countries have realized that the need is urgent, and the are ready to achieve more actual results. From the current development of the technology and possible scenario that might occur in the coming future, it is possible on the technical level and necessary for the international community to push forward the arms control negotiations of autonomous weapons systems. The “Eleven Guiding Principles” is one of the remarkable results in recent years, which shows the determination of the international community to push forward the process of AWS arms control.
Since the technology has caught everyone’s attention, the term “Lethal Autonomous Weapon System (LAWS)” has been used in many researches and articles, while in my research I use the term “Autonomous Weapon System.” Depending on how it will serve the purposes, the word lethal deserves more discussion. The AWS does not have to be lethal. Although there are discussions about “killer robots” is in many contexts, the discussion of how to control and regulate is also within the frame of many study. It is very questionable to consider AWS and AI technology as evil and homicidal technology. There is nothing wrong with the technology itself. It is the user and the way to use it that creates the problem. The designers construct the input and process international law and other normative data into the machine. At the same time, at what level should AI be trained before it can be put on the battlefield has also caught attention. These are the new points of the problem. In order to avoid misleading in my research, I decided to use the term “Autonomous weapon system” instead of “Lethal Autonomous weapon system.”
Development and deployment of AWSs is known as the ‘the third revolution in the warfare.’ And it will be necessary to establish a proper rule to regulate the military use of AI and the increasing potential appearance of AWSs in the future. While there are many articles that focus on the technical problem and ethical aspects, international concerns, there is a lack of research in the field of why these are problems and what is happening in the process of setting control. What is the prospects of the solution and what can we expect from the future. AWS has the ability to respond to changes in actual circumstances, which automatic weapon systems do not and can only operate based on human instruction entirely. An automatic weapon system is a step on the way to explore autonomy in the weapon system. This study will focus on autonomous weapon systems instead of automatic weapon systems.
Since 2013, attention of the international community has been attracted to the emergence of AWS and problems of its arms control. In 2014, the parties held dialogues at the United Nations, and the whole dialogue has been conducted within the framework of the Convention on Conventional Weapons (CCW). The informal dialogue evolved into formal negotiations in 2017, involving more than 70 countries and many scientists and non-governmental organizations. Even though there are issues that slow down the process of setting control during the years, more and more countries have realized that the need is urgent, and the are ready to achieve more actual results. From the current development of the technology and possible scenario that might occur in the coming future, it is possible on the technical level and necessary for the international community to push forward the arms control negotiations of autonomous weapons systems. The “Eleven Guiding Principles” is one of the remarkable results in recent years, which shows the determination of the international community to push forward the process of AWS arms control.
Since the technology has caught everyone’s attention, the term “Lethal Autonomous Weapon System (LAWS)” has been used in many researches and articles, while in my research I use the term “Autonomous Weapon System.” Depending on how it will serve the purposes, the word lethal deserves more discussion. The AWS does not have to be lethal. Although there are discussions about “killer robots” is in many contexts, the discussion of how to control and regulate is also within the frame of many study. It is very questionable to consider AWS and AI technology as evil and homicidal technology. There is nothing wrong with the technology itself. It is the user and the way to use it that creates the problem. The designers construct the input and process international law and other normative data into the machine. At the same time, at what level should AI be trained before it can be put on the battlefield has also caught attention. These are the new points of the problem. In order to avoid misleading in my research, I decided to use the term “Autonomous weapon system” instead of “Lethal Autonomous weapon system.”
Development and deployment of AWSs is known as the ‘the third revolution in the warfare.’ And it will be necessary to establish a proper rule to regulate the military use of AI and the increasing potential appearance of AWSs in the future. While there are many articles that focus on the technical problem and ethical aspects, international concerns, there is a lack of research in the field of why these are problems and what is happening in the process of setting control. What is the prospects of the solution and what can we expect from the future. AWS has the ability to respond to changes in actual circumstances, which automatic weapon systems do not and can only operate based on human instruction entirely. An automatic weapon system is a step on the way to explore autonomy in the weapon system. This study will focus on autonomous weapon systems instead of automatic weapon systems.
The deployment of AWS may have some positive consequences, like decreasing the casualties, improving operation efficiency, reducing the physical and cognitive pressure on officers and soldiers, reducing the burden of the decision-making though improving command and control support systems, reducing the costs of war, fighting within the prescribed scope of moral operations, providing realistic, comprehensive training, expanding the breadth and depth of operations. These have stimulated the arms race and intensified the unstable global situation. The possible benefits of AWS and AI increased investment and research for the core technologies and talents of advanced technology in order to seize the technological advantages and initiatives.
However, in a short period of time, there are problems such as black box that might lead to malfunction. The unpredictability and uncertainty bought by the black box and malfunction might violate sovereignty and cause unnecessary conflict. Moreover, because AWS is based on AI technology, there are problems with accountability during design and operation. Who should take the responsibility remains. Thus, maintaining meaningful human control within the AWS is important, which could stop situations from being escalated by machines and prevent the machine from being completely out of control.
In the future, more and more autonomy will be found in the characteristics of military weapons and systems, which will be used more and more widely in different real scenarios. Whether the emergence of AWSs will bring harm to human society may more and more depend on the capability to make AWSs under control through technical programming and regulatory constraints. AWSs will focus on how to pursue the target successfully instead of thinking about whether it is right. Therefore, that leads to another problem that AWSs does not consider the consequences of the action and the social justice of the attack in the absence of moral judgment and legal statute. AWSs will meet challenges from IHL, which is the principles of distinguishment, the principle of proportionality, and whether it could compliance with the "Martens Clause." AWS neither does have the ability to learn ethical standards nor the standard can be coded into the program. AWS also faces the ethical level as well. This is not only a violation of a certain law or moral standards but also a threat to the whole society. Although there are many challenges from moral and ethical level, but these are not enough for the arms control on AWSs.
AWSs may effect strategic stability. How the emergency of AWS and AI will have an impact on the strategic stability depends on the degree of development and the way how AWS and AI technology will be applied to the weapon systems. AWSs can provide support to command and control system defenses to make the system more efficient as well as defenses against potential interference. What is more, AWS and AI technology may be used for intelligence, surveillance, and reconnaissance systems that will help to gain intelligence. This will increase the vulnerability of nuclear infrastructure.
The technological process will enlarge the gap between countries. Unlike the first industrial revolution, rather than population size of countries, technologies will become one of the most important elements to national power. And the small or not so well developed countries that might lack the objective technology to develop AWS and AI technologies are also the supporters to total ban or set control on AWS. However, at the same time, countries with technological advantages are lack of willingness to control. One of the reasons behind this is the variety of possibilities and potential benefits in the future. Besides, fear of rivals developing AWS in secret is one of the driving forces behind the development, which reduce global security and led to a new arms race. AI has a significant dual-use nature. Thus, it is not difficult to transform civilian AI technology into military AWS.
Therefore we can learn that, currently AWSs have not greatly threat international security or strategic stability, as well as challenge the humanitarian and human rights, thus leading countries have no willingness to set international control on AWSs. As leading country repeatedly expressed their desire to explore potential “advantages” or “benefits” to developing and using autonomous weapons systems, the AWSs seems quite profitable. An important factor that weakens the country's willingness to promote AWS arms control is fear such a regulation could be disadvantageous for their military strategy. No one will take any risk of losing advantages in field like AWS, especially in the era of AI. Other reasons like AWSs as a set of software program, ban or regulate could led to the proliferation in the black market. It will be even more difficult to set control if that happen. And AWSs as a new technology is too complicated to terrorists all become reasons of no need to set control. Besides, many AWS and AI technology in military were initially designed to meet the needs of special combat environments, does not have great harmful to innocents or civilians yet. However, that does not mean it will not happen in the future.
The necessity of setting control over the AWS is being showed from different prospectives. Several element is urgently need of attention, such as standardize the research and development and definition of the control function in the AWSs. These will make sure the development of AWSs will always with control of human. Several parts should be pay attention to are:
Firstly, time interval between human decision making and machine decision making. This will help human to stop machine from doing unexpected thing and give human enough time to react. In order to clarify the part that human plays in the process of AWSs, the specific restrictions and requirements should be necessary made by human on the identification, decision-making and method selection of their attack targets. Secondly, deployment environment should be take into consideration, which will minimize the accidental damage to civilians and innocents. And thirdly the quality of machine operators such as designer and the skill of operator. This will help to maximize the the advantages of AWSs and minimize the potential malfunction and human negative interface.
Although the total ban of the AWS is almost impossible, the possibility of arms control is not lost. The AWSs gives us a glance off the possibility of the intelligent weapon that can bring change to the warfare. Imposing a total ban will depriving ourselves of tools that can reduce the suffer of people during the war. One of the reasons is that at present, more and more world leaders and military experts of various countries have realized that if the AWS is completely out of human control, the world could slip into a dangerous situation which intensifying arms race, threatening international security and strategic stability, challenging humanitarian and human rights seriously, and posing a threat to human ourselves.
Discussions about AWS have become a topical issue at the international level. The framework of the CCW, which brings together a wide range of national and non-State actors, has become a central platform for international LAWS arms control discussions. CCW has played an important role in promoting the AWS arms control discussion and has achieved some success. However, with increased competition from big powers and the temptation of the militarization of AI, no substantial progress has been made on how to proceed with AWS arms control negotiations and what form of effective international legally binding instrument should be constructed.
Many issues, such as debates over the AWS definition, the pros and cons of AWS military application, and legal challenges, remain unresolved, which slow down the arms control of AWS. Thus, it needs countries to continue and to be even more actively participate in AWS arms control discussions within the CCW framework, managing the risks, ethical and legal challenges posed by AWS. In addition, the negotiations of the Convention also require small and medium-sized developing countries to join together, and their voice needed to be heard and strengthen in the international community. The breakthrough was made in 2019 when all the participants in the CCW for the first time reached the "Eleven Guiding Principles," which is being accepted broadly. As a starter to the future consensus showed the international community that arms control of AWS is possible and doable. Besides, the arms control of AWSs can start from less relevant field to core technologies. For example, as it is mentioned previously the operation field and operator.
The revolution of AWS and AI will bring unprecedented change to the international community, and the process of change itself has already created unpredictable risks. The use of AWS and AI is like the use of electricity. There are many different ways to use it based on method and level. Countries have their own expectations of AWS and AI. Thus it is almost impossible to prohibit any country from using it. How to have forethought so that the impact of the militarization of technology does not harm humanity itself is a solution to make it better develop in the future.
However, in a short period of time, there are problems such as black box that might lead to malfunction. The unpredictability and uncertainty bought by the black box and malfunction might violate sovereignty and cause unnecessary conflict. Moreover, because AWS is based on AI technology, there are problems with accountability during design and operation. Who should take the responsibility remains. Thus, maintaining meaningful human control within the AWS is important, which could stop situations from being escalated by machines and prevent the machine from being completely out of control.
In the future, more and more autonomy will be found in the characteristics of military weapons and systems, which will be used more and more widely in different real scenarios. Whether the emergence of AWSs will bring harm to human society may more and more depend on the capability to make AWSs under control through technical programming and regulatory constraints. AWSs will focus on how to pursue the target successfully instead of thinking about whether it is right. Therefore, that leads to another problem that AWSs does not consider the consequences of the action and the social justice of the attack in the absence of moral judgment and legal statute. AWSs will meet challenges from IHL, which is the principles of distinguishment, the principle of proportionality, and whether it could compliance with the "Martens Clause." AWS neither does have the ability to learn ethical standards nor the standard can be coded into the program. AWS also faces the ethical level as well. This is not only a violation of a certain law or moral standards but also a threat to the whole society. Although there are many challenges from moral and ethical level, but these are not enough for the arms control on AWSs.
AWSs may effect strategic stability. How the emergency of AWS and AI will have an impact on the strategic stability depends on the degree of development and the way how AWS and AI technology will be applied to the weapon systems. AWSs can provide support to command and control system defenses to make the system more efficient as well as defenses against potential interference. What is more, AWS and AI technology may be used for intelligence, surveillance, and reconnaissance systems that will help to gain intelligence. This will increase the vulnerability of nuclear infrastructure.
The technological process will enlarge the gap between countries. Unlike the first industrial revolution, rather than population size of countries, technologies will become one of the most important elements to national power. And the small or not so well developed countries that might lack the objective technology to develop AWS and AI technologies are also the supporters to total ban or set control on AWS. However, at the same time, countries with technological advantages are lack of willingness to control. One of the reasons behind this is the variety of possibilities and potential benefits in the future. Besides, fear of rivals developing AWS in secret is one of the driving forces behind the development, which reduce global security and led to a new arms race. AI has a significant dual-use nature. Thus, it is not difficult to transform civilian AI technology into military AWS.
Therefore we can learn that, currently AWSs have not greatly threat international security or strategic stability, as well as challenge the humanitarian and human rights, thus leading countries have no willingness to set international control on AWSs. As leading country repeatedly expressed their desire to explore potential “advantages” or “benefits” to developing and using autonomous weapons systems, the AWSs seems quite profitable. An important factor that weakens the country's willingness to promote AWS arms control is fear such a regulation could be disadvantageous for their military strategy. No one will take any risk of losing advantages in field like AWS, especially in the era of AI. Other reasons like AWSs as a set of software program, ban or regulate could led to the proliferation in the black market. It will be even more difficult to set control if that happen. And AWSs as a new technology is too complicated to terrorists all become reasons of no need to set control. Besides, many AWS and AI technology in military were initially designed to meet the needs of special combat environments, does not have great harmful to innocents or civilians yet. However, that does not mean it will not happen in the future.
The necessity of setting control over the AWS is being showed from different prospectives. Several element is urgently need of attention, such as standardize the research and development and definition of the control function in the AWSs. These will make sure the development of AWSs will always with control of human. Several parts should be pay attention to are:
Firstly, time interval between human decision making and machine decision making. This will help human to stop machine from doing unexpected thing and give human enough time to react. In order to clarify the part that human plays in the process of AWSs, the specific restrictions and requirements should be necessary made by human on the identification, decision-making and method selection of their attack targets. Secondly, deployment environment should be take into consideration, which will minimize the accidental damage to civilians and innocents. And thirdly the quality of machine operators such as designer and the skill of operator. This will help to maximize the the advantages of AWSs and minimize the potential malfunction and human negative interface.
Although the total ban of the AWS is almost impossible, the possibility of arms control is not lost. The AWSs gives us a glance off the possibility of the intelligent weapon that can bring change to the warfare. Imposing a total ban will depriving ourselves of tools that can reduce the suffer of people during the war. One of the reasons is that at present, more and more world leaders and military experts of various countries have realized that if the AWS is completely out of human control, the world could slip into a dangerous situation which intensifying arms race, threatening international security and strategic stability, challenging humanitarian and human rights seriously, and posing a threat to human ourselves.
Discussions about AWS have become a topical issue at the international level. The framework of the CCW, which brings together a wide range of national and non-State actors, has become a central platform for international LAWS arms control discussions. CCW has played an important role in promoting the AWS arms control discussion and has achieved some success. However, with increased competition from big powers and the temptation of the militarization of AI, no substantial progress has been made on how to proceed with AWS arms control negotiations and what form of effective international legally binding instrument should be constructed.
Many issues, such as debates over the AWS definition, the pros and cons of AWS military application, and legal challenges, remain unresolved, which slow down the arms control of AWS. Thus, it needs countries to continue and to be even more actively participate in AWS arms control discussions within the CCW framework, managing the risks, ethical and legal challenges posed by AWS. In addition, the negotiations of the Convention also require small and medium-sized developing countries to join together, and their voice needed to be heard and strengthen in the international community. The breakthrough was made in 2019 when all the participants in the CCW for the first time reached the "Eleven Guiding Principles," which is being accepted broadly. As a starter to the future consensus showed the international community that arms control of AWS is possible and doable. Besides, the arms control of AWSs can start from less relevant field to core technologies. For example, as it is mentioned previously the operation field and operator.
The revolution of AWS and AI will bring unprecedented change to the international community, and the process of change itself has already created unpredictable risks. The use of AWS and AI is like the use of electricity. There are many different ways to use it based on method and level. Countries have their own expectations of AWS and AI. Thus it is almost impossible to prohibit any country from using it. How to have forethought so that the impact of the militarization of technology does not harm humanity itself is a solution to make it better develop in the future.