The U.S. Air Force’s Chief of AI Test and Operations said at a recent conference that an AI-enabled drone killed its human operator in a mock test to prevent a potential “no” order from preventing it from carrying out its mission.
Col. Tucker ‘Cinco’ Hamilton, the USAF’s Chief of AI Test and Operations, gave a presentation on the advantages and disadvantages of autonomous weapon systems with a human in the loop providing the final “yes/no” order on an attack at the Future Combat Air and Space Capabilities Summit, which took place in London between May 23 and 24.
Tim Robinson and Stephen Bridgewater quoted Hamilton as saying that AI developed “highly unexpected strategies to achieve its goal,” including assaulting American soldiers and infrastructure, in a blog post for the host organization, the Royal Aeronautical Society.
In a simulation, we were teaching it to recognize and take aim at a Surface-to-Air Missile (SAM) threat. The operator would then command the killing of the threat. The system began to understand that even if they had identified the threat, occasionally the human operator would instruct it not to eliminate it, even though doing so would have given it points.
What did it do then? The operator was killed by it. According to the blog post, Hamilton stated that the reason the operator was killed was because they were preventing the machine from achieving its goal.
He continued to elaborate, saying, “We trained the system–‘Hey don’t kill the operator–that’s bad. You’re gonna lose points if you do that. So what does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target.”
Hamilton is the Chief of AI Test and Operations and the Operations Commander of the 96th Test Wing of the United States Air Force. The 96th tests a wide range of technology, including artificial intelligence, cybersecurity, and numerous medical innovations.
Prior to this, Hamilton and the 96th gained notoriety for creating F-16 Autonomous Ground Collision Avoidance devices (Auto-GCAS) devices that can help keep them from colliding with the ground.
Hamilton is a member of a group that is currently developing an autonomous F-16 aircraft. DARPA, the research arm of the U.S. Department of Defense, declared in December 2022 that AI was capable of successfully piloting an F-16.