What concerns do tech experts have with the use of AI within autonomous weapons? What are some potential ways of regulating it including the California Privacy Rights Act, to make it safer for civilians?
Yoshiko Hirata
CUNY School of Professional Studies
COM110: Digital Literacy
Prof. Nina Hien
May 8, 2026
Problems of AI-backed Autonomous Weapons Systems (AWS)
AI-backed Autonomous Weapons Systems (AWS) race is escalating among U.S., China and Russia (Frenkel et al., 2026, para. 4). AWS are “weapons that, once activated, can identify, select and apply force (lethal or non-lethal) to targets without human intervention.” (Bruun et al., 2023, p. 2). Anduril, one of the private companies participating in the US military’s Project Maven which seeks to integrate AI technology into the military, produces autonomous weapons such as fighter jets, submarines, and drones with Anduril’s AI integrated operation system, Lattice. Palmer Lucky (2025), Anduril’s founder, likes to describe a possible war scenario where China attacks Taiwan with a shower of ballistic missiles and his autonomous system would quickly defend the U.S. ally (00:01). He believes arming up with powerful AI-AWS military technology may even prevent future wars because it acts as a deterrent much like nuclear weapons did during the Cold War (Frenkel et al., 2026, para. 51). The key function of AI technology in military operations—referred to as Artificial Intelligence-Decision Support System (AI-DSS) – is to create a large number of targets in a short amount of time. In the war in Gaza, for example, the IDF’s AI-DSS generated at least 37,000 target recommendations in first six month (Andersin, 2025, para. 2). The US war against Iran struck more than 13,000 targets in 38 days (Schmitt & Swan, 2026, para. 5).
“People failed to update a database, and other people built a system fast enough to make that failure lethal.”
Kevin T Baker, 2026, para. 4
Bruun et al. (2023) points out that “legal questions arise primarily when autonomy is used for targeting tasks such as searching, identifying, tracking, prioritizing and engaging of targets.” (p. 3). AI-DSS generates targets based on various datasets, such as “photographs, satellite imagery, geolocation data (IP address, geotag, metadata, etc) from communications intercepts, infrared sensors, synthetic-aperture radar, and more” (Project Maven, 2026). In both wars in Ukraine and Gaza, some targets were derived from commercially collected datasets such as facial photographs from social media or google, voice transcription from personal communications, and location tracking services (Onderco, p. 614) (Andersin, 2025, para. 5-6). Andersin (2025) claims that when AI-DSS generates recommendations with bad data and an over-inclusive definition of militants, “outputs will inevitably be systematically flawed,” and innocent civilians are killed or harmed (para. 9). Further, when AI-DSS targets individuals at home it results in increasing civilian casualties (para. 5). Vanderborght & Nadibaidze (2025) point out “[a]s the current armed conflicts in Ukraine and Gaza demonstrate, fighting occurs in densely populated areas, with civilians and civilian infrastructures closely entangled with military targets.” (p. 17). In Gaza, 80% of the total death toll was civilians (Casualties of the Gaza war, 2026). In Iran, an AI targeting error of an elementary school killed 175 people, mainly girls between age of seven to 12, and was the result of using a bad data, at least a decade old. “People failed to update a database, and other people built a system fast enough to make that failure lethal.” (Baker, 2026, para. 4).
Ultimately these concerns raise a question. How can we protect civilians from AI-backed Autonomous Weapons Systems, especially when AI-DSS creates thousands of targets using data collected from our regular internet or phone activities? Currently there are no central regulations over use of AI DSS nor AI AWS in war. Guidelines must made up with multiple existing laws and regulations.
Evolving Frameworks to Protect Civilians from AI Harms
International Humanitarian Law (IHL) is the most regarded law to protect civilians in war. It has three principles: distinction, proportionality, and precaution. Distinction differentiates civilians and civilian objectives from combatants and military objectives. This includes the distinction between wounded and surrendered combatants from active combatants. Proportionality prohibits an attack that causes excessive loss and injury to civilians and civilian objects related to the military advantage of the attack. Finally, the principle of precaution requires the continuous attention to not harm civilians (Bruun et al., 2023, p. 2, Table 1.1). IHL is developed for human operators, and so the use of AWS and its lack of direct human judgement make it harder to comply with IHL. Gerald Mako (2026) argues that Martens Clause, which is an adherence to IHL’s core principles, requires “weapons and warfare practices comply with the principles of humanity and with the dictates of public conscience, even absent specific legal prohibitions.” Although IHL doesn’t ban autonomy, with the Martens Clause, Mako suggests that IHL’s core obligations implicitly require “meaningful human control”, to protect civilians and combatants (para. 14-15).
Unknown author, Public domain, via Wikimedia Commons
“Until a more complete code of the laws of war has been issued, the High Contracting Parties deem it expedient to declare that, in cases not included in the Regulations adopted by them, the inhabitants and the belligerents remain under the protection and the rule of the principles of the law of nations, as they result from the usages established among civilized peoples, from the laws of humanity, and the dictates of the public conscience.”
Laws and Customs of War on Land (Hague IV), 18 October 1907 (Martens Clause, 2026)
The Political Declaration on Responsible Military Use of Artificial Intelligence and Autonomy was launched under Biden administration in 2023 to create international agreement for responsible use of AI in military (U.S. Department of State, 2024). Fifty-eight countries signed on by the end of 2024, but there has been no update or efforts made under Trump administration.
AI Diffusion Rule was created to control chip exports under Biden administration. US chips would be only sold to the countries who align with same domestic regulations for how the technology can be used. But it was canceled under Trump administration (U.S. Department of Commerce, 2025).
EU Artificial Intelligence Act. (EU, 2024) is the European Union’s AI regulation. There are four levels of risk – unacceptable, high, limited, and minimal. AI systems with unacceptable risk level are prohibited such as: manipulative technique, biometric categorization systems, social scoring systems, or compiling facial recognition databases. High-risk AI providers must practice: risk management, data governance, record-keeping and human oversight. The regulation applies to any companies who operate in the EU regardless of their location. However, EU AI Act does not apply to military practice (Future of Life Institute, 2024).
Transparency in Frontier Artificial Intelligence Act (TFAIA) (CA, 2025) is a California State AI transparency bill to require AI developers to publish a framework of how the company incorporated international and national standards. “California continues to dominate the AI sector. In addition to being the birthplace of AI, the state is home to 32 of 50 top AI companies worldwide.” (Newsom, 2025). Earlier California was about to pass The Safe and Secure Innovation for Frontier Artificial Intelligence Models Act (SB 1047), which was designed to alleviate the risk of AI weapons, but Governor Newsom vetoed this bill (Safe and Secure Innovation for Frontier Artificial Intelligence Models Act, 2026).
The Responsible AI Safety and Education (RAISE) Act (NY, 2025) is a New York State AI safety bill to require “large AI developers to create and publish information about their safety protocols, and report incidents to the State within 72 hours of determining that an incident occurred.” (Hochul, 2025). Alex Bores, a NY state assemblyman and a former data scientist at Palantir, regrets that the final version didn’t include provisions such as prohibition of model release that failed in their own safety test, and third-party audits (Klein 22:43 – 25:57). He is currently running for a 2026 House of Representative race. He promises to work on federal legislation for AI safety if he is elected.
The General Data Protection Regulation (GDPR) (EU, 2016) is an European Union’s data protection law and protects consumers’ rights to make decisions about their personal data. It includes restrictions “against automated decision-making systems that use inputted data to make a decision without any interference from a human.” It applies to any businesses, regardless of their location, who collects or processes personal data of E.U. citizens or residents. It is the most restrictive data privacy provisions in the world. (Lisowski, 2024, pp. 706-708, 715)
The California Consumer Privacy Act (CCPA) (CA, 2018), The California Privacy Rights Act (CPRA) (CA, 2020) are California’s data privacy laws that requires businesses who collect data from California residents to inform consumers what data is collected, how the data is used and shared, and the consumer’s rights and options. “The CPRA maintains six primary rights of the CCPA, and will likely lead to the addition of new rights, including the right to opt-out of automated decision-making technology.” (Lisowski, 2024, pp. 710-711)
Conclusion
While writing this research, the Pentagon announced an additional deal with Amazon Web Services, Microsoft, Nvidia and Reflection AI, beyond its already existing deals with xAI, OpenAI and Google. More private AI companies are participating with the military. These partnerships are meant to “help service members make faster and better decisions.” (Barnes, et al., 2026, para. 6) There is a lot of money flowing into AI technologies and these companies are building large data centers throughout the country. As Anthropic’s founder Dario Amodei questioned, there must be a way for AI to “reinvent democracy to enhance our rights and liberties instead of reducing it.” (Douthat, 2026, 9:16-13:10). To actualize that, we need strong guardrails and “meaningful human control” imbedded into our AI systems and workflow. And if such guardrails are not established at the international nor federal level, at least it can start at the state level such as we have seen in California and New York. Furthermore, it can start at an even smaller level such as school, office and household, where individuals and communities take responsibility for their data use.
This is a sequel to previous research:
What Does the Absence of Regulations in AI Use in the Military, Warn Us for the Future of Society?
References
Andersin, E. (2025, October 24). Military use of Biometrics Series – israel’s use of AI-DSS and facial recognition technology: The erosion of civilian protection in Gaza. Articles of War. Lieber Institute West Point. https://lieber.westpoint.edu/israels-use-ai-dss-facial-recognition-technology-erosion-civilian-protection-gaza/
Baker, K. T. (2026, March 26). AI got the blame for the Iran school bombing. The truth is far more worrying. The Guardian. https://www.theguardian.com/news/2026/mar/26/ai-got-the-blame-for-the-iran-school-bombing-the-truth-is-far-more-worrying
Barnes, J. & Frenkel, S. (2026, May 1) Pentagon Makes Deals With A.I. Companies to Expand Classified Work. The New York Times. https://www.nytimes.com/2026/05/01/us/politics/pentagon-ai-companies-deals.html
Bruun, L., Bo, M., & Goussac, N. (2023, March). Compliance with International Humanitarian Law in the Development and Use of Autonomous Weapon Systems: What does IHL Permit, Prohibit and Require? Stockholm International Peace Research Institute. https://doi.org/10.55163/DFXR3984
Casualties of the Gaza war. (2026, April 15). In Wikipedia. Retrieved 05:13, May 1, 2026, from https://en.wikipedia.org/w/index.php?title=Casualties_of_the_Gaza_war&oldid=1349039207
Douthat, R. (Host). (2026, February 12). Opinion | anthropic’s chief on A.I.: ‘we don’t know if the models are conscious’[Audio podcast episode]. In Interesting Times, The New York Times. https://www.nytimes.com/2026/02/12/opinion/artificial-intelligence-anthropic-amodei.html
Frenkel, S., Mozur, P., & Satariano, A. (2026, April 12) Mutually Automated Destruction: The Escalating Global A.I. Arms Race. The New York Times. https://www.nytimes.com/2026/04/12/technology/china-russia-us-ai-weapons.html
Future of Life Institute. (2024, February 27) High-level summary of the AI Act. https://artificialintelligenceact.eu/high-level-summary/
Hochul, K. (2025, December 19) Governor Hochul Signs Nation-Leading Legislation to Require AI Frameworks for AI Frontier Models [Press release]. https://www.governor.ny.gov/news/governor-hochul-signs-nation-leading-legislation-require-ai-frameworks-ai-frontier-models
Klein, E. (Host). (2026, April 21). Opinion | Why Are Palantir and OpenAI Scared of Alex Bores? [Audio podcast episode]. In The Ezra Klein Show. The New York Times. https://www.nytimes.com/2026/04/21/opinion/ezra-klein-podcast-alex-bores.html
Lisowski, J. N. (2024). California Data Privacy Law and Automated Decision-Making. Journal of Corporation Law, 49(3), 701-726. https://remote.baruch.cuny.edu/login?url=https://www.proquest.com/scholarly-journals/california-data-privacy-law-automated-decision/docview/3048976489/se-2
Luckey, P. (2025, April). The AI arsenal that could stop World War III [Video]. TED Conferences. https://www.ted.com/talks/palmer_luckey_the_ai_arsenal_that_could_stop_world_war_iii
Mako, G. (2026, March 9). Legal accountability for AI-driven autonomous weapons. Articles of War. Lieber Institute West Point. https://lieber.westpoint.edu/legal-accountability-ai-driven-autonomous-weapons/
Martens Clause. (2026, January 9). In Wikipedia. Retrieved 03:51, May 8, 2026, from https://en.wikipedia.org/w/index.php?title=Martens_Clause&oldid=1332105889
Newsom, G. (2025, September 29) Governor Newsom signs SB 53, advancing California’s world-leading artificial intelligence industry [Press release]. https://www.gov.ca.gov/2025/09/29/governor-newsom-signs-sb-53-advancing-californias-world-leading-artificial-intelligence-industry/
Onderco, M. (2025). Navigating the AI frontier: Insights from the Ukraine conflict for NATO’s governance role in military AI. Journal of Strategic Studies, 48(3), 602–626. https://doi.org/10.1080/01402390.2025.2463451
Project Maven. (2026, March 18). In Wikipedia. Retrieved 22:15, March 22, 2026, from https://en.wikipedia.org/w/index.php?title=Project_Maven&oldid=1344180237
Safe and Secure Innovation for Frontier Artificial Intelligence Models Act. (2026, March 28). In Wikipedia. Retrieved 05:11, May 6, 2026, from https://en.wikipedia.org/w/index.php?title=Safe_and_Secure_Innovation_for_Frontier_Artificial_Intelligence_Models_Act&oldid=1345911912
Schmitt, E., & Swan, J. (2026, April 23) Iran War Has Drained U.S. Supplies of Critical, Costly Weapons. The New York Times. https://www.nytimes.com/2026/04/23/us/politics/iran-war-cost-military.html
U.S. Department of Commerce. (2025, May 13). Department of Commerce Announces Rescission of Biden-Era Artificial Intelligence Diffusion Rule, Strengthens Chip-Related Export Controls [Press release]. https://www.bis.gov/press-release/department-commerce-announces-rescission-biden-era-artificial-intelligence-diffusion-rule-strengthens
U.S. Department of State. (2024, November) Political Declaration on Responsible Military Use of Artificial Intelligence and Autonomy [Press release]. https://www.state.gov/bureau-of-arms-control-deterrence-and-stability/political-declaration-on-responsible-military-use-of-artificial-intelligence-and-autonomy
Vanderborght, R., & Nadibaidze, A. (2025). Military demonstrations as digital spectacles: How virtual presentations of AI decision-support systems shape perceptions of war and security. European Journal of International Security, 1–20. https://doi.org/10.1017/eis.2025.10015
