SAN FRANCISCO — Iran, China and groups in Israel used artificial intelligence in novel ways in recent months to create first-of-their-kind influence campaigns on social media, ratcheting up concerns among U.S. officials and security researchers about the fast-evolving technology.
The countries and groups combined AI tactics to ramp up the speed and scale of online campaigns beyond what people could easily achieve and with minimal human interaction, U.S. officials and security researchers with knowledge of the efforts said.
First, the countries and groups turned to Chinese "open-source" AI models, which are increasingly powerful systems that are freely available to download and modify. Then they used those AI models to create "agents," which are bots that can operate software and complete tasks on their own.
Once hundreds of those AI agents were let loose, they autonomously opened networks of fake accounts on platforms such as Instagram, Facebook, X and TikTok, the officials and security researchers said. The agents then rapidly seeded the fake accounts with false posts about politics and current events to sway opinions and inflame issues.
It was one of the first times that AI agents backed by Chinese models were used to handle every stage of influence campaigns, from the creation of the fake accounts to coordinating their messaging online, said the U.S. officials, who were not authorized to speak publicly on sensitive issues. In most of the cases, the safeguards that would normally stop an AI model from creating a fake account or manipulating discussions online were turned off by those operating the systems, they added.
The agentic campaigns were crude, but they caught the attention of U.S. intelligence, major tech companies and security experts, the people with knowledge of the campaigns said. These groups then worked to detect who was behind the efforts and their methods, finding that the Chinese and the Iranian campaigns were state-backed, while two Israeli campaigns were traced to private companies.
The campaigns have stoked fears over how publicly available AI models can be used to carry out online influence efforts essentially on their own. Some U.S. officials said they worried similar moves could be made to target U.S. voters before the midterm elections in November.
To varying degrees, Russia, Iran and China have all targeted American voters with online influence campaigns during past election cycles, and each has used at least rudimentary AI. Some U.S. officials and lawmakers have worried that recent AI advances could make election influence campaigns in the United States and globally more frequent and easier to orchestrate.
In published reports this year, TikTok and Meta, which owns Facebook and Instagram, said AI was being used to create accounts and content on their platforms. In one August report, Meta said it had seen a "technically significant development" of bad actors experimenting with AI. The company identified Iranian and Israeli campaigns that used AI, without providing details.
AI was now "embedded within automated systems that can produce, adapt, and distribute content with minimal human intervention," Meta said in its report.
AI has long been used to create misleading videos and text and to generate convincing photos. But the agentic campaigns stood out, security researchers said.
Agents made by companies like OpenAI have been misused before, including by Iran, as early as August 2024, but open AI models allow for a new level of autonomy, said Kyle Chan, a researcher at Brookings. Unlike closed AI models from companies like OpenAI and Anthropic, where the underlying code is not disclosed, open AI models can be run on private hardware, meaning no company can intervene to shut the systems down.
The state-backed Iranian campaign was especially concerning, the officials and security researchers said. Iran targeted U.S. audiences, with the agent-generated fake accounts posing as everyday Americans who lived in major cities. The accounts tagged journalists and politicians in comments that spread popular memes and anti-Republican Party views, the people said. Nearly 80,000 people followed the AI-created accounts, which were active in the first half of the year.
Of the two agentic campaigns that were traced back to companies in Israel, one was generated over the summer by IntelEye, a Tel Aviv-based firm founded in 2023 by alumni of the NSO Group, a spyware firm that was blacklisted by the United States, according to the two people with knowledge of the campaign.
A U.S. security official briefed on IntelEye's campaign said the company's activities came to the attention of social media companies and U.S. intelligence earlier this year. The campaign used AI agents that created thousands of accounts on Facebook, Instagram, TikTok and X; the accounts responded to posts and comments discussing next month's national elections in Israel. The accounts also posted content that both supported and opposed Israeli Prime Minister Benjamin Netanyahu, the official said.
