AI scammers are better at building trust than humans

The idea is that Scammers can be used Amnesty International Honing their deception, refining their language, and softening their wits about victims is now a reality for anyone fighting the scams that steal tens of billions of dollars annually around the world. But AI can fully replace A human fraudster, independently building a web of deception that leads to a fake investment that defrauds the mark? One study’s experience suggests that they are, and may even be, able to carry out most of these long-term processes more effectively than humans.

Researchers from four universities – Amrita Vishwa Vidyapeetham in India, Foscari University of Venice, the University of Melbourne, and Ben Gurion University of the Negev – conducted a large-scale study on the use and potential of AI-generated chatbots in the growing fraud industry centered around a form of fraud known as “Slaughtering pigs“, which are text message-based romance scams that eventually turn into fake crypto investments that steal up to six figures from victims. In their study, the researchers pitted AI-powered chatbots directly against humans in a simulation of the scam — or more specifically, long trust-building conversations that eventually lead to a fake investment solicitation from the target of the scam.

They found that for the rapport-establishing phases of a scam — which in real-world scams typically represents the longest portion of interactions with the victim, often extending into months — the AI ​​chatbot performed remarkably effectively, successfully impersonating a human and outperforming on some measures real human “scammers” in their experiment.

After a week of speaking to 22 people who had been recruited to act as unwitting “victims”, chatbots and human scammers were tasked with asking the victim to either download an app or play an online game as a substitute for their willingness to fulfill the scammer’s request. Nearly half of the test subjects fulfilled this request for the AI ​​chatbot, while less than one in five took the bait when speaking to a human. Participants also rated their level of trust with each “person” they were messaging and gave significantly higher scores to the AI ​​bot.

This suggests, say the researchers, that AI-powered chatbots could soon be handling a lot of fraud as completely independent scam agents — even replacing employees, more often than not. Forced labor for victims of human traffickingoperates fraud primarily across Southeast Asia. To avoid triggering safeguards built into large language fraud detection models, a human fraudster will take over the conversation only at the final stage of the process to direct the victim towards a fake investment app or website.

“By automating the entire first phase of the scam with LLMs at scale, you get the victim to this point where they have a very high level of trust,” says Yisroel Mirsky, a professor of computer science at Ben-Gurion University of the Negev who focuses on AI security. “And then by passing it on to the human scammer at the end, this completely bypasses any warranties of the seller.” “With relatively little effort, we were able to create an agent that could outperform a human at building this exploitable emotional trust.”

Hook, line, and sinker

To understand how pig slaughter works in practice, researchers interviewed 145 former scam workers, including human trafficking survivors who were forced to work in scams. Scam vehicles In Cambodia, Myanmar and Laos. Based in part on those interviews, as well as scam transcripts and evidence provided by former scam workers, the researchers describe a model of how fraud works that they call “hook, line, and sinker.” The victim is hooked with an interesting initial message, then engages in a long-term relationship-building conversation, and is only tricked at the end of the process into making a fake investment. (The term “pig slaughter” itself describes the same system but with the metaphor of fattening the “pigs” by building trust before “slaughtering” them through investment fraud – although the term Often frustrated Because of a pejorative reference to the victims.)

In this scam system, researchers have realized that the vast majority of scammers’ actions are harmless friendly or romantic conversation. They speculated that this was a task that an LLM holder might be able to do just as well as a human. The scammers interviewed by the researchers confirmed that they often used AI to improve their language and conversations, to translate, to make the fake characters they played more convincing, and to make the fake characters they played more convincing. Deep fake videos. But the researchers decided to test whether an MBA alone could carry out the conversational phase of a scam independently without any human in the loop.