These startups are fighting deepfakes by making deepfakes

I wasn’t sure if my parents would notice that the voice on the other end wasn’t mine – or that it was kind of my voice, but it wasn’t. I. The voice said hello, asked my father how he was doing, and asked again when he didn’t respond quickly enough. “What is this, Gabe?” He realized something was wrong almost immediately. I explained to him that I tried to trick him and it obviously didn’t work. “It didn’t happen,” he said. “He looked like a robot.”

It was not an ideal experience. My parents were out of the country, which led to a shoddy connection. They were having lunch with friends, and the audio couldn’t handle cross-talk or audio delays—it tried to fill in the silences. Most importantly, the voice sounded human, but it didn’t sound like me.

The audio was created by deepfake detection company Reality Defender. The problem of manipulated media is not new, but the emergence of consumer-grade AI tools has made creating fake audio, video, and images essentially easy, and a number of companies have sprung up in recent years to combat it. Reality Defender, Pindrop, and GetReal are part of the fast-growing deepfake detection industry Its value is estimated at approximately $5.5 billion As of 2023. These startups use machine learning to identify manipulated media. To fight deepfakes, you need to be able to create them.

The term “deepfakes” refers to a specific type of manipulated media created through “deep” learning, but regardless of how they are made, there is no single common denominator that unites all deepfakes. They have been used for scams, harassment, and memes. Tools like Grok AI have led to… The spread of non-consensual sexual deepfakesincluding child sexual abuse material. Scammers have Reproducing people’s voicescalled their relatives, and heard a voice saying that they were being held for ransom. During the 2024 elections, a political strategist and magician cooperated To create a deepfake of former President Joe Biden, which they used to discourage registered Democrats in New Hampshire from voting in the state’s primaries. Chairman of the Senate Foreign Relations Committee Received a Zoom call From someone using artificial intelligence to pretend to be a Ukrainian official. At the corporate level, Deep fraud It is now “industrialized”, according to one study.

The deepfake detection industry exists primarily to address one of these issues: the issue of corporate fraud.

Reality Defender effectively trains AI to combat AI. The company uses a “heuristics-based model” to detect deepfakes, CTO Alex Leslie told me. “Our foundational model uses what’s called the student/teacher model. We take a bunch of real things and say: ‘This is real,’ and then a bunch of fake things and say: ‘This is fake.'”

For the fake “me,” we spent some time fine-tuning the voice: manipulating consistency, stability, and tone to make it sound more like the real me. We can only do so much. There’s not a lot of publicly available footage of me speaking in Spanish — the language I use to communicate with my parents — except for one podcast interview from 2021, and most of it is unusable due to background music. But with nine seconds of audio and data culled from years of posts, we were able to put together a fairly convincing AI agent that was able to carry on a conversation with my father, albeit an impersonal one. The English model we used with my brother was better, because we had more training data, but even then it was not convincing enough.

But the family is the most difficult test.

“They know what you sound like,” Scott Steinhardt, head of communications at Reality Defender, told me. Steinhardt created the deepfake with my permission and modified it until it looked a bit like me. That might not fool my family, but it’s probably good enough for, say, colleagues or corporate entities like banks.

To be effective, these tools must work quickly. Generative AI is rather slow. The model we used to call my father sacrificed quality for speed. For the audio to respond quickly, we had to accept lower quality all around. The text-to-speech feature was much better, but took longer to create. When we heard the voice we read Lucky’s monologue Waiting for GodotIt looked just like me.

“As a person, it’s very difficult not to deepfake,” Nicholas Holland, Pindrop’s chief product officer, told me. “I think the challenge is, ‘How do I protect my personal identity?’ That’s something the world hasn’t figured out yet. I think, ‘How do my organizations know it’s me?’ Different organizations implement different layers of security.

It’s also a matter of resources. I don’t have the money to hire a deepfake detection company to screen my calls, but my bank does — and my bank has a lot to lose, in absolute if not relative terms. One survey 2024 It found that companies lost $450,000 per deepfake incident, with more than one company losing over $1 million in a single fraudulent transaction.

Some of these cases involved fraudsters who posed as executives, called their subordinates, and asked them to transfer large sums of money into their accounts. Before I logged into the call with Holland, I received a pop-up notification on Zoom:

This meeting is analyzed. Pindrop Security and third party providers record audio and video of your meeting to determine if you are a real person and/or the right person. By clicking “Consent” below, you consent to Pindrop collecting, using and storing your meeting, audio, voice and facial scans (which may be considered biometric information), and your IP address (to further identify your state, province, or country) for the purposes described above.

They assured me that my face, voice and IP address would be kept for no more than 90 days.

Holland told me that companies are now inundated with fake job applicants – ironically, this has happened even at Pindrop. “We see a wide range of that,” Holland said. “We see where people actually do their work, maybe they work in IT.” “We’ve had clients who hired someone, but then that person made referrals. They hired two more people, and it turned out to be the same person hired three times using three different voices, three different faces, and three different Slack identities.”

Typically, these are not entirely AI-generated video characters; They’re people who use deepfakes to change their features, almost like a digital mask. There was a trick to discover this: asking a person to place three fingers in front of their face.

“That doesn’t work at all now. The AI ​​models are so good that they can absolutely create hands, and you can put your hands in front of your face,” Holland said. “It’s basically imperceptible to your eyes now.”

Reality Defender’s Liesl told me that as technology improves, attacks become less effortful. While scammers used to impersonate a single executive, they are now targeting employees at all levels of the company. He told me about a recent attack at a publicly traded company that he declined to name, where the scammer went to LinkedIn, pulled the name of every current employee, and then scraped TikTok and Facebook to create a “pool of information” and get a voiceprint for each of those people. Their information and voiceprints were put into LLM’s software, which built a contextual window and map, and then “scattered the entire company” by calling up employees at all levels.

“In cybersecurity, we talk about these things called ‘confidence boundaries,’” Liesl said. “The problem with deepfakes is that there are always implicit trust boundaries, which is that seeing and hearing is believing. We’ve spent more than the last 40,000 years believing our ears and eyes, but now we can’t. “There are all these boundaries of trust that we didn’t have to think about before that hackers take advantage of in interesting ways.”

Currently, this program is only aimed at larger companies – they have the need, the high risk and the money to pay for it. But ordinary people don’t have deepfake detection software, and won’t have it in the near future. As Holland explains, the biggest challenge to mass adoption is awareness, as “many consumers aren’t aware of the threat, so they don’t know how to look for a solution — ground zero is with the companies that serve the consumer.” Pindrop doesn’t have a consumer product yet, but it hasn’t ruled out developing one in the future. The challenge, Holland said, is “to make these systems fast, accurate and trustworthy enough for people to rely on in everyday moments.”

Reality Defender has a different perspective. A consumer product would create an “unequal and choppy playing field for people,” Steinhardt said.

“Think of it like an antivirus: Whereas this used to be something people worried about (or, worse, didn’t), now our browsers, email providers, ISPs, and the like scan files before they get to our computer for malware,” Steinhardt said. “This is our approach to detecting deepfakes.”

My deepfake wasn’t able to fool my family, but I didn’t really put it to the test. For years, law enforcement agencies across the country have warned of a deep-dive kidnapping scam: A parent will receive a call from a very convincing voice pleading for help, and then the “kidnapper” will demand a ransom. Even if the sound isn’t completely convincing, the crying and screaming are convincing. I couldn’t bring myself to do that to my parents, even if it was fake. I briefly considered other scams: I could have contacted my bank, or perhaps my health insurance company, but the idea of ​​locking myself out of my own accounts — or committing actual, legitimate fraud — made me feel frustrated with the experience. Instead, I called my brother. “Oh, no,” he said as soon as the voice greeted him. And he wasn’t deceived either.