AI-powered social engineering has changed what a convincing request looks like, and the old advice for spotting one no longer holds up. In January 2024, a finance employee at the Hong Kong office of Arup, the British engineering firm behind the Sydney Opera House, received an email that seemed to come from the company’s UK office. It asked for a confidential transaction. He had doubts, which is exactly the instinct we try to build in people, so he joined a video call to confirm the request with the chief financial officer and several colleagues he recognized.
Every face and every voice on that call was a deepfake built from publicly available footage. Think about how ordinary that moment must have felt. He was looking at colleagues he recognized and listening to voices he knew, in exactly the kind of meeting we tell people to ask for when a request feels off. By the time it ended, he had sent roughly $25.6 million across 15 transfers to accounts controlled by the attackers.
I keep coming back to this story because of what it says about the advice most of us, myself included, have given for years. We told employees to look for the misspelled word or the sender address that is one character off. When something feels wrong, we told them to get the person on the phone or on a call and confirm. By the standard we were teaching a few years ago, that employee did nearly everything right. The confirmation itself was the attack. The habit we worked so hard to teach had been turned into the way in.
So what do we tell our teams when the thing we taught them to trust is the thing being faked?
Key takeaways
- Deepfakes can fake the video call or phone call your team uses to verify a request, so verification cannot depend on recognizing a face or voice.
- Business email compromise and AI-enabled fraud are growing, with billions in reported losses.
- The strongest defense is process control: fixed rules for how money moves and how access is granted.
- Fast, blame-free reporting is a financial control, because quick reports can help recover fraudulent wires.
Why AI-Powered Social Engineering Is a Different Problem
Cybersecurity Awareness Month is a good time to sit with that question. This October, CISA is running under the theme “Securing the Next 250,” and the National Cybersecurity Alliance chose “Don’t Make It Easy for Them.” I find the second one especially useful. It moves the conversation away from catching every fake and toward making the attacker’s job harder, no matter how convincing the fake is. That is a more honest goal for a security team, because it does not depend on anyone winning a guessing game under pressure.
The numbers suggest that shift is overdue:
- The FBI’s Internet Crime Complaint Center (IC3) logged 24,768 business email compromise complaints in 2025, with roughly $3.05 billion in reported losses, up from 21,442 complaints the year before.
- For the first time, IC3 tracked AI-related complaints as their own category: about 22,000 complaints and $893 million in losses.
- IBM’s 2026 Cost of a Data Breach Report found that one in four malicious breaches now involves AI, mostly deepfake impersonation and AI-generated malware. Those breaches averaged around $6 million, about a million more than the global average of $4.99 million.
Taken together, these figures describe a problem that is growing in volume and in cost, and that now carries enough weight for the FBI to count AI-related complaints separately. Whatever our instincts tell us about how obvious a fake should be, the reported losses say that plenty of them are working.
Why Security Awareness Training Alone Can’t Stop Deepfake Fraud
None of this means awareness training has failed, and I want to be careful here, because I have seen how quickly a statistic like that turns into blame. The employee in Hong Kong was not careless. He was working inside a process that allowed a video call to stand in for verification, and that is a design problem far more than a people problem.
When AI can write in the cadence of a CFO’s emails and then put the same CFO’s face on a screen, asking people to spot the fake means asking them to win a contest the attacker has already rigged. Is it fair to make the last line of defense the person with the least information and the most pressure in the moment? Detection puts the whole burden on one person at one moment, and it is the moment when the attacker has done the most work to make everything look right.
Control Beats Prediction
In my doctoral research, I spent a good deal of time on the distinction between prediction and control, drawing on Wiltbank and colleagues in the Strategic Management Journal. They describe how organizations either try to forecast what is coming or focus on shaping what they can actually influence.
Most security awareness programs are prediction exercises at heart. They ask an employee to forecast whether a given message is real. A forecast can be wrong, and when the forecast is the only thing standing between a request and a transfer of money, a wrong forecast is expensive. The more durable approach is control: deciding ahead of time how money moves and how access is granted, so the authenticity of any single email or call matters far less.
If a change to a vendor’s banking details always requires a callback to a number already on file, and a wire above a set amount always needs a second approver who never saw the original request, then even a flawless deepfake hits a wall it cannot talk its way past. The rule does not care how real the request looks. It asks the same questions of every request, whoever appears to be making it, and that sameness is where its strength comes from.
How to Defend Your Business Against AI-Powered Social Engineering
The practical side is less dramatic than the threat, which I suspect is part of why it gets skipped.
- Name the transactions that need more than one message. Write down the small handful of actions that should never be approved on the strength of a single request, such as payment detail changes, large wires, and credential resets. The shorter and more specific the list, the more likely people are to remember it and follow it.
- Agree on an out-of-band check. Use a phone number already on file or a code word that never travels by email. The check only works if it lives outside the channel the request arrived in, because anything that travels with the request can be faked along with it.
- Give people explicit permission to slow down. That includes telling an executive who seems to be in a hurry that the request has to go through the normal path. Urgency is the oldest lever in social engineering, and AI has not changed that. A rule nobody feels able to use is not a control, so this permission has to be given clearly and out loud.
- Add the technical layers. Phishing-resistant multi-factor authentication makes stolen credentials far less useful, and a properly enforced DMARC policy makes it much harder to send mail that looks like it came from your own domain.
Both technical controls are worth the effort. I would still put the process work first, because it holds up even on the day a technical control gets bypassed.
What to Do After an Employee Gets Fooled
Think about what happens after someone gets fooled, because sooner or later it happens, even in well-run organizations. The FBI has had real success freezing fraudulent wires when they are reported to the bank and to IC3 quickly. An employee who fears blame will hesitate, and that hesitation can be the difference between recovering the money and writing it off. A culture where people report fast and without fear is as much a financial control as any approval workflow.
Where Technology Meets People
This is the part of the work I enjoy most at XDuce, because it sits right where technology and people meet. Some of the most valuable conversations I have with clients have little to do with firewalls. They are about how a finance team verifies a payment change, and the moment a controller realizes their approval process has a gap nobody had noticed. We hold ourselves to the same discipline, and going through our own SOC 2 Type II audits has made me appreciate how much of good security comes down to documented process that people actually follow.
The Lesson Inside the Arup Story
The Arup story is unsettling, but I think the lesson inside it is a hopeful one. The goal was never to turn every employee into someone who can detect a synthetic face on a video call, and thankfully it does not have to be. What protects an organization is an unusual request getting the same answer every time, regardless of who appears to be asking.
Seeing may no longer be believing, and that is all right, as long as verifying is something we decided on before the call ever came in. If you use this month to change one thing, make it the rule for how money leaves your organization.
If you would like a second set of eyes on your own payment and access controls, our Managed IT & Cybersecurity team can start with a security assessment.
Frequently Asked Questions
AI-powered social engineering is the use of artificial intelligence to impersonate trusted people and trick employees into sending money or granting access. Attackers use tools such as voice cloning, deepfake video, and AI-written messages to copy how a real executive writes, speaks, and looks. The Arup case shows how far this can go: every face and voice on the verification call was a deepfake, and the employee sent roughly $25.6 million across 15 transfers.
Not on its own. A video or phone call can no longer prove who is asking, because deepfakes can reproduce a real executive’s face and voice on a live call. For years, employees were taught to get the person on the phone or on a call and confirm. Verification now needs an out-of-band check, such as a callback to a number already on file or a code word that never travels by email, so that confirmation does not depend on recognizing a face or voice.
The best defense is process control: deciding ahead of time how money moves and how access is granted, so no single email or call can authorize anything alone. In practice, that means naming the transactions that need more than one message, such as payment detail changes, large wires, and credential resets, and requiring a callback to a number already on file and a second approver who never saw the original request. Phishing-resistant multi-factor authentication and an enforced DMARC policy add technical layers, but process comes first because it holds up even when a technical control gets bypassed.
Yes, but it cannot be the only defense. Training asks employees to predict whether a message is real, and when AI can copy a CFO’s writing and face, that becomes a contest the attacker has already rigged. Training works best alongside clear rules that remove the guesswork, plus explicit permission to slow down, including telling an executive who seems to be in a hurry that the request has to go through the normal path. Urgency is the oldest lever in social engineering, and AI has not changed that.
Report it immediately to your bank and to the FBI’s Internet Crime Complaint Center (IC3). The FBI has had real success freezing fraudulent wires when they are reported quickly, so speed matters. An employee who fears blame will hesitate, and that hesitation can be the difference between recovering the money and writing it off. A culture where people report fast and without fear is as much a financial control as any approval workflow.
Related Blogs
