AI Voice Cloning Exposes Critical Security Vulnerabilities

Our team recently conducted a red team exercise using AI voice cloning as an attack vector, revealing critical vulnerabilities in a client’s security protocols and demonstrating how sophisticated social engineering attacks have become. The results and recommendations are relevant to all firms in financial services. 

The Red Team Exercise 

A client was concerned about emerging threats, particularly AI voice cloning. They wanted to understand whether and how this new technology could exploit their teams and systems—and whether they had the defenses in place to protect themselves. Given the rise in AI-powered scam calls targeting individuals and companies, this forward-thinking approach was well-warranted. 

We designed a custom red team exercise that simulated how sophisticated attackers might leverage publicly available information and AI voice synthesis to target a firm. The multi-phase approach tested executive-level vigilance and the firm’s operational security procedures. 

Gathering Intelligence  

Our team began with open-source intelligence (OSINT) gathering on key executives. We discovered the firm had extensive information publicly available—from executive profiles on their website (including direct phone numbers and email addresses) to videos on their YouTube channel featuring key personnel. 

We identified a director who regularly appeared in the company’s YouTube videos and used those publicly available recordings to gather sufficient voice samples to create a convincing AI clone of her voice using commercially available tools. 

Conducting an Initial Voice Cloning Test 

Using the cloned voice and a specially prepared soundboard of pre-generated clips, our team contacted the firm’s CFO and presented a plausible business scenario about developing metrics to evaluate the financial benefits of their social programs. 

The CFO engaged in conversation for over two minutes—a significant achievement for the test. While he ultimately remained skeptical and didn’t fall for our subsequent phishing attempt, the interaction gave us enough voice samples of the CFO himself to create another voice clone. 

Exploiting the Critical Vulnerability 

For the final phase, we created a voice clone of the CFO and contacted the company’s help desk during off-hours, specifically on a holiday weekend when all calls were being forwarded to cell phones. The results were alarming. The help desk support agent: 

• Immediately recognized what they believed to be the CFO’s voice 

• Failed to follow proper verification protocols 

• Changed the CFO’s password 

• Texted the new credentials to an unfamiliar number 

This successful compromise demonstrated significant vulnerabilities in the firm’s verification procedures and help desk protocols, particularly for high-privilege accounts. 

We provided a complete report of our findings, including detailed methodology and evidence, results, recommendations, and clean-up practices, via our client portal, AgioNow. The portal is a command center that delivers all real-time technology insights and intelligence related to your environment from a single location for easy access and complete transparency.   

You can see how it works here:  

Understanding Voice Cloning Limitations 

While our test proved successful, it’s worth noting that current AI voice cloning technology currently has limitations that security-conscious firms can watch for. For example, it doesn’t yet allow for real-time conversations. Instead, we opted to generate specific voice clips from a sample source, build out a full scenario in advance (making sure to have enough filler words like um, uh, etc.), and play the clips at appropriate moments. 

Voice cloning also struggles with certain aspects of natural interactions—the subtle give-and-take of human dialogue is lost on AI. For example, voice cloning doesn’t handle interruptions well and can struggle with natural pauses and transitions.  

These limitations provide potential warning signs for those who know what to listen for, but they’re easily missed in high-pressure support scenarios or when dealing with perceived authority figures. 

Implementing Protective Measures 

Even when recognizing a voice—especially of senior leadership—it’s important to follow established verification protocols. In other words, trust but verify. Based on this red team exercise, we recommend several protective measures for financial services firms: 

• Implement Passphrase-Based Verification: Require all callers to provide a unique, pre-established passphrase before processing sensitive requests, ensuring only verified users can proceed. 
• Require call-back verification. For unusual or sensitive requests, hang up and call the person back at their official number on file. A bad actor can spoof the phone number of someone like the CFO, but if you actually hang up and call the number on file, the call will go to his real phone because hackers can’t typically intercept calls at that level. 
• Education Help Desk Staff: Use simulations and real-world scenarios to train employees on identifying and mitigating social engineering attempts, including AI-driven vishing. 
• Develop escalation paths. Implement protocols for handling unusual requests from executives, especially those involving security changes or financial transactions. 
• Reduce your digital footprint. Regularly audit and limit unnecessary public exposure of executive information, including office phone numbers and email addresses on your website. While maintaining a professional presence is important, oversharing details creates unnecessary risk. 
• Enforce in-person verification when necessary. For scenarios where a user claims to have lost access to their phone or MFA device, require in-person verification rather than bypassing standard security protocols. 

It’s important to foster a security culture where verification is normalized and not seen as questioning authority. Even the most security-conscious executives can become targets and help desk staff need to feel empowered to follow security protocols regardless of who’s calling. 

Addressing the Human Element 

This red team exercise demonstrated that social engineering attacks ultimately target human psychology. The help desk employee who fell for our ruse wasn’t lacking technical skills; they were influenced by recognition of authority (the CFO’s voice), the pressure of an emergency scenario, and possibly the contextual pressure of handling a work call during family time. 

Regular security awareness training that specifically addresses these psychological factors is critical for building resilience against these sophisticated attacks. 

Securing Your Firm’s Future 

Voice cloning represents just one way artificial intelligence is transforming security challenges for financial services firms. The most effective defense combines robust technical protections, well-trained staff, and regularly tested procedures. By proactively identifying and addressing vulnerabilities before malicious actors can exploit them, your firm can protect both its operations and its reputation. 

For firms handling sensitive financial information and client data, these proactive security measures aren’t just good practice—they’re essential to maintaining the trust that forms the foundation of your business. 

If you want to see how your firm holds up against AI voice cloning or have a deeper conversation around cybersecurity, contact us.