The greatest risk from a compromised AI-enabled account is how quickly an AI assistant can help attackers uncover sensitive information, identify targets, craft convincing communications, and advance an attack using access the victim already possesses. In a new controlled proof-of-concept attack, Barracuda‘s red team demonstrates how a single compromised employee account can escalate into CEO compromise and wire-transfer fraud to net attackers $250K using wholly legitimate, existing processes. The controlled attack leveraged Copilot, but it applies equally to other widely available AI assistants.
The attack unfolds using a compromised employee account, the attackers first ask the AI assistant to help establish persistence by creating inbox rules that hide sign-in alerts and other suspicious notifications from the user.
They then use the assistant to uncover the organization’s structure, identify high-value targets and surface relevant conversations buried within months of emails, attachments and calendar activity.
Armed with this intelligence, the attackers prompt the AI assistant to draft a highly convincing phishing message to the CEO in the employee’s natural writing style. Because the email originates from a legitimate internal account and reflects genuine business context, it is far more likely to succeed.
Once the CEO’s account is compromised through a session-token theft attack, the threat actors repeat the process, using AI to maintain persistence and uncover the most valuable financial information within the executive’s mailbox.
It is a matter of concern that AI can transform an inbox into a searchable intelligence repository. In the proof of concept, a simple prompt asking for recent financial activity unearths active invoices, wire transfers and approval workflows, including a pending $247,500 payment. The attackers then use the CEO’s account and the AI assistant to draft a convincing request to finance staff to change the destination bank account before the transfer is approved.
Because the request comes from the CEO’s legitimate mailbox, references a real transaction and matches the executive’s communication style, traditional email security controls have little reason to flag it as suspicious.
Attackers also create forwarding rules to intercept confirmation messages and use the AI assistant to quickly locate and remove evidence of the fraud.
“A user’s email history is full of sensitive information and context that can be leveraged by attackers, including emails sent and received, documents shared, attachments, and calendar invites. Company structure can be deduced from implied relationships in messages or directly viewed via organizational charts,” said Mr. Daniel Avulov, Senior Cybersecurity Researcher, red team at Barracuda. “The controlled attack shows how AI assistants can become unwitting malicious insiders and improve both the quality and speed of an attack. The most effective defensive approach is to recognize that attack patterns remain the same, take advantage of the telemetry that already exists, and ensure mean time to detect is as low as possible.”
As AI assistants become deeply embedded in business communications and workflows, organizations must treat AI-enabled accounts as high-value assets. Protecting identities, monitoring account compromise and securing AI-assisted access to corporate information will become an increasingly critical part of modern email and identity security strategies.
If you have an interesting Article / Report/case study to share, please get in touch with us at editors@roymediative.com roy@roymediative.com, 9811346846/9625243429
