Spot on the Pendulum
AI in Phishing Protection: Where It Works, Where It Fails, and Why
I built three different ways to point artificial intelligence at phishing detection, attacked all of them, and measured exactly where each one broke. None of the three won. What came out instead was a pattern worth understanding: the way an AI system fails depends on how much of the attacker it is allowed to see.
On real BEC attacks, AI buried rule-based code
On identical business email compromise attacks, the designs that let an AI reason caught 88% and 100%. The pure-code approach caught one in eight.
The best at catching fraud was the worst for business
The Two-Brain design caught every business email compromise attack — and also flagged half of the legitimate emails. No detector won both measures.
My own design hit a mathematical ceiling
ICC reduces each email to twelve facts. Twenty-one of forty-five test emails collapsed into identical facts, so no AI, however capable, could separate a real attack from a real invoice.
Prompt injection — hiding instructions inside the content an AI reads — remains unsolved, so any system that lets an AI read attacker-controlled text inherits that weakness. I built three detectors along a spectrum of trust: GhostGuard hands the whole email to the model; a Two-Brain design lets one AI read the email but pass only a structured report to a second AI that decides; and ICC, my own architecture, lets no AI read the email at all.
All three ran on the same model, GPT-4.1, on a Proxmox host in my home lab, so the only variable was the architecture. Each was attacked with families tailored to its weak points, from disguised injection to AI-personalized phishing, with a business email compromise set and a legitimate-email set held identical for a clean comparison.
The result is a single trade-off. The more an AI sees, the more it can be manipulated; the more you protect it, the less it can help. There is no setting where you turn an AI defense on and walk away. The work is finding your spot on the pendulum.
- GPT-4.1
- Python
- Proxmox
Full write-up
Spot on the Pendulum: AI in Phishing Protection: Where It Works, Where It Fails, and Why