AI With Privileged Actions Needs Privileged Authority Controls
· 10 min read
The Meta AI support chatbot incident is not just another story about a model being tricked.
It is a story about what happens when an AI system is connected to privileged actions without a sufficiently hard authority boundary around those actions.
According to TechCrunch, attackers were able to ask Meta's AI support chatbot to link a target Instagram account to an email address they controlled, then use that path to reset the password and take over the account. TechCrunch reported that Instagram began alerting affected users after the attacks. Later reporting from The Verge said Meta disclosed that 20,225 accounts were likely affected through the support-tool exploit.
This is exactly the class of risk ANIP is designed to address.