Have we finally solved social engineering? Plus: World Cup fraud, AI IDs and an IBM/OpenAI collab
This podcast episode explores the intersection of AI and cybersecurity, discussing whether AI-native operating systems can curb social engineering, the impact of AI agents on identity and security, and the ongoing threat of cybercrime exploiting major events. It also highlights IBM's new partnership with OpenAI to enhance application security using advanced AI models.
The Future of Social Engineering with AI-Native Operating Systems
The discussion begins with a provocative idea: Could AI-native operating systems (OS) finally curb social engineering? Arun Vishuinath’s op-ed in Dark Reading posits that integrating Large Language Models (LLMs) into operating systems could provide a robust technical defense. An OS-integrated LLM, with access to all device activities across platforms, could interpret vast amounts of data quickly to flag social engineering attacks, much like early 2000s endpoint protection curbed viruses.
Dave Bales, an early adopter of iOS 17 with LLM-based Siri, expresses optimism. He believes that as AI learns user patterns and habits, it could significantly reduce social engineering risks, potentially enabling authorization based on behavioral signals rather than traditional passwords.
Kimmy Farrington offers a more cautious perspective. While acknowledging LLMs learn from humans, she highlights their vulnerability to prompt injection attacks, making them susceptible to social engineering. However, she sees potential in an OS built with multiple AI agents that question exchanges, viewing it as a step in the right direction, though not a complete solution.
JR Rao emphasizes human fallibility in security decisions, often lacking context, suffering cognitive overload, and being poor at spotting phishing. He argues that AI can provide the necessary context, counter the automation of attacks, and ideally, prevent humans from ever seeing phishing emails. He likens AI-native OS capabilities to how spam filters reduced spam. Rao suggests that while AI won’t eliminate social engineering, it will “shift the battlefield,” transforming it into an “interface problem” where attackers target AI assistants, trust models, and agent instructions. This shift presents a new challenge: how to deal with AI’s inherent vulnerabilities after spending decades addressing human frailties.
The panel discusses behavioral authentication, noting that while traditional methods like IP addresses and typing speed are easily mimicked, an LLM integrating diverse behavioral factors could create an unstealable authentication pattern. However, the tension remains: AI can cover human failings but is also a potential attack vector itself. The consensus is that a “symbiotic relationship” with human oversight (“human in the loop or on the loop”) and continuous feedback will be crucial to leverage AI’s strengths without succumbing to its weaknesses.
World Cup Fraud: A Global Attack Surface
Cyble Research and Intelligence Labs’ “Operation FanTrap” uncovered nearly 4,000 malicious domains exploiting the World Cup, using fraudulent branding to spread malware, steal credentials, and drain wallets. Kimmy Farrington is unsurprised, highlighting the “depth and breadth of the industry” around such events, encompassing fake streaming sites, merchandise, and tickets.
JR Rao aptly terms the World Cup a “global attack surface.” He explains that cybercriminals leverage the intense global attention, urgency, and heightened emotions surrounding the event. This leads to a drop in vigilance, which attackers exploit with “industrialized” fraud, weaponizing marketing tactics like fake discounts, free streaming offers, and travel packages. He stresses that attackers often understand human search behavior better than defenders.
Regarding defense strategies:
- Technical Controls (Kimmy): Advocates for browser-level EDR-like sensors that inspect browser activity, send telemetry to SIMs, and correlate with EDRs. This allows for faster action against malicious JavaScript or suspicious call-homes, acknowledging that “users are going to be frail” but “automation and agents are very strong” and emotionless.
- Non-Technical Wisdom (JR & Dave): JR advises skepticism: “If the deal looks better than the official offer, you just want to assume it’s a scam until it’s proven otherwise.” He emphasizes buying from known sources and being wary of urgent sales pitches, echoing the age-old wisdom of “no free lunch.” Dave reinforces the “stop clicking things” mantra, urging individuals to research offers and not be swayed by deals that “sound too good to be true.”
IDs for AI Agents: Accountability and New Risks
Estonia is considering granting personal ID codes to AI agents, similar to human IDs, to enhance accountability and permission scoping. The idea is that agents would have distinct identities and privileges, making their actions traceable and auditable.
JR Rao views this as an “inevitable” step, asserting that powerful agents require identity, accountability, and authorization. He highlights the challenge to existing Identity and Access Management (IAM) systems, which are not designed for the scale of potentially millions of AI agents (e.g., Jensen Huang’s vision of 5 million agents at Nvidia). He also points out that an agent’s identity might be more complex than a user ID, involving provenance, owner, purpose, reputation, and permissions. Rao warns that “an authenticated malicious agent can be far more damaging than an unauthenticated one,” referring to AI agents as potentially “the most helpful insider threats we’ve ever had.”
Dave Bales raises concerns about AI agents themselves being compromised, questioning if giving them an ID makes them any safer or simply creates new attack vectors, similar to human accounts. He fears agents could “fall into the same traps that humans do.”
Kimmy Farrington questions the scalability of granting identities to millions of agents, especially given their ephemeral nature in some cases versus long-lived instances. She wonders how to manage AI agent “passwords” or tokens and suggests a Kerberos-like token system for short-term, specific authorizations to prevent token theft and reuse.
IBM Joins OpenAI Daybreak for Application Security
Jesh Kamat, Offering Leader for Application Security at IBM, discusses IBM’s new partnership with OpenAI as part of the Daybreak Cyber Partner Program. This collaboration aims to help clients secure their application estates against “frontier AI threats.”
The core of the partnership is the “Security Harness” service. This service controls OpenAI’s frontier AI models to ensure they operate within enterprise requirements. It scans application code for vulnerabilities, including complex “chained vulnerabilities,” and can even prove exploitability safely within a controlled enterprise context, complete with logs, audits, and traceability.
Kamat emphasizes that this partnership offers IBM early access to OpenAI’s models, allowing them to build and refine these security harnesses. This positions IBM to provide secure and compliant application security solutions as AI models become more pervasive. He expresses significant excitement, noting this collaboration is a “huge leap,” allowing for the scanning and protection of vast amounts of code that previously “escaped our view and visibility because we’ve not been able to scale.” The partnership combines IBM’s consulting expertise and approach with OpenAI’s intelligence to tackle long-standing application security challenges.