Part 2: Social engineering, malware, and the future of cybersecurity in AI

The landscape of cybersecurity is not just a technical challenge; it’s deeply rooted in human factors. This includes understanding the adversaries, their tactics for manipulating people, and how the rise of AI is reshaping this dynamic.

The Spectrum of Bad Actors

Adversaries in cybersecurity can be broadly categorized, each with distinct motivations:

• Nation-State Actors: These groups are typically focused on geopolitical objectives, such as espionage or offensive cyber operations in support of ground warfare. A significant concern is “prepositioning,” where actors infiltrate critical infrastructure like power grids, not for immediate attack, but to maintain access and control in case a “hot war” begins. They periodically check in to ensure their access remains viable.
• Sub-Nation State and Financially Motivated Actors: This category includes independent groups, sometimes loosely affiliated with nation-states, whose primary goal is financial gain. Ransomware is their most prominent tool. The modern ransomware attack involves a slow, quiet compromise of a company’s network, culminating in the encryption of both primary and backup data, effectively taking the entire business offline until a ransom is paid in cryptocurrency. Some nation-states also engage in this activity as a means of generating funds.

The Vulnerability Marketplace and Project Zero

Zero-day vulnerabilities are highly valuable, creating a gray market where they are bought and sold. Buyers include governments and law enforcement agencies, as well as malicious actors. The value of some exploits can run into millions of dollars.

AI has the potential to disrupt this marketplace by making the discovery of these vulnerabilities cheaper and more accessible, which could arm more bad actors. To counter this, initiatives like Google’s Project Zero are crucial.

Project Zero is really an absolute world-class effort, combining I think some of the most elite hackers that have ever lived that their job day in and day out was to find novel vulnerabilities in systems.

A key innovation from Project Zero was the 90-day disclosure timeline. When a vulnerability is reported to a company, they have 90 days to patch it before Project Zero makes the details public. This policy fundamentally changed industry incentives, forcing companies to prioritize security patches to avoid public disclosure and widespread exploitation.

The Human Element in Defense

Even when technical solutions exist, human and organizational factors can be the weakest link. The WannaCry ransomware attack, which heavily impacted the UK’s National Health Service (NHS), is a prime example.

A patch for the “Eternal Blue” vulnerability used by WannaCry was available from Microsoft before the attack. However, critical environments like hospitals are often “institutionally risk-averse to change.” Applying patches to medical equipment running standard operating systems can be risky, potentially breaking life-saving systems. This creates a paradox:

…the systems that are the most critical then have a side consequence of having the least acceptance of change and that lack of acceptance of change means that you have the least ability to affect security improvements into the environment.

This short-term prioritization of operational stability leads to long-term security vulnerabilities in critical infrastructure.

AI-Powered Social Engineering and Modern Defenses

AI significantly enhances social engineering attacks, making them more personalized and believable.

• Deepfakes: AI can clone a person’s voice and likeness for live video calls. This has been used in attacks where a fake CFO instructs an employee to make an urgent wire transfer, or where a cloned daughter’s voice is used in a fake ransom call.
• Advanced Phishing: AI allows for the mass creation of highly tailored “spear phishing” attacks, and even “whaling” attacks targeting high-profile executives.

Despite these advanced threats, established security best practices remain effective. Strong defenses include:

• Moving Beyond Passwords: The industry is transitioning to a post-password era with technologies like Passkeys. Passkeys allow users to log in by scanning a QR code with their phone, leveraging the device’s existing security. It’s both easier and more secure than traditional passwords.
• Risk-Based Authentication: This system calculates a real-time “trust score” for a user’s session based on a wide array of behavioral signals (e.g., mouse movement, typing cadence, IP address). If the system detects anomalous behavior, it can “step up” the security friction, requiring an additional factor of authentication. AI is continuously improving the sophistication of these behavioral models.

The Next Frontier: Autonomous Agents

The move towards an era where autonomous AI agents act on our behalf introduces a new paradigm of security and privacy challenges.

• Identity and Trust: The internet’s foundational “plumbing” is not yet built to distinguish between a human, a simple bot, and an agent acting on a human’s behalf. Establishing an agent’s identity, providence, and the level of trust it deserves is a fundamental challenge.
• Security: The risk escalates when an agent can both process untrusted input (like an email or website) and take action in the real world (like controlling a smart stove). Security for agents requires carefully defined constraints on the tools they can use and the actions they can perform.
• Privacy and Contextual Integrity: This is a profound challenge. An agent must understand nuanced social and privacy norms. For example, it should know to share a social security number with a tax agency but not with social media contacts. This is the problem of “contextual integrity.” The initial step towards solving this is teaching agents to recognize when a situation is sensitive and to ask their human user for permission before proceeding.

While the challenges are significant, there is a strong sense of global cooperation among major AI labs to work together on these safety and security problems, aiming to defend the future of AI for everyone.

• Tags:
• Autonomous Agents
• Cybersecurity
• Social Engineering
• Vulnerability Management
• Passkeys
• Risk-Based Authentication
• Project Zero