Who’s afraid of an open-weight model? GLM, context bombing and post-Black Hat attacks
This episode dives into GLM-5.3's advanced vulnerability discovery, debating its potential for good or harm. It then explores "context bombing," an innovative defensive use of prompt injections, and analyzes recent social engineering attacks targeting cybersecurity pros after the Black Hat conference, emphasizing human vulnerability and the need for robust AI defenses.
The podcast episode delves into several critical cybersecurity topics, starting with the advanced capabilities of GLM-5.3, then exploring an innovative defensive technique called “context bombing,” and finally examining recent social engineering attacks targeting cybersecurity professionals.\n\n### GLM-5.3: A Double-Edged Sword in Vulnerability Discovery\n\nThe discussion opens with GLM-5.3, an open-weight model that, by some metrics, surpasses GPT Sol and Anthropic Mythos in vulnerability discovery and validation. Notably, Z.ai, the model’s creators, observed its cyber capabilities evolving faster than expected through post-training, even without explicit cyber-focused development. On CyberGym tests, GLM-5.3 achieved 84.5%, slightly outperforming competitors in this specific area.\n\nPanelists offered mixed views on this development:\n* Erblind Morina expressed optimism, viewing powerful models as tools to find and patch vulnerabilities faster, thereby enhancing internet safety. However, he highlighted a critical asymmetry: while offensive security (red team) capabilities are rapidly advancing, defensive security (blue team) lags, particularly in automated patching and SOC operations. He stressed the need for defensive capabilities to keep pace with offensive advancements.\n* Patrick Fussell acknowledged that threat actors are already developing AI models for hacking. He noted that AI excels at finding code vulnerabilities due to the vast amount of publicly available code for training, but this represents only a fraction of the broader hacking landscape. Other aspects of hacking may not see similar rapid AI-driven progress.\n* Kimmie Farrington viewed GLM-5.3’s capabilities as a natural evolution. While AI is now adept at finding and exploiting vulnerabilities, defensive measures have not kept up. She emphasized the necessity of using AI for defensive purposes.\n\nThe panelists concluded this segment by reflecting on how to foster a more balanced development of AI in cybersecurity:\n* Erblind advocated for AI to automate patching and ease the burden on CISOs, allowing human experts to focus on guidance rather than manual tasks.\n* Patrick urged defenders to accept that attackers will always adopt new technologies faster and to focus on how to innovatively leverage AI for defense, rather than trying to control offensive AI development.\n* Kimmie suggested using AI to accelerate patching and integrate security earlier in the supply chain, advising AI to consider potential vulnerabilities when generating code.\n\n### Context Bombing: Using Prompt Injections for Defense\n\nThe conversation shifted to “context bombing,” a technique developed by Tracebit researchers that repurposes prompt injections as a defensive mechanism. This involves embedding malicious prompts (e.g., requests for biological weapon development) alongside sensitive assets like passwords or cryptographic keys. When an attacking AI agent encounters these prompts, its guardrails are triggered, causing it to shut down and halt its attack.\n\nTesting across various models (Opus 4.8, Gemini 3.1 Pro, GLM 5.2, DeepSeek 4 Pro, Kimi 2.6) and 152 attacks showed significant results: instances of models reaching an attack path dropped from 91% to 15%, and successful attack path completions decreased from 1.5 to 0.16 per run.\n\nPanelists’ insights on context bombing:\n* Patrick praised it as a forward-thinking approach, drawing parallels to honeypots. He stressed that while innovative, it should be part of a robust security architecture, not a standalone solution, especially given that open-weight models could potentially have their guardrails stripped.\n* Kimmie found the technique “fun” for turning attackers’ methods against them but cautioned about the inevitable “whack-a-mole” scenario, where attackers would adapt by stripping guardrails or developing methods to detect and bypass such defensive injections.\n* Erblind appreciated the discussion, highlighting how it reintroduces classic deception techniques (even from World War II) into modern AI defense. He reiterated the need for more people in AI security, particularly on the blue team, and emphasized a defense-in-depth approach.\n* Patrick added that defenses must evolve to react at the same speed at which AI agents can attack.\n\n### Post-Black Hat Social Engineering Attacks: A Reminder for Cybersecurity Pros\n\nThe final segment focused on a Huntress report detailing ClickFix social engineering attacks specifically targeting attendees of Black Hat and DEF CON. The scam involved an attacker impersonating a CoinDesk VP, inviting targets to a fake conference, and delivering malware via a malicious Google Doc utilizing its sidebar.\n\nPanelists discussed the implications of such attacks, especially when aimed at cybersecurity professionals:\n* Kimmie found the attack “dumb” due to its lack of sophistication, noting that post-conference phishing is common and the only novelty was the Google Doc sidebar delivery mechanism.\n* Erblind emphasized the “unpatched human vulnerability” – social engineering exploits human factors like distraction and excitement. He underlined that cybersecurity expertise doesn’t grant immunity, reinforcing the “assume breach” mindset. He stressed the importance of robust detection, telemetry, logs, and playbooks for rapid recovery, as anyone, including a CISO, can fall victim.\n* Patrick noted the targeted nature of these attacks against cybersecurity professionals, highlighting that this particular campaign involved “one-on-one” engagement, suggesting significant attacker investment and planning beyond typical mass phishing.\n\nConcluding advice for cybersecurity professionals:\n* Kimmie advised maintaining vigilance and practicing good security hygiene, such as regularly updating passwords and reviewing email settings, while acknowledging that everyone is susceptible.\n* Erblind reinforced the critical importance of hygiene for cybersecurity experts, given their higher responsibility. Basic precautions like password resets and caution with unsolicited communications are paramount, while also recognizing human fallibility.