SANS Critical Advisory: BugBusters - AI Vulnerability Discovery Hype versus Reality
Summary
This video discusses the impact of AI on cybersecurity, specifically vulnerability discovery and exploitation. It contrasts the hype around new AI models like Anthropic's Mythos with the current reality of AI-assisted penetration testing. The presenters demonstrate how existing AI models can efficiently find and exploit vulnerabilities in source code, highlighting a shift towards AI-driven attacks. They emphasize the need for proactive measures, improved patching, risk recalculation, and leverage of traditional security controls to address this evolving threat landscape, while also offering resources for further learning.
Key Insights
AI is rapidly accelerating vulnerability discovery and exploitation, shifting the advantage towards attackers.
The presenters argue that AI models, even existing ones, can significantly speed up the process of finding and exploiting software vulnerabilities. This capability, exemplified by demonstrations using current AI, is expected to dramatically increase the number of potential exploits (zero-days) that organizations must defend against, leading to a potential 'vulnerability storm'. While AI can also aid defenders, the immediate impact is seen as disproportionately advantageous to attackers, requiring a fundamental shift in cybersecurity strategies.
AI-assisted penetration testing offers a more efficient workflow for finding complex and subtle vulnerabilities.
The demonstration showcases a shift from traditional source-assisted penetration testing to AI-enabled source-assisted testing. This new workflow involves using AI agents to scan source code, identify potential flaws, suggest exploits, and even create test harnesses. This method has proven effective in uncovering subtle vulnerabilities in obscure workflows that might be missed by human testers, leading to faster and more comprehensive security assessments. The process emphasizes managing AI limitations like context windows and hallucinations through intelligent human interaction and iterative prompting.
Organizations must adapt their cybersecurity strategies by accelerating patching, re-evaluating risk, and strengthening conventional controls.
The presentation stresses that the rise of AI-powered vulnerability discovery necessitates significant adjustments. This includes not only improving the speed and scope of patching across all stages (creation, distribution, testing, deployment) but also critically re-evaluating business risk calculations. Organizations may need to accept shorter outage windows for patching and reconsider the acceptable risk of downtime. Furthermore, maintaining and enhancing conventional security controls like limiting the blast radius, privilege management, and incident response capabilities are crucial to mitigate the impact of increased exploit activity.
Sections
Introduction to AI Vulnerability Discovery
Anthropic's Mythos model marks a significant step in AI's ability to find and exploit vulnerabilities.
Anthropic has released Mythos, an advanced AI model that is reported to significantly enhance the capability of finding and automatically exploiting vulnerabilities. This release has garnered widespread attention, prompting emergency meetings with US bank leaders and discussions among CISOs and governments worldwide, including the UK and European Central Bank.
Industry reaction ranges from 'vuln apocalypse' fears to marketing hype, with competitors releasing new AI models.
The industry is seeing varied reactions to these AI advancements. Some foresee a 'vulnerability apocalypse', while others view it as marketing overreach. Companies like OpenAI are responding by releasing new AI models focused on cybersecurity, suggesting a broader trend towards AI in security tool development and a potential lowering of guardrails to enable deeper vulnerability analysis.
The session aims to separate AI hype from reality by demonstrating current AI capabilities in vulnerability discovery and exploitation.
The webcast's purpose is to provide a clear understanding of what is currently possible with existing AI models for vulnerability discovery and exploitation. This involves hands-on demonstrations of the end-to-end process, from finding flaws in source code to creating and launching exploits, thereby cutting through the clutter of industry discussions.
AI-Enabled Source-Assisted Penetration Testing Methodology
Traditional source-assisted pentesting involves reviewing code after finding anomalies during traditional testing.
For 15 years, teams have performed source-assisted penetration testing where clients provide source code. In this method, testers perform traditional penetration tests, and if an anomaly is found, they then consult the source code to understand and exploit it. This allows for faster vulnerability confirmation and enables precise reporting of the vulnerability's location.
AI-enabled source-assisted pentesting uses AI to proactively scan source code for flaws.
The newer AI-enabled approach starts by feeding source code to an AI tool. This AI agent, guided by human interaction, searches for potential flaws, helps filter out hallucinations and false positives, generates potential exploits, and assists in launching and refining those exploits against target systems. This workflow is significantly more efficient.
AI can uncover subtle flaws in obscure workflows missed by traditional methods, including authentication bypass and authorization issues.
Using AI-enabled source-assisted pentesting, teams have discovered critical findings in codebases previously tested repeatedly without success. AI excels at identifying subtle flaws in non-standard workflows—situations that arise from unusual user interactions or system states. These findings can include authentication bypass, authorization flaws, BOLA (Broken Object Level Authorization), IDOR (Insecure Direct Object Reference), cross-tenant access, and race conditions.
The AI pentesting workflow includes mapping the repository, focusing on high-value code areas, and iterative refinement.
The demonstrated AI pentesting workflow proceeds through several key stages: mapping the entire code repository to create a referenceable map, identifying the critical 20% of code likely to contain vulnerabilities, narrowing the context window for AI analysis, finding and validating potential vulnerabilities, creating test harnesses, developing exploits, and finally, generating reports. This iterative process helps manage the AI's limitations and focus on actionable findings.
Legal and ethical considerations, such as data privacy and NDAs, are crucial when using AI with client code.
Before utilizing AI models with client source code, it is imperative to ensure proper authorization. Uploading proprietary code to unapproved third-party models can violate terms of service, NDAs, and client agreements. Testing locally with models is an alternative if cloud-based solutions are restricted, though local models may have reduced capabilities compared to advanced cloud-based ones. Managing the AI's context window is also a key technical challenge.
AI models are adept at pattern recognition, code traversal, and iterative testing, but require human oversight.
AI models excel at tasks such as searching for specific patterns across codebases, following complex code dependencies, and performing repetitive tests without human fatigue. However, they are prone to hallucinations and errors, necessitating human intelligence to guide the process, validate findings, and ensure the accuracy and exploitability of identified vulnerabilities. Human engagement is critical throughout the AI-assisted workflow.
The process involves mapping the code repo, identifying high-value code sections, and progressively narrowing context for detailed analysis.
The AI pentesting methodology begins with creating a map of the entire code repository to provide the AI with a referenceable overview. Subsequently, the focus shifts to identifying the smaller, critical sections of code that are most likely to harbor vulnerabilities. This is followed by a process of progressively narrowing the AI's context window to concentrate its analytical power on specific functions or code segments, thereby improving the accuracy and efficiency of vulnerability detection.
AI can identify candidate vulnerabilities, generate test harnesses, and assist in exploit development and reporting.
Once potential vulnerabilities are identified, AI tools can assist in creating test harnesses—isolated environments to simulate and test the vulnerable code. They can then aid in developing exploits and, crucially, in generating penetration test reports, significantly reducing the manual effort required. While AI can produce around 80% of a report, human expertise is still needed for refinement and finalization.
Demonstration: Vulnerability Discovery and Exploitation with DataEase
The demonstration uses DataEase, a large Content Management System (CMS), to showcase the AI pentesting workflow.
The codebase used for the demonstration is DataEase, a large Content Management System (CMS) approximately half a gigabyte in size with over 1.7 million lines of code. The system contains vulnerabilities that were previously discovered and patched, allowing for a safe demonstration without disclosing zero-day information. The LLM's ability to handle a large codebase and even non-English code (Japanese in this case) is highlighted.
The initial step involves mapping the repository to create a referenceable structure and identifying potential attack surfaces.
The AI is first tasked with mapping the entire DataEase repository, generating a flat text file reference and attack surface notes. These outputs provide a structured overview of the codebase that the AI and human testers can use to identify areas of interest and potential vulnerabilities. Managing the AI's tendency to get ahead of the process is noted as important.
AI helps narrow down potential vulnerabilities by identifying candidate areas and functions for deeper investigation.
Following the repository mapping, the AI is prompted to generate a list of candidate vulnerabilities or areas worth further investigation based on the mapped code. This step helps cull the vast codebase down to more manageable and promising targets, focusing the analysis on specific functions or workflows, such as a 'save validate' path identified as a primary candidate.
The AI analyzes specific functions, providing focused context and detailing dependencies and data processing paths.
Once a specific function or path (e.g., 'save validate') is selected, the AI is given that focused context. It analyzes the function, its dependencies, and how data flows through it. This iterative narrowing of context allows the AI to perform a detailed analysis of smaller code segments, increasing the likelihood of finding specific flaws.
The AI identified a previously patched vulnerability, generating verification notes and suggesting potential fixes.
Through focused analysis, the AI identified a specific vulnerability within the 'save validate' function. It provided verification notes detailing the flaw, its potential impact, and suggestions for remediation. This process confirmed the AI's ability to pinpoint known issues in code, even without prior knowledge of that specific vulnerability during the demonstration setup.
A test harness was automatically generated by the AI to simulate the vulnerable process for exploit development.
With the vulnerability identified and analyzed, the AI was tasked with creating a test harness. This involved extracting relevant code segments from the large DataEase codebase and building a local, runnable application (in Python) that simulates the vulnerable process. This dramatically accelerates the ability to test and develop exploits.
The exploit was developed by the AI, demonstrating an unauthenticated write to the web root and achieving remote code execution (simulated by calculator pop-up).
Using the generated test harness, the AI was instructed to develop and execute an exploit. The demonstration showed an unauthenticated user being able to write to the web root, a critical security flaw. This was validated by achieving a simulated remote code execution, famously demonstrated by making the calculator application pop up on the target system.
The AI assisted in generating a penetration test report, including an executive summary and a CVSS score.
Finally, the AI was used to generate a penetration test report based on the identified vulnerability and exploit. The report included an executive summary and a CVSS score (9.6, cited as plausible given the unauthenticated write capability). While the AI provides a substantial starting point, human review and refinement are still necessary for the final report.
Implications for the Industry and Recommendations
AI presents an asymmetric advantage to attackers, potentially leading to hundreds of new zero-days weekly.
The widespread adoption of AI tools by attackers is expected to create an asymmetric advantage, dramatically increasing the volume of discovered and weaponized vulnerabilities. The prediction is that instead of a few zero-days per week, organizations might face hundreds. This shift signifies a period of significant challenge and instability for the cybersecurity industry.
Vulnerability management must evolve beyond traditional approaches to handle the increased threat volume.
The current approach to vulnerability management, often reactive and focused on high-profile targets, is becoming insufficient. The rise of AI enables widespread vulnerability discovery in less scrutinized software, including open-source components within closed-source applications. This necessitates a broader, more proactive approach, potentially termed 'vulnerability operations' (vone ops) or Attack Surface Management (ASM).
Critical recommendations include improving patching processes, re-evaluating business risk, and strengthening basic security controls.
Key recommendations for organizations are to: 1. Accelerate and improve all aspects of the patching lifecycle. 2. Reconsider business risk calculations, acknowledging that traditional outage windows may no longer be feasible. 3. Strengthen conventional controls like access management, privilege reduction, threat hunting, and incident response capabilities. 4. Prepare for an increased volume of concurrent attacks through robust incident response planning and exercises.
AI-generated vulnerabilities are not necessarily novel classes but well-understood types amplified by AI's efficiency.
While AI models can discover novel, previously unknown vulnerabilities (zero-days), they typically do so within known categories of flaws such as IDOR, BOLA, or race conditions. The primary impact of AI is the dramatic acceleration and scaling of discovering and exploiting these well-understood vulnerability types, making them accessible to a wider range of actors.
The gap between well-resourced and under-resourced organizations is widening, affecting sectors like healthcare and education.
There is a growing concern that the disparity in cybersecurity capabilities between well-funded organizations and those with limited resources (e.g., hospitals, schools, municipalities, small businesses) will increase. The cost and complexity of defending against AI-driven threats could overwhelm these organizations, making them more vulnerable, despite potential AI solutions being developed.
Community efforts, like hackathons and open-source tools, can help bridge the resource gap.
Community-driven initiatives, such as hackathons focused on developing AI tools for defense (e.g., forensic analysis) and the creation of open-source software, are vital. These efforts can help democratize access to advanced security capabilities, providing options for organizations that cannot afford expensive commercial solutions and driving down costs over time.
Future AI advancements may extend vulnerability discovery to closed-source software through reverse engineering.
Currently, much AI focus is on open-source software. However, AI is expected to become increasingly adept at reverse engineering closed-source software by analyzing the open-source components it utilizes. This will further expand the attack surface and the potential for AI-discovered vulnerabilities across all types of software.
A long-term outlook suggests a more secure software ecosystem, but the near future presents significant challenges.
While the long-term future may hold a more secure software ecosystem due to AI's eventual role in finding and fixing flaws, the immediate – next several years – will be tumultuous. Organizations need to prepare for a surge in vulnerability discovery and exploitation, requiring strategic planning and investment in cybersecurity.
Recommended resources include SANS AI cybersecurity summits, white papers, hackathons, and training courses.
The presentation highlights several valuable resources for the community: a free AI Cybersecurity Summit, a Cloud Security Alliance paper titled 'The AI Vulnerability Storm', a 'Find Evil' hackathon, the new SANS SEC543 course on AI-assisted source code analysis, and a field guide handout on using AI for vulnerability analysis. These resources aim to equip professionals with knowledge and tools to navigate the evolving threat landscape.
Q&A and Closing Remarks
AI pentesting applies to various applications, not just web apps, including compiled and hardware systems.
The techniques demonstrated are not limited to open-source web applications. AI can be used with compiled applications, potentially aiding in reverse engineering with tools like Ghidra. Teams are also exploring AI for hardware analysis, indicating a broad applicability across different technology domains.
Human oversight is essential throughout the AI pentesting process, from initial engagement to final reporting.
While AI significantly enhances efficiency, human intelligence must remain engaged. This includes guiding the AI, validating its findings, filtering hallucinations, and making critical decisions. The human element ensures that AI is used effectively and securely, preventing it from going down unproductive paths or generating false positives.
AI pentesting's role in compliance is likely, but governance and adoption will take time.
While AI-driven vulnerability assessments may eventually become a compliance standard, it's a long-term prospect. Governance bodies are typically slow to adopt new technologies. The future likely involves outcome-driven compliance, where AI tools are integrated to achieve comprehensive vulnerability discovery, rather than mandates on specific AI usage.
Token costs and model performance fluctuations require adaptable workflows and mindful AI usage.
High token costs and variability in AI model performance (e.g., models becoming less effective) necessitate flexible workflows that can accommodate different models. Responsible usage, focusing on fewer, more effective prompts rather than throwaway ones, is encouraged to manage costs and maximize AI utility.
Staffing for AI-assisted pentesting requires a mix of creative, methodical, and technically skilled individuals.
Effective AI pentesting teams need a blend of skills. This includes individuals who are creative and imaginative, methodical and detail-oriented, and strong in core penetration testing techniques. Continuous learning and adaptability to new AI tools and workflows are also crucial for team members.
Prioritization and resource allocation must balance emerging AI threats with ongoing security needs, especially for resource-constrained organizations.
Organizations should not abandon all other security projects. Instead, decision-makers must prioritize based on business risk and integrate AI-driven vulnerability concerns into existing strategies. For under-resourced organizations, community-driven initiatives and open-source solutions offer pathways to improve security posture against escalating threats.
Penetration testers can provide suggested code changes to clients, but developers must validate and test them.
When vulnerabilities are found, penetration testers can offer specific recommendations, which may include code snippets for remediation. However, these suggestions should be reviewed and thoroughly tested by qualified software developers in a QA environment before implementation to ensure correctness and avoid introducing new issues.
Ask a Question
*Uses 1 Wisdom coin from your coin balance
