AI Pentesting: Comparing AI and Human Approaches to Vulnerability Discovery

Evolution of AI Pentesting Methodologies

Early AI pentesting experiments at Intruder involved strict methodologies, where agents focused on specific vulnerability types. This approach led to agents missing issues outside their assigned remit or miscategorizing findings, resulting in confusing reports.

Conversely, a complete lack of structure also proved ineffective, as unguided agents struggled to manage large applications and missed numerous vulnerabilities. The current development balances agent freedom with broadly defined roles and guardrails to prevent damage.

Beyond Strict Checklists

Traditional human pentests often rely on strict methodologies and checklists, providing assurance that all listed items are tested. However, this can create a false sense of repeatability, as impactful findings often stem from a human tester's experience and ability to notice unusual behaviors.

The AI pentester at Intruder achieved more impactful results by not guiding agents through specific vulnerability types. An example cited is a complex authentication bypass discovered by AI agents that had gone unnoticed by human testers over five years.

Comparing AI and Human Strengths

While AI can uncover complex vulnerabilities through flexible exploration, human pentesters bring nuanced experience and the ability to follow up on subtle cues. The article suggests that AI pentesting is not a direct replacement for human expertise but offers a different approach to vulnerability discovery.