A developer built a deliberately vulnerable web application and spent $1,500 on API credits to test whether large language models (LLMs) could autonomously hack it. The experiment involved several LLMs, including GPT-4 and Claude, tasked with exploiting common vulnerabilities like SQL injection and cross-site scripting. Results showed that while LLMs could identify vulnerabilities, they struggled with multi-step exploitation and required significant human guidance. The developer concluded that current LLMs are not yet reliable for automated penetration testing without expert oversight.
This experiment is a reality check for the AI hype cycle. LLMs are impressive pattern matchers, but they lack the strategic reasoning of a human hacker. They can read a manual but can't improvise when the plan fails. The $1,500 price tag tells us something important: we're still in the era of expensive, supervised AI assistants, not autonomous agents. Cybersecurity remains a human craft.
But that's not a bad thing. It means the future of AI in security is collaborative, not replacement. Tools that augment human experts will emerge faster than fully autonomous hackers. The smart money is on hybrid systems: AI for reconnaissance, humans for decision-making. This experiment didn't prove AI is useless. It proved AI needs better scaffolding. And that's a solvable problem.