Gamifying AI evaluation: How well do chatbots perform in cybersecurity challenges?

Abstract

As general-purpose AI chatbots become more advanced, their potential as assistants in cybersecurity learning is gaining attention. This paper evaluates the effectiveness of the most popular chatbots in solving Capture The Flag (CTF) challenges, a widely used gamified approach to developing security skills. From cryptography and reverse engineering to web exploitation and forensics, we analyze how well these AI models navigate real-world CTF problems compared to human participants. Through a student-centered lens, we highlight the strengths, limitations, and occasional hilarity of AI-assisted problem-solving. Can AI crack the code, or will it get stuck in a loop of confusion? Our findings provide insights into the evolving role of general-purpose AI chatbots in cybersecurity education.