A security researcher used artificial intelligence (AI), specifically OpenAI’s o3 model, to find a serious flaw in the Linux kernel, which is the core part of the Linux operating system.
The flaw, which has been documented and fixed, was found in a part of the kernel, called ksmbd, which lets computers share files over a network using the SMB3 protocol.
The researcher was checking ksmbd for problems and tested o3 to see how well it could spot bugs. Using only o3’s basic interface, without extra tools, the AI found the vulnerability.
The flaw is a use-after-free error, meaning the system tries to use memory after it has been cleared, which can cause crashes or let attackers run harmful code. It happens when two connections to the server try to use the same session at the same time, and one frees a piece of data while the other still needs it. The AI understood this complex issue and pointed out where the error occurred in the code for the “logoff” command, which ends a user’s session. This discovery is notable because it’s one of the first times an AI has found such a flaw in a major system like the Linux kernel.
How the AI performed
To test o3, the researcher first used a known flaw in the kernel’s Kerberos authentication path, which verifies user identities. The AI was given about 3,300 lines of code and found this flaw in 8 out of 100 tries, better than other AI models. When given more code, about 12,000 lines, o3 found the new flaw.
The AI’s report was clear, like a human’s, explaining how the error could be exploited. It also suggested better fixes than the researcher’s initial ideas, showing AI’s potential to help experts. While o3 made some mistakes, producing false positives, it proved useful for finding real issues.
This work suggests AI can make security researchers more effective, though it’s not perfect. A fix for the flaw has been released, and researchers are encouraged to use AI tools like o3 to improve cybersecurity.