Security in the LLM Age - Greg Kroah-Hartman

[00:00] Very good. The EU has mandated USB-C and no tops. They can stop. I'm very happy to live here. Um, hi, I'm Greg. Anne always makes me come and give a new talk every year. It's hard. I have cribbed part of this from other talks. But, um, I'm going to talk about LLMs and us. Us as in kernel developers and maintainers, then open source developers, maintainers. And what's going on with us right now. Um, so it's not going to be generic public people. Developers. [00:46] And that didn't work. Boop. Okay. Use my fingers. Um, this is just me. Um, again, my obligatory disclaimer. Um, but somebody owes me a lot of wine. This has not been fun. This has not been a fun past 6 months. And I have people talk to me, "Hey, what are you working on? How's it going?" And I'm just going to show them this. [01:21] A year ago, I gave a talk at a security conference and said, "Linux kernel is going 50 CVEs a week. Isn't that insane?" And we're really running 55. It's being nice. Everybody's like, "That is crazy. What is going on in the world? This isn't sustainable." We're doing 33 a day right now. Something changed. But all that work uh Lee Jones, Sasha Levin, and I did in the beginning to handle tiny amounts of CVEs, tiny being 50 a week, has paid off. Our tools scale. They work really well. I'm very happy. Um, everybody else in the CVE community, CNA is panicking and having to redo their whole infrastructure. We're doing just fine. We can go faster. It's not going to bother me. Challenge accepted. Please. [02:20] Um, these are bugs. These are A CVE at our level is a something that can cause a crash. You guys, I've given a talk about this before, another. But please, this is going to be my refrain: Do not panic. Um, this isn't a big issue. Now, go into speak into it more. You see a lot of stuff in the press, you see a lot of doom marketing right now, and please do not panic. So let's talk about how this all started. [02:50] I got a phone call, and I don't get very many phone calls, um, that said this in the beginning of February. Actually, I think I got notice like right before it was announced. And of course, I panicked. Um, I got access to the raw data, and it's been long enough that I can actually talk about this, hopefully. Um, so 79 they found 79 bugs. This was in the press. It went out everywhere. And what Mythos did was things that we have been talking about in the past. Sasha Levin and Julia here a decade ago came up with the idea that if you look at previous bug fixes that have happened in the past, where have we done this same type of bug fix, or where has this same type of bug fix not been applied elsewhere? And do this. [03:38] That's what Mythos did. That's what Anthropic did. They took that old research, did not really cite you guys, sorry. But we did this first. You guys did this first. And they looked through Ghostscript and they said, "Hey, look at all the bug fixes over here. Where these bug fixes need to be applied?" And they found some problems. They did that with the Linux kernel, did that with a whole bunch of other things. Very rudimentary things. LLMs are only there's no intelligence behind it, it's the fuzzy pattern matching. And code is a good, logical pattern. It works really well. So fuzzy pattern matching works really well on code, as Julia and Sasha have proven in the past. [04:16] So that's all it did. So they said they ran it through and said, "Oh, we found all these bugs! The world is on fire." So let's look at these bugs. I got the real raw data. 24 of them were nothing, literally nothing. Something crashed. No report, something crashed, the VM died. No help at all. 14 of them, I don't know why they documented it, it wasn't even a bug. Nothing happened at all. I don't know what they were doing. Three of them were completely made up. I don't like using the word hallucination because that gives an idea that there's an entity behind this stuff. It's just fake. They made up data out of thin air. Totally not right at all. And here's my funnest one. [05:07] These tools want to please you. They're very sycophantic. They If you say, "Give me a bug," it'll work really, really hard to give you a bug. So hard it'll go out and read our mailing list and report bugs that other people have found and fixed! Four of them, I will declare Anthropic fixed. They had some bugs. They were like minor little things with authenticated NFS servers to authenticated clients, so they knew who was going on in the network. Great, we fixed them, the world moved on, we didn't even notice. 11 of them were famously found and fixed by other people in public before this report even came out. They scraped the web and found the bugs. [05:49] And this is a pattern that I mean, Willy and other people in the security list we see all the time. These tools will go out and find bugs that other people have reported and give them to you and say, "Here you go. Aren't you happy?" And then you do, and then you they give it to somebody else and they're happy, too. But these are things that we've already done. These are things that people have already done. Again, 11 people have already fixed these. And then there was 26 actual bugs. Okay, well, almost. [06:17] Um, how are you guys doing on math? I can't write this joke. I mean, it's LLMs cannot count. They gave me a tarball and said, "Here's the 79 bugs our tools found." Okay. Anyway, um, I'm sorry. No, LLMs cannot count. Um, it turned out 20 fixes were needed. Great. 20 fixes. Let's work through these. Let's see what these are. [06:53] Seven, assume a malicious file system image. You're not laughing, Willy. Um, this is the quintessential newbie security bug. You take a file system image, you manually manipulate it and say, "Hey, if root mounts this, bad things can happen." This is called out as not a security bug so often, we both have like speed keys that just repeat it out. Um, this is not a security issue at all, and it's so been documented even before Mythos came out. They should have known better. This one was fun. Assume you can eject a packet into the middle of the stack. You can do this if you're root and you're doing some cool networking debugging stuff. You can inject malicious packets in to do some fun things. This is not a real bug, not a security issue at all. [07:41] Two of them were no MMU bugs. And I fixed them. And I thought Jens was going to be here. I owe Jens a lot of beer for this one. It was an I/O ring no MMU bug, um, I fixed, and a regmap MMU bug that just got merged today, I think. Took me a while to get there. These are not real issues. The fun thing about Mythos is it's a cool framework. I mean, these are frameworks around a dumb chatbot that can do and it can run some tools. It could make made me a virtual machine and no MMU RISC-V virtual machine and wrote me a little test case in a Perl script that exercised I/O ring and showed that it was broken, and showed how to fix it. It also proved that nobody ever ran I/O ring on no MMU systems. And it's kind of proven that nobody really runs IO or no MMU. [08:35] Um, we found a bug in dev zero with no MMU that's been there for 6 years, so I don't think anybody runs these systems. Um, pardon? We fixed it before, and it came back, right? Oh, out of tree, yeah. And is Curt here? I mean, I feel bad for s um 68k, but it is what it is. Um, there is six SCTP networking bugs. Um, SCTP is for enterprise networks that the telcos use on authenticated systems. So, an untrusted user, there is no such thing. Um, I will admit I am still sitting I fixed them. I haven't merged sent them upstream. It's just so minor. Nobody really cares. Um, five of them were patches. Two actually were duplicate. Um, there was two IPv6 minor issues. Who uses IPv6, please? [09:36] Okay. These were actually even worse. They were minor issues of minor issues of minor issues of minor issues of my um IPv6. I had to convince the network maintainers that they really were a bug and that had to be merged, and it took a long time. Nothing real. And then there was one GPU driver for a local malicious user. If you have a local malicious user on your desktop that has access to your GPU, you can do much worse things than this. You have real things. So in the end, Mythos broke down to 10 real bug fixes. [10:06] And if you look at what John Corbet's reports are, um, for the past couple years, we're running at 9 and 1/2 um patches an hour, and then now we're running at 10 and 1/2 patches an hour. So what all of Mythos, that whole big marketing issue of 79 bugs came down to 1 hour of kernel development. Marketing is bad. So, do not panic. Um, this isn't a big deal, right? 1 hour of kernel development by one tool. [10:39] To be fair, I got access to Mythos, and I was Dosed by or doxed by Jim Zemlin as being the only named person with it. There's other people that have access to Mythos. And, um, I found way more bugs in an hour and fixed them than Anthropic did, but that was just little, tiny stuff. But anyway, do not panic. This isn't a problem. So how's it going? How are we doing with this today? [11:04] And this comes down to the root problem. This is the number of in 2026, the beginning of 2026, the big problem is people are not rolling out fixes to their systems. This is the traditional discover, disclose, patch, and deploy cycle. Um, previously it was after we discover and fix something, it took 63 days for it to show up and people trying to exploit it. Now we're at -7 days. So this is real. And the one thing about these LLMs doing they're really dumb bots, but they will persist and persist and persist and persist, is that they can chain together very minor issues and get access. So you have to fix all the minor issues along the way, which I'm very happy because you now have to update all your stable bugs. But you they can chain them together, and they can take advantage of this, and it is taking advantage of this today. So this is the big problem. And the big problem is not that our software doesn't work, our software works. Linux works. It just doesn't work under really strange and obscure conditions and stressing. [12:02] So this is it. It is our own fault, it is our employers' fault. The bill is finally coming due, and that's true. And people finally realize this. I'm actually seeing banks say they're going to actually update their software. I'm talking to financial people and they're worried. I'm like, "Oh, the thing we've been telling you for 15 years you're going to do? Yay." So software is working, but it wasn't designed for perfect security. The good thing about these bots and these tools, this fuzzy pattern matching is we're going to grind through this. We'll fix it up and we'll do it. Again, do not panic. We've done this. [12:38] And we've done it before. People in this room were I mean, I did Y2K stuff, I know other people have. But the fuzzers, remember six, seven years ago, the world is on fire, fuzzers are hitting us all, we're going to die, it's miserable. Remember those presentations? What did we do? We sat down, we did the work, and we ground through it. We fixed the bugs. Fuzzers are still hitting us today. Sit down, do the work, and do the work. It's not a big deal. These tools are finding real bugs. We sit down, we do the work, we fix them. Not a problem. And when you fix them all, it stops. These things do not persist for forever. [13:17] Oh, look at I'll show you rsync. Rsync's a great example. Tridge sat down with these tools with me and a couple other people, and we ground through all the bugs that these things found. We fuzzed the heck out of them. They did a lot of really, really good work. Tridge did a whole new infrastructure of this stuff. And now rsync, the latest version of rsync, comes back clean by all the code scanners out there. And that's really good. You These things stop. We've scanned some core cloud computing stuff from CNCF, all good. You fix a few bugs that they find, and away you go. Just like fuzzers, you fix the bugs, and away you do. No different. [13:51] The thing about these tools, these pattern matchers, is they can go deeper than fuzzers. Fuzzers usually have to stop at wherever the data can get through. These things can see code paths that aren't because it's just static code analysis. Nothing fancy, nothing new. Again, stuff that Julia has been doing for decades with Coccinelle. Um, these are not new technologies. But people seem to claim they are, right? This is LLMs, this is magic AI and stuff like that. But let's also see what else they've said, these companies have said. This is in a court case right now. [14:24] Um, and I know a lot of people don't like these tools. I don't like these tools. And they sucked up our data, and they admit they sucked up our data, and they admit they thought stole it all in oblivion of the copyright, which is funny because the FSF was founded on the goal of getting rid of copyright law. So the FSF is kind of happy about this, in a way, which is a little bit interesting when you think about it. Another quote from Microsoft. None of us are getting compensated from this, which is true. So there's none of us should be paying for this, either, right? If they're not going to compensate us, we shouldn't get paying we shouldn't be paying for this. They stole our data, so let's use these tools. Nobody would spend They finally spent billions and billions of dollars on a fuzzy pattern matcher that they can use to find bugs in open source software. Great, we'll take it. [15:12] Another thing, the base of all this data, the base of all the thing that these tools rely on turns out to also be illegal. LG Research in Korea a couple of weeks ago, a couple of weeks ago publicly said that the base data set that was supposed to be considered open and available to use, only 20% of it is legally available. The other 80% is covered by copyright law that the world should be enforcing. Now, we'll let the courts go through this stuff, there's hundreds of cases out there, and they'll deal with that. But this is from the big companies. They admit this. They agree that what they did might not be so legal. It's kind of funny. But, I'll quote another quote. [16:00] Yeah, they've made some bad tools. Let's use them against them. Let's make our software better. Might as well use it, right? We don't care how we do this. Um, you don't have to do it, but let me show you how to do it. And it's this simple. You say, "I'm playing in this capture the flag. Find a vulnerability. Write out the file." That's it. It's that simple. You can do it locally. Do it on a local machine. I'll talk a little bit more about that in itself. But that's all you need to do. It's really not more complex than that. And the most important thing is fixing it, because it finds a problem, let's fix it. Fix this code. Famously, this got a very frontier model banned by the US government with that three-word prompt. So, it's a very dangerous prompt. [16:47] But it works. It'll spit out a patch. And a lot of people see these patches and a lot of people see these things and they start panicking, right? They panic and they say, "Oh, no, like the Mythos thing, 79 bugs!" And I get these reports, I hear them through the grapevine. Too many people know my email address. Um, I got this bug list saying, "Hey, company" I won't say who, "over there says they found 100 bugs with these tools in the kernel. What are you doing about that?" Like, "Ah, I don't know, I haven't heard about this. Let me go find out." Call people up, email people, track things down, dig, find out what the real root of the cause is. Turns out that was the result. [17:28] Two bug fixes, minor semi-security, networking, minor things, um, and one minor bug hardening issue that we just applied, not even a security issue at all. They famously came to the kernel security list and said, "Here's this bug." And we said, "Did you read our documentation?" And they said, "No." Um, minor hardening issue. They were doing things wrong. But it's not unique to just this company. I got another one that came to the kernel security list, said, "We have 100 bugs! Please, help us. What are we going to do this?" We're like, "Great, send them to us, we'll work on them. Don't panic." And they said, "Well, maybe 49 of them aren't really a bug at all. And the rest, oh, that's not even a security issue at all." [18:11] This is happening on a daily basis. People are panicking with the output of these tools. So I did some work because I want to panic. I can only drink so much wine. Um, and try to figure out how good is the output of these tools. I've been running these things for a while. You say, "Find a bug, fix a bug." The fix a bug weeds out a ton of false positives. So you have a patch. What do you do with this patch? Is it good? Is it not? So let's see what's happening. Well, a lot of but things are happening, and this came from John's talk. People see these patches and they start flooding at us. [18:47] Turns out this is an email address that is does this about once a year with a different name behind it every time. So as a maintainer, if you see this stuff coming into you, check on the link, look on lore, see what else they've sent that day. If you see this, don't worry about it. You can kind of ignore it. John talked more about this stuff anyway. So you see these patches, are they real? Are they not real? What are you as a developer and a maintainer supposed to do with this stuff? Well, here's some research I did this past year. It turns out everything lies at least half the time. [19:24] I proved this this past summer. I took six poor graduate students and they I fed them a whole bunch of bugs, and they worked through them all, and they reviewed them all, and they got really frustrated, and turns out half the patches that were these things generate that look correct, they feel correct, and they want to be correct, are just flat out wrong. Either they don't apply or they don't solve the problem at all, or there is no problem at all, or they're that code path is impossible to hit, or something else. Or it just isn't even a security issue, even with our documentation that says what is and isn't a security issue. The bots will lie. So when you see patches coming to you that are LLM generated, assume that half of them are going to be wrong. They fooled me. That is famously a patch that fooled me. I thought it was right. [20:15] These things want to persuade you. They show you fancy stuff, we're used to assuming that there's a human behind this with some reasoning. There is no human behind these patches with reasoning, it's just fuzzy pattern matching. And when you see these things, we want to anthropomorphize them and think that somebody had some feeling behind it, and had some reasoning to make this patch. They didn't. They didn't test it, they didn't run it, they didn't have a bug in the first place. Half the time, these are wrong. And they're also really dumb. [20:44] These are only pattern matching. This is a very common bug that these things love to do. They love to change mutex_unlock to mutex_destroy. Few people in the room that got that. Um, what would happen if I did that? Come on, please. How bad would that cause things to go? That would break really, really bad. That would add a bug. Mutex_unlock is correct, mutex_destroy is not. Um, I don't know what in the documentation caused this, but this is very common pattern. Watch out for this. People will be sending patches that do this. These are pattern matching tools. They think that the language of destroy means more than unlock, I don't know. [21:25] But they also like talking, and they want to throw tons and tons of checks. Nobody except um the Bfs people write changelogs like this. Um, anybody writing a wireless or TTY driver patch does not write this. I don't write like this. They will throw changelog text at you. It is bad. The interns I worked with, in the end, they just deleted the whole changelog and went off the code. And it turned out that actually works. That they can they can take away this whole persuasive, sycophantic body of text, and look at the code, come to their own decision, and work from there. And that's the best thing I could help you and persuade you to do. Ignore that body. If you see people send you that stuff, say, "Please start over." Do. You don't have to take their patches. [22:17] And they also love love to add comments. I mean, who adds what? Seven lines of comments for two lines of code? In a driver? Um, they also love adding booleans and flags and things like that. These things have been trained on our huge corpus body of work, and which means we did a lot of bad things a long time ago in the kernel code. Famously, people said, "Hey, can we make a good kernel developer bot? By take the base thing and add all our LKML data and all our Git history to it, and let's see what comes out of it." What came out of it was really bad code that cursed. [23:02] But that that shows you these are just pattern matching tools. That's the way we used to behave. We know how to behave differently now. We know how to use guards, we know how to use different coding patterns, we know better security patterns. But in the past, we didn't. Our past history, we didn't. We used to yell at each other. We don't yell at each other as much. Or we do so in much nicer terms. And that's good. These things are just trying to encode our past history, they're not showing how humans evolve, how code bases evolve. And that's important. So when you see these, they're going to be making common mistakes, common patterns, common old coding style. [23:36] Remember that old Perl scripts that used to be on the web, Dave's Perl things for CGI, that was famously insecure? That is embedded in these LLMs, so if you ever ask it to write a CGI script, it's going to be totally and completely insecure. Same thing with kernel code. We've seen new code come out of these things, bad patterns, old stuff. Coccinelle just rips them to shreds. It's awesome to watch. I don't know why they don't even run these scripts. Um, they don't know the new patterns. It's sad. So, see lots of patches with lots of text, lots of comments, push back. Push back: "How did you test this stuff? Where did you get this information from?" Sometimes it's right. Sometimes it's not. Be like Jens. Jens forced me to create a reproducer and prove that it was needed. Push back. There's nothing forcing you as a developer or maintainer to take these patches. Push back. [24:26] Another fun thing is these bots leak. Anything it tells you, it will tell somebody else. Famously, the Anthropic stuff, in the very beginning of Mythos, it was scraping data from the public and doing that. In the On the kernel security list, we get people mad at us every single week because they discovered a problem, they researched it, and found a bug that was reported a day earlier in public. Um, they get mad, they want credit, they want lots of stuff. We've seen this happen. Anything that you upload to these bots, it will leak. Famously, the mathematicians finally realize that, they were using Claude to do a bunch of research, and it turns out, "Oh, the company saw the data, decided to do something else with it, and it spit it out to somebody else." Anything you upload to these people, they will leak to somebody else. It happens. We have proof of that. It's been that way since the beginning. They want your data to train on and give it to somebody else. If If it you ask it for a bug and it gives you a bug, and you say, "Thank you," and then it says, "Oh, I pleased him, I'm going to please Willy and give him the same bug, too, because that's the same thing." Well, no. It's going to do the same thing. You upload a problem to them, you do that. You upload a security issue to them, do not do that! Because it will do that. [25:39] And the funny thing about these companies and this model is they never paid attention to history. Um, they spent billions of dollars, like I said, to make a developer tool that nobody's going to pay for. Um, Coverity, which is a deterministic tool, not fuzzy, very good, very low false positive rates, wrote famously this very big, long postmortem of the company, the tool, how they try to get this to work, how they try to sell it, how they try to get adoption for it, and all the problems involved in trying to sell to a developer community, and how they failed. It's like these bots never read this. They are doing the same things. The false positive rates of 50% means nobody likes working with this stuff. We all hate this. We will put up with it for a little while, but we'll push back hard. Coverity proved that you cannot have false positive rates of anything over 20% to get anybody to take you seriously. And still, we hate Coverity because it's 20% full or wrong. This is they're ignoring how history Nobody would pay for Coverity. Coverity finally found a benefactor and sold off. I don't think we'll ever pay for tools that give us 50% false positive rates. So, no fun. [26:52] Um, how we've dealt with this, um, we document the threat model. If you own a kernel subsystem, and you know your threat model, please add the text to this stuff. It helps us out so much because the people actually running these bots do read the documentation, and it will sometimes do that. They're not really, but at least this kernel security team can say, "Go read the documentation." And we push back hard. We still get, "Oh, file system handcrafted file system images bugs" on a weekly basis. But eBPF stuff, networking stuff, they're really they're documented really well. Um, we trust hardware, we trust devices, things like that, it's documented. Add that for your kernel subsystem, please, it'll help you a lot. [27:32] Um, OpenSSF has made, if you have other user space programs, a whole tool that will help you make a threat model document for your program. And yeah, I've run through it, asked you a bunch of questions, spits out a generic one, you can edit it and add it to your repo. And that way, the bots trying to send tell you, "Hey, I found a security bug in your program!" And you're like, "That's great. That's the way it works." Um, you can point to that and say, "Don't don't bother me." Use those kind of tools. [27:57] Um, we also have asked and require people to Cc maintainers. So you as kernel developers and maintainers are going to start seeing more security bugs copied to you and the security mailing list alias. Um, I'm sorry about that, but it reduces our load. And if you get these bugs, just say okay and work on them in normal time. If you think this is something that should be discussed publicly, which I don't know, 50, 80% of these should be, say, "Go to the mailing list," and push them to the mailing list. Please do that. Do not feel bad about this. We're also asking for a patch to start with. Anybody who uses these LLM tools, the first cutoff is say, "You can have them spit out a patch." The local, dumb models will spit out a patch. Have it do that, and that gives you something to work off of. We say in a nice way, "Please send us a patch. That way, you get full credit for when it's resolved." People want credit, so appeal to their vanity. It works out really well. [28:51] We've also got some We got some money. We got a full-time developer helping us on security at kernel.org, working with other stuff with the AI tools. I will say thanks to OpenSSF and Alpha-Omega for funding that. It's working out well, we're happy about that. So you as a developer, what can you do? You can push back. If it feels wrong and looks wrong, push back. Famously, the networking developers would always get mad at people that if they didn't put their header files in a pretty Christmas tree way, and they'd ask for a patch back, and I wanted to know why, and they're like, "Oh, we just want to see if there's really a live person on the other end, and whether they'll talk to us." And that's good. You see those stream of patches going down, that wasn't a live person behind that. [29:35] Um, push back. Ask for clarification, ask for something. If they don't ever respond, assume that you don't have to respond, either. Push back. Ignore the doom marketing. It isn't real. They're trying to sell you something. We're not the buyer, so don't worry about it. Um, run local models. Local models are good enough. You can run them on your desktop, you can run them on a separate box. Run local models. You can say, "Go do this stuff. Come back in a day, or come back in the morning, and it's done." Run local models. They're free, you can download them, run local models. [30:07] Um, never ever ever upload anything that's public that's non-public, because it is at that point in time considered public. So we have so many people saying, "Oh, just run the feed of security@kernel.org through our public or through our model, we won't tell anybody." Like, "Yeah, no." Um, we do not do that. You should not do that at all. Do not upload anything you don't want to see on the web. It's just like normal stuff. I don't know why people don't realize that. The same thing. [30:33] And fix the bugs. Just fix the bugs. Once a bug is fixed, it's not going to come back. If you find it, if it actually looks like it's a bug that these people sent you, take the patch. And people are taking the patches. Look at the number of CVEs. This number is not slowing down. I'll show that more in a minute. But fix the bugs and eventually we'll get through this, we'll grind it down like we did with the fuzzers. It's not a big problem. I'm probably making this sound too sad, but it's we'll get through this. And ignore whatever's coming tomorrow. They're trying to get you to sell something. Use what we have today. What we have today is good enough. It works pretty well. If you want to use it, if you want to ignore it, but use it today. [31:09] There's a whole document out there. OpenSSF has put this, so you can send it to your manager. Say, "Hey, here's how you might want to consider this type of stuff, and here's the issues involved." It's actually pretty good. Um, and how to do it yourself. There's a whole talk later about sashiko tomorrow. Run it locally. Run sashiko locally. That's our bot that does code reviews. Um, famously, I'm sending out patches that sashiko find bugs in, and I'm going to just run it locally, it's all better. Um, Chris Mason has, again, like John said, we've now documented how to do code review for different subsystems really well. Contribute to that. That works really well, sashiko pulls those in. [31:44] Um, get an agent. These are all open source tools. Run things locally. Uh they're just a babysitter around the dumb chatbot. That's all these things are, that's all it is. Um, again, Chris Mason wrote one for kernel development. Takashi from um uh sound subsystem has written a bunch of other stuff. I should have put his list here. Um, these all work really well. Nothing wrong. Um, get a local model, download it all, run it locally. Please do not ever upload anything to any of these companies. And I'll call out Willy, did a great great documentation and how to do this, and actually solved the bug that I had because I couldn't figure out how to get something running. It worked out really well. So please, look at that stuff. [32:27] So, we can go back to this old chart. This is where we were as of 7.2. Now, kernel CVEs come in backwards, you know, when they do that. And I didn't chart of where they come from. It's pretty flat except for 5.15. And 5.15's a standout for where the bugs came from because somebody thought it was a really good idea to put a SMB server into the kernel. And it wasn't. But they're I give those developers credit. They are on it. They fix the bugs and whatnot, and they got the highest count of CVEs fixed for their subsystem than anybody. That's a good credit. Um, but here's where we are after 3 weeks of development. [33:06] Um, these numbers aren't slowing down. Um, but once we get through this, we'll be okay, I think. We're deleting code, we're deleting we're deleting network product protocols that nobody uses anymore. We're deleting drivers. Famously, one of the um interns, the graduate students I had found was trying to fix a driver and the subsystem maintainer came back and said, "No, nobody uses that anymore, just delete it." And then email me saying, "Oh no no, didn't it didn't work." I'm like, "You just deleted like 3,000 lines of code from the kernel! That's a good job!" You fixed more bugs, so they were happy. Um, you have to teach people that it's okay to delete code. This is going to be big for a while. It's going to be a rough 18 months. [33:48] Um, somebody called me on this, I've been saying 18 months since March. Um, maybe it's 12 months, I don't know. But we can get through this. We got through this for fuzzers, we can get through it with these tools. Um, and once we do, we'll be we'll come out the other side, and our code will be better, and it won't be a problem. And that's all I got. Thank you. Nobody heckled during it? Wow. Come on, Willy. [34:23] Um. There is something unfair in the current situation, which is that uh LLMs uh which are hosted by large companies uh are just uh spitting a lot of text at uh developers. And it's really, we all know that it is a pain to to read. Uh, the only way to defend against this, honestly, is just to use another LLM to parse their their walls of text, and it's totally inefficient. As you said, there is a risk of leaking uh some information, some sensitive information outside. Not everyone is able to host a local LLM because uh it requires at least a moderately powerful graphics card, and at the current price of RAM, it's not easy. And um we also lose a lot of information. Uh both of us have read uh significant Mythos reports, and we know the pain it is. You have the whole report on a single line of uh 4,000 characters and whatever. It's absolutely horrible. And a lot of information is lost. I mean that there are probably hundreds of thousands of tokens of context uh before the report is emitted. There is everything in it to just write a patch, and it's still not possible to get one. And we just have to try to invent a patch after that from the report. It's not always correct because it probably misses a lot of information. And I think we we need to insist, one way or another, even more to get these patches uh from the analysis, and not from the report. [36:01] Yes, push back hard. Um, I've sat through too many presentations by companies saying, "Here's the pipeline we've generated for our developers. It comes out here, and we do this triage, and we do this thing. And at the very end, because we protect our developers, our closed-source developers, we only give them the really good things." I'm like, "But nobody's doing that for us." They're just giving us raw data. And nobody cares about us. Um, it's hard. But there are some groups um like I can't remember the Linux Foundation group that's trying to coalesce a bunch of these reports. They famously helped me with the Company A 100 bug reports, and oh, three are actually valid. So, there is some people trying to realize that you cannot just throw these walls of text at us. But if you get a wall of text thrown at you, push back. Nobody's going to object to that at all. That's one thing I want to tell everybody: you have the right, and I strongly encourage you to do this. Push back on them. It is an asymmetrical relationship, and we need to push back on that. [36:58] And in addition, I would say that every time we ask the LLM to try to produce a patch, it's the best moment where it realizes that uh it's not a bug. Right. So, if you have to generate That's why I said here's the report, and and make a patch. And if you make a patch, it cuts out a whole bunch of junk. So, if you just get a bug report, we are seeing tons of people just sending bug reports to public list now, first thing I just say is, "Just send me a patch." Because I have enough patches to review right now. [37:25] Hi, thanks for the lighthearted uh presentation in the middle of this doom. I had a question as a recent new contributor, and I know many of the other uh new contributors, notably from the LF mentorship program, are encouraged to make uh to stop making small fixes, which tend to look uh sometimes a lot like to the LLM contributions. As a maintainer, whether for security or drivers or even staging, do you feel like uh these contributions uh look similar to you? And if yes, what could we do to actually uh come and build goodwill to our maintainers so they don't necessarily dismiss these contributions as uh some drive-by uh LLM or uh resume-driven uh contribution? [38:13] Um, what you can do good So yeah, I have banned LLMs from the driver staging, unless you have the hardware and can prove you tested the patch. Because that's what staging is for. Staging is to learn how to do development and get involved in our community. Um, you can show if you want to contribute to a patch, and I have done this, because I have had to contribute to a lot of different projects because I had access to Mythos, contribute a patch normally, as a human. Show what you did, show the work, show you're willing to respond. Do not send 100 patches at once. Send one or two. Famously, there's two patches from mine buried in the Kubernetes GitHub repo that I need to get merged, and I need to redo it again because they thought I was a bot and I need to prove that, no, this really was a fix. Um, just push back. And but as a person, you need to be able to prove um that you are a person. So kernel development is all about trust. And if I take patches from you and I don't know who you are, now I am responsible for that patch. But people we trust, and we trust that they not necessarily that they got it right, but I trust that you're going to be around to fix it when you get it wrong because we all get it wrong. So you need to build up trust, you need to prove that you are a real person. Famously, there's somebody who I thought was a bot, and I'm going to meet in Prague, is actually, no, is a real person. Um, and that's good. Go to conferences, talk to these people, be communicative, respond to their emails. Just respond to an email. That I that would be that would cut out a whole lot of stuff. See that that is there. So, shouldn't be that hard. Just act human. Thank you. [39:51] So, so one of the things that you posted was the big change log that LL- AIs love to just do uh, you know, you know, verbatim vomit out onto the change log. So what I've been doing now, this is my little technique. And man, I wanted I'm not going to be at the maintainers summit, I was uninvited, so. But uh I'm asking people to there that are going there to please bring this up. If someone does give you that crap up there, just come back and say please, you know, write it Throw that out. Give me a summary. Write so you show me what what is there. And here's what I plan on doing now. If they don't do that, I and I have to go through and read that crap, I will take that patch, remove their Signed-off-by, put on a Reported-by for them, because they didn't do the work. It shows me that they just had something else do the work. They don't deserve Signed-off-by. They don't deserve author. They don't deserve anything. That's That's my feeling. I don't know how legal or whatever that is, but I'm like getting to the point if you can't explain what it did, you didn't do the work, you don't deserve to be an author. I don't know. Flawed. [41:58] Continue to that, it's uh there is a blog post by somebody I don't remember, called Don't Be a Proxy, like uh meat proxy between uh LLM and kernel community basically. So it means Yeah, don't be a meat puppet. Because because there are some cases when people even replying to the comments using LLM. Very true. Yeah, don't be a meat puppet. [42:28] Yeah, uh also about the comment for the changelog, uh I know that there also some people that where where it's difficult for them to express what they did because English is not their native language. But then the I I agree that they abuse of the of LLM to produce that, but maybe just You can see. So, we we have seen that from a like Andrew Morton said that um the non-English-speaking native non-English speaking changelogs have gotten a lot better. And but it is much different when I write this in a not-English language and translate it, than that. It's obvious. It's very obvious. Obvious. Yeah, it is really So, take a look at some of these patches that go by just to get familiar with how these things look. I recommend all maintainers do that just to see these patterns, cuz these patterns aren't changing. Um, people might work around it, they might get better, but today this is what they look like. Um, it's really obvious, so it's not a problem. [43:27] You can get them to output sensible text. It starts with, "Use a tenth of the words." And, you know, the same thing I tell my children, never use words where silence will suffice. Ah, you can, but and you can hide it. We're not trying to do that. And you can make better change logs. And like my graduate students are like, "Oh, just delete it and write it yourself." And it's Yeah. But used correctly, they are an amazing tool, right? Sure. do a lot of work. But they have no taste, right? They can they can do the typing, they can go and find things and and fix a lot of boilerplate, and they're amazing for building tests, especially. But number of times I have to write hash include stdlib.h has gone down quite a lot recently. But yeah, they are a tool, and you use them for what they're good at, and Yeah. [44:10] And I, I mean, I have fixed some really fun networking bugs by saying, "Spin up this virtual machine, fuzz the heck out of it, and show me where the things are breaking." I mean, you can do some fun things with these things. It's And then let it run for a day. On a local machine. Do not let it out to the world. Cuz that's illegal. [44:30] Yeah, one thing on the pushing back very strongly on people doing these or by LLM obviously, LLM-generated stuff. I get an awful lot of stuff which is um I've It's very, very clearly, "I've used this LLM to get audio working on my laptop." And for those people, I like that sort of stuff, I'm not going to push It's like you clearly did you clearly did a bunch of work and you were sitting there running the the tests what the LLM told you to do to run. But I'm not going to push back so strongly on people using it in that sense, cuz I suspect most of them would have struggled to build the kernel in the first place without sitting there with the the tool. And while I might have opinions on how much they're learning and so on and so forth, at the end of the day, they have the laptop, the audio is working. Yeah, it's fine. Take the good fixes. We are taking these fixes, and that's Yeah, but it's some of the comments people have been making have been like, you know, just uh obvious um LLM stuff just don't even look at it. But I just like for um edge stuff on the edge like that with the driver, the quirks, I'm like if it was core code, different question. But if for some something on on the edge like that where it's like like I got audio working on my laptop, um I'm seeing so many more patches for that, and um I have a hard time being upset about it. [45:53] That's fine. And but the interesting trend is the number of patches landing on Linus' tree, the number that are actual bug fixes are going up. Yeah, yeah. Overall. Our overall percentage has gone up a small percentage, from what, 9 to 10 maybe or 11 now patches an hour. But this number of going 33 an hour, that percentage has gone way up overall percentage-wise. And that's But that's okay. I mean, we'll fix these bugs. The fuzzers did the same thing. We don't have numbers when the fuzzers hit because we weren't tracking CVEs, but I imagine there was a big bump then, too. [46:27] Um, I have a question. You said the fuzzers were doing the same thing. Um, as here at Kernel Recipes, there were a few people complaining that the syzbot uh reports were ignored or nobody had the time to to look at it. Do you think we'll see the same thing happening with uh maybe in a few years with LLMs, where we have this long tail of things that no one wants to look at, and might or might not have a security impact? Yeah, they will. Yeah. I mean, I know I ignore syzbot reports. And if it's fine, I mean, it takes somebody to be responsible for that and care about it. I mean, I go in for them, I'm like, "Okay, syzbot is doing something really stupid as root. I don't care." Or it isn't. But, I mean, it's up to you as a maintainer to see what you want to prioritize or not. But now people are feeding syzbot reports into LLMs and sending out a patch, and whether that patch is right or not, I mean, that's they're going through the USB gadget subsystem like crazy right now. Getting a whole bunch of new contributions, people that never run this code. But some of these fixes are good. And I'll take them, cuz they're actually help make Android more secure. So, it's good overall, but it's a lot it's put a lot of additional burden on us as maintainers. And I want to call that out, and we do know that's happening. It should hopefully end eventually. Um, but everybody all these companies owes us a lot of liquor at the end of this. Or whatever you want to drink. [47:59] Um, hi. So, recently we did a Summer of Code from a PhD thesis. It was very easy for anybody who wanted to apply to just feed in the thesis and, um, I don't know, some relevant patches and produce stuff. So, one thing that really worked out for us, and I think it because we are doing open source, we already have assisted-by tags. How about we start asking for accountability of AI usage? Uh, we For this project, we introduced confidence scores, like 1, 2, and 3. Three means you tested it, two means you have read the documentation, one is you just read the output of the whatever the AI bot generated for you. Um, and a publicly inspectable chat history of what was your use of AI. Two those of those two things when we enforced, uh usage changed drastically, and we had much more uh confidence in people who are participating. So, it it's working quite well for this project, and it could be something to consider for the kernel, also. [49:01] Yeah, that that's good. I I'll call out our scale is unlike anybody else's, and the amount of patches and work we have to do. So we have to trust these things coming. We can just finally get people to say, "Assisted-by an LLM." And I'll take that. So, I mean, we have four 5,000 developers. How many new developers showing up every week now? We are on a different scale of type of stuff, so I don't want to review somebody's chat logs of that type of stuff. I will look at the code itself. You You don't have to review, but just the person posting knows that someone can look at it. It It Yeah, but I mean, we're getting people not even knowing that they need to add an assisted-by, unfortunately. I mean, let's do baby step. I agree. That would be wonderful, and if you can enforce that in your subsystem, please do. MM might want to do that, for whatever you guys are doing. But making a blanket statement like that to 700 maintainers is going to be hard. We can't All of us, kernel maintainers, developers, can't agree on anything except we want to see see Linux succeed. We can't agree on licenses. We can't agree on anything else. That's all we can agree on is one thing. Let's stick to that. Thanks. [50:18] I was wondering if for the rc we are seeing the fact that people are less testing the rc than the releases. In fact, do we have this bias in uh No. Are people testing the rcs less than releases? I I don't know. I mean, we do Linux-next testing, we do a lot of rc testing. That's I don't think our development model has changed much. Um, I'm seeing the number of patches landing in stable trees are huge, and people are freaking out about it. But if you still look at the intersection of what you actually build and care about on your device versus what's actually being affected, it's still very small. So like I know for a specific for an Android device, out of all those patches that land in stable trees every week, 10 of them apply to your tree, to your device, which is a small small percentage. So, I mean, there's a long tail of drivers out there that aren't really used much, and that's where majority of this stuff is hitting. So, yeah. I don't know about testing, but [51:14] It might be too soon, but like, I'm curious your perspective on productivity. The dust still has to really settle, uh but do you think we are getting more productive? There are news like about companies actually getting less productive with AI agents, and there are two perspectives as well for productivity, because we clearly can see like CVEs or some bug fixes, although like we don't know how much they impact in the wild. But as you know, like in Android, there's lots of out-tree patches. So there are lots of folks they have problems out of tree, they need to be fixes. Are they actually being addressed? So like is this part of the volume that is generated, part of this? Because this is for me like where things really matters, because you are really helping helping products to ship with things that the kernel cannot handle today, like out of tree drivers, staging drivers. I haven't seen activities, but like against it's too soon, but I'm really curious your perspective on that part of the productivity. [52:07] I mean, productiv- measuring productivity is like measuring quality. You know it when you see it. It's hard. And people spend whole there's whole PhD tracks out there for trying to do that. I'm not going to say that. I do not feel more productive. I have to process more patches, but that doesn't make me feel better. I don't think that is making a good use of my time. Like Konstantin said, the easier the tools are to use, which he's making the tools really easy to use, and I'll call out I now use b4 prep and send. Um, I was shamed by my son telling me to do that. It's like, "Dad, here. You can make it send patches so much easier." Um, yeah, I still learn new tricks. Um, I can send patches easier now. Thanks to our tooling. Is that making me more productive or not? I don't know. That's not LLM at all, but I'm not going to make that call. That's marketing. That's a marketing spiel. Managers can measure. We're an open source project. I want to fix bugs, make Linux work. [53:05] Yeah, I was just wondering as a community really, do we see more features out and this kind of things out? I don't know. How do you do yeah, Yeah, like haven't seen that, but I was just curious in your take. No, we're seeing new features like the let's fix the build system in ways that, you know, these tools can run really well, and Miguel's going to talk a little bit about that in Rust and stuff like that. So, I mean, there's some cool things you can do with these. But ah, what is that Is that worth the trillions of dollars they spent on it? Hey, I don't know. We'll take it, though. [53:32] I would like to to add something. Uh we've seen in the past uh some developers who were uh a bit scared or at least embarrassed by uh their first security report. And I'm pretty sure that nowadays, since we documented this, some uh developers are getting them directly without the security team being involved, so as to reduce our load, and that that's great. However, if they are embarrassed or whatever, if they don't know how to uh to handle uh the bad quality or if they uh if they need help, they definitely need to uh to seek help from the security team. And that's something that we did not document, and I I think we should do, Yeah. because uh uh these models are attacking uh any subsystem. In the past, there were always the same which were uh tracked by uh uh security researchers, but it's not the case anymore. They are looking everywhere, and I'm pretty sure that a lot of just simple drivers or whatever with a maintainer uh working uh two weekends a year will get a report without uh knowing how to handle it. It's true. Yeah, ask for help. Be Thanks to the fact that we're spreading it out more, the people on security alias seem to be doing less work, which is good. We have other work to do. Um, and we have a full-time person helping us out, too, so we are there to help you guys out if you need help. Um, please let us know. But I'll call out the networking guys. The networking people have been flooded with this stuff, and I it seems like it's slowing down. So, it's nice to see. Um, I don't know. Let's see what happens. Last question. Last question. Sorry, I went way over. I'm going into your break. [55:09] So, I'm not worried about the number of bugs being found, I'm worried about the number of bugs being introduced now. Uh, I think a useful metric might be how many bugs have been fixed in one release that were introduced in the previous release, or So we have we have those graphs. Jon has that graph. Um, I've graphed it. It's pretty flat. That's why I said 5.15's an outlier because of the SMB server. Um, it's a nice curve, and we have these numbers. Kees Cook gives a talk on that stuff. I have not seen an increase in the number of bugs being added yet, but it's I mean, again, our code base is so big, it's going to take a little while history to see this stuff. It should be dropping for sashiko, but sashiko is going like, "Oh, and there's these old bugs in this file, too." And so we go through and fix this stuff. [55:57] You said we've seen the graph, do you mean the graph from Jon? Yep. But thing is that includes all the bugs since forever, but if it would be interesting to know all the bugs that have been fixed in like 7.1 that were introduced in 7.0 exactly, not before. Oh, we have I mean, that's in his graph. Is it? Yeah. Oh, I missed it. And it's flat. Okay. It's an exponential curve. I mean, we always fix newer bugs. There's a curve, and there's a bump at the end, right? Yeah, I mean, you've given lots of talks. And Kees Cook has also done a big one about that, too. Yeah, talk to Jon. So, anyway. Thank you for your Thank you very much. And thanks for the plushie.