0:00
Good morning everyone. Um My name is Dirk Hohndel. I have been doing Linux and open source and a lot of other things for a very long time and I always enjoy the opportunity to talk to this guy.
0:15
Yes, I'm I'm Linus and I don't do public speaking anymore and haven't for a long time. So that's why we keep on doing these Q&A fireside chat kind of sessions. And this is actually number 30. So we are we are celebrating our own little anniversary here today. Uh lovely weather in Minnesota.
0:34
Uh Sunday night's slide show was very impressive. I was super lucky. I made it to the hotel right before it started. So I enjoyed that. Uh the other thing we always hear about Minnesota nice, how how nice people are here.
0:51
I learned that yesterday morning crossing the street two blocks from here on a walk signal at the light and with a daring leap uh managing by about this much not to be killed by the SUV driver. So fantastic. I love this place.
1:06
Uh >> [laughter] >> Uh on on that happy note, I I need to hide after this presentation so the one uh twice again. So uh Let's do something different. We've done this so many times. Oh.
1:21
>> [laughter] >> So um I know that you're a huge fan of 3D printing. We actually happen to have the same 3D printer. Yeah. And one of the things that is interesting in this technology space is that basically everything is open source. The whole software stack, everything is built on open source.
1:37
Those who follow this stuff know that there's currently a bit of a curfuffle about the AGPLv3 and what it might mean.
1:53
But what I found interesting that in this in this super bleeding edge constant in new development space, everything is done in open source. All the development happens there. And the tooling that we use is is typically in open source. And 3D modeling is is a big part of doing 3D printing. And I learned from you that we both don't like the visual tools.
2:10
We both like tools where you actually write program code. So, OpenSCAD, have you actually used the code? Have you worked on the project or you're just using it? >> I'm just a user.
2:25
Um I don't know how many people here actually do things like 3D printing, but >> Show of hands. Oh, yeah. Actually, a fair amount. And and the same people among you use these nice visual tools where you can see what you're doing.
2:41
But when you come from a programming back background like me, I actually like describing things in text and treating 3D printing as programming.
2:56
And uh And yes, the all the software is open source, but at the same time, this is something where I like doing the the toys and tinkering and having something physical, which obviously my normal work is not. Uh but when I do coding, I like to be really close to the hardware and and work at a different level.
3:13
So, I use all these other open source tools without actually ever wanting to get involved in that project because it's doing something that is so different from what I'm used to coding. Yeah. And and so, for me, it's it's always this the this fascination.
3:32
Oh, this thing doesn't work. I want to fix it. And then I look at the code and I realize I have absolutely no idea how I would fix that. So, it's it's a wonderful learning experience that there are a million more things I know nothing about, but still enjoy using.
3:47
But what is really interesting to me is that for the longest time, we all have said, you know, the open source tools are kind of maybe a little clunkier and the proprietary tools are so much nicer. And the UIs are actually really cool. I I love the the quality of tools that you get there.
4:03
You talk about the the 3D modelers. They are I believe in it. I think we're actually past the point where people think that open source is just for engineers and and we largely kind of gotten over the hump.
4:18
And I mean just look at all the people who now talk about Linux in the gaming space, too. Yeah. I'm not going to say this is the year of the Linux desktop because that that has been said before.
4:34
And that is an inside joke because I actually did this to the press in 1998.
4:49
But But But I I do think that open source obviously has been long accepted in the commercial like serious space.
5:04
And but at the same time my argument has always been that software is very complicated and the only really good way to manage the complexity of of a complex infrastructure is open source and having everybody be able to be involved. And then most people don't actually want to be involved like like me and 3D printing software.
5:20
It's not that I am going to be involved in every project I use. But by virtue of it being open source, the people who do want to get involved can regardless of what company and what country you happen to live in.
5:36
But so we often talk about your non mainstream projects and I have been joking for a long time that Linus has done not two projects like Jim always says Linux and Git. He's actually done three.
5:51
Linux, Git, and Subsurface, a dive lock program that these days I maintain. But it turns out as of a few weeks ago, Linus has done four. Guitar pedal is on Linus's GitHub, [clears throat] and you can actually play with a guitar effects pedal written by the one and only Linus Torvalds.
6:07
Well, be careful though. Before you start playing with it, you have to manufacture it. All the manufacturing files are also open source.
6:25
So, you can you can choose to actually send the design files over to your favorite PCB manufacturer. You could try to place the components by hand, which I would suggest you not do because it's pretty fiddly.
6:41
Uh I started out doing it that way, but then I decided no, for fancier projects, you actually want to have the professionals place the components for you to. But yeah, if you if you're into guitar pedals or in into music and want a really bad guitar pedal, you can now make one yourself.
6:58
And it comes with both the hardware and software design files and I I am one of the better tester of a pre-version of it, and it's not bad. It's not bad. And of course there is a 3D printed uh housing for it and everything. It's really fun. So, it's github.com/torvalds/guitarpedal. Check it out.
7:13
>> I will change the world of music, too. Yes. I am certain you will.
7:30
So, talking about hobby projects, uh a little while ago you started another hobby project, nothing big and professional like GNU, but something that somehow here took off, and recently you released last Sunday you released 7.1 RC4 of the Linux kernel. Uh my usual question, what's going on? How are we doing?
7:45
Um I've been saying that we've been doing the same thing now for I mean, I've been doing it for 35 years, right?
8:00
But, we have actually had the same process now in place for pretty much exactly 20 years where when we switched over to get and then it took a while to kind of get the whole modern release process going. Uh so, for 20 years we've done the same old thing. And I used to say that it's all working fine and it's smooth sailing and it's steady progress.
8:15
And then about half a year ago, uh things changed. Um We've in the last 6 months we've seen a lot more commits, small details uh just doing the numbers.
8:34
Um I think the last two releases it's been about 20% more commits than we had in the previous releases over many years.
8:50
And at first I thought, "Hey, people are excited about the 7.0 release because I changed the major number every once in a while." And I thought, "Hey, we saw that with 6.0 that a lot of companies wanted to get their stuff in for for the .0 release." And uh it turns out I was wrong.
9:05
The real change that happened in the last 6 months was uh the AI tools actually got good enough for a lot of people that we're seeing a definite definite uptick in in just development on pretty much all fronts.
9:20
And we talked about this uh 6 months ago in Tokyo that uh the tooling actually lowers this initial barrier.
9:37
Your your barrier of entry to to writing a Linux kernel patch, the tooling does a big chunk of the work and that certainly has had an impact and it's had not only a positive impact. So in the announcement of RC4, you announced the new AI as the security disclosure guidelines with a special point of view on AI. Yeah.
9:54
So again, I have a hard time seeing everybody because I have the lights in my face. But how many people here use actually AI for coding? Yeah. >> Pretty much everybody.
10:09
>> so I have a love-hate relationship with AI. I actually really like it from a technical angle. I love the tools.
10:24
I find it very useful and interesting, but it is definitely causing pain points and and the big pain points in in Linux traditionally and I suspect in most projects have not been so much the code itself, but has been when you're forced to change how you work and people get into a rut.
10:45
People know this is this is how I work and and this is how I like working.
11:00
And then something comes along and for me it was like long time ago around 2000, I had to change how I work because I did not scale anymore for the project as Linux was growing and that was I still remember it being one of the more painful episodes in kernel development and it was literally 25 years ago.
11:15
Uh and I think we're seeing some of the same effects now with AI where it kind of forces people to get out of their comfort zone.
11:32
So we've had a lot of social issues with the AI-generated bug reports, but also code. And and there's uh it's not a Linus doesn't scale like it was 25 years ago.
11:48
Now, it's people don't necessarily scale, and it takes a while to learn how to use the tools in a maybe responsible isn't the right word, but in a way that actually works with the community and works with the other developers.
12:03
And we are definitely seeing some of the point pain. One of the pain points recently has been the security mailing list where when people find bugs with AI, a lot of the time I mean, what a lot of bugs can be security issues.
12:21
And and I've said for years, decades that a security bug is a bug, but the other side of that is a bug is often a security bug, and you may not even realize it.
12:37
And and people think that when they find a bug with AI the first reaction seems to sometimes be, "Let's send it to the security list because this may have security implications." And the security list for the kernel was overrun by duplicate reports by tens, hundreds of people running AI often the same AI tool and getting slightly different results.
13:02
Often with the same underlying cause, but we were flooded by people sending sending bugs. And then you have this list with fairly few people on it because it's all supposed to be super secret.
13:17
And and we spent all our time just forwarding these reports to to other the other developers who knew that area better.
13:33
And we made the policy change basically if you find a security or any bug with AI you should basically consider it to be public. And uh just because if you found it with AI 100 other people also found it with AI.
13:48
Uh and obviously that when it comes to things that really are security issues you may not want to make the exploit public, right? Don't go and say to the world, "Hey, this is how And this By the way, I want to make it very clear. This is not just about the Linux kernel.
14:03
When you find an exploit for any piece of software don't be that guy or that gal who then crows about it publicly and say, "Look, hey, I could bring down uh this big company or whatever." Uh just let people know what the problem was and don't necessarily tell them exactly how to make somebody's life really miserable on a Friday afternoon.
14:29
Right?
14:45
[laughter] Uh so we are seeing some pain points with AI and uh and uh it's not that the technology is bad and I I've personally really enjoy it but but there's like the social issues that it causes uh have yet to be all worked out, I would say. A- And social issues I think is the key here. We have had this tendency for a long time.
15:02
I I don't know if Heartbleed was the first one, but the companies who are branding the bug and create the domain and the logo and they want to get all the fame for the bug and then release it before they ever talk to the victims, i.e. the maintainers who then need to fix that. And we've had four recent local uh privilege escalation bugs in the Linux kernel. Yeah.
15:18
of which were disclosed exactly this way. My response is always, here is a company I never want to work with because if you do that to the Linux kernel, you do this to anyone. Don't want to work with you.
15:34
However, we've had these four bugs, and I think they have really brought out the challenge that you just talked about, that between the Linux kernel developers learning about the the local privilege escalation and the world learning about it, there is a difference of zero minutes. So, you have no opportunity to fix this.
15:52
You have no opportunity to get patches ready. You have no opportunity to actually help the people who run the code. And so, you said you you want all AI bugs to be public, which I totally understand the logic of.
16:08
But can you explain a little more how that doesn't create the situation that the the maintainers are always on the back foot? Um I'm sadly of the opinion that we can't get around it. Um we had a small bug.
16:25
This was not a huge huge issue, but any project in the Linux kernel, we have I forget, 35 million lines of code. We have a lot of code. And and that means that we do have bugs. And uh they they're like daily occurrences.
16:46
This is not something shocking, and this should not really surprise anybody.
17:01
And uh it used to be that we found bugs that were security issues, and we would let people like distributions know that, hey, you really need to upgrade, but we would not describe exactly what the security issue was. But being open source obviously meant that the fixes were all very public. But they were hard quite often subtle security issues that really hard to figure out.
17:17
Uh in the time of when you have AI you can just automate the figuring it out part. So, it used to be that uh we'd fix a bug. Um we wouldn't tell people exactly what the problem was.
17:34
We'd just tell them, "Hey, we fixed a bug. Please upgrade." Uh most of the time nobody would figure out what happened. Uh last week we fixed the bug.
17:51
Uh within 3 hours there was a blog post up about uh the implications of that bug fix because security people love getting attention. They're they really it's I mean, don't get me wrong. We all do.
18:09
But but they often have their business to they that's the way they make a name for themselves is they talk about security issues. And if you're the first one to talk about it, it's news. So, you get more more eyes on your blog posts.
18:24
So, there's a very real incentive to be the first one and to run all these AI tools and to make a big splashy announcement about how bugs happen. And I think this is just this is how it's going to be. Yeah. And there is no nothing you really can do.
18:40
And I don't think, for example, that the solution is to not do open source because if you think that AI can't do reverse engineering of closed source, you're in for a surprise. >> Yes.
18:56
Uh I think closed source is even worse in this respect because the AI can't help you fix the problems, but the AI sure can help find those problems in the first place.
19:12
You just said something very important because one of the things that I've noticed, especially with the more splashy ones in the last few weeks, is that these companies enjoy spending a lot of money and a lot of tokens on pointing out a bug, and strangely none of these came with a patch, even though all of these were in open source code.
19:27
So, to me there is we're going back to the social responsibility and the it's kind of sad that you get all the attention from the from the journalists for finding a bug, and you don't think you could get even more attention. Hey, any journalists in the room?
19:43
Uh you could get even more attention for then fixing the bug. And so To be fair, sometimes it is easier to find than to fix. So, uh I understand. And And don't get me wrong.
19:59
I'm This may sound negative to a lot of people, but I actually I I think it's all very good. When AI finds a bug in any source code, uh that is a short-term pain.
20:16
But long-term is you found a bug. We fixed it. The The end result is better for it. So, I'm actually This is part of why I actually really enjoy AI. And again, I want to stress that the conflict is not that AI is bad.
20:33
The conflict is then that there are some social choke points and social pain points that that come with this new tool that we have 35 years of code that it sometimes finds issues that we did not find.
20:48
And And it will it's going to take us some time to to just work through the the new issues that are found. And So, I'm actually very positive about this whole thing. I think finding bugs is great because the real problem is all the bugs you didn't find.
21:05
So, we we talked backstage about the impact all of this has on maintainers because the maintainers were already struggling with burnout before and now this flood of bug reports isn't really helping.
21:20
But are there any good tools that help you as a maintainer with code review, with with understanding the patches that come in to help you with your workflow?
21:36
We have a I mean we have a ton of tools for that and I I'd like to point out that the Linux kernel is actually doing really well.
21:53
We have every single release, we have over a thousand people involved and we have a solid cotter of maintainers that are there and who get most of the time they get well paid for being there, too. And so we're doing well. Uh all these problems that I've been talking about, I've been talking about the Linux kernel because that's what I do, obviously.
22:09
But think of all the tens of thousands of random projects that people maintain that are not the Linux kernel.
22:24
And that maybe somebody's like pet project that they've been working on for a decade or more and they have one person or three people that are somewhat involved and they get really burnt out when they get a flood of these of sometimes obviously AI reports that just it's a bug report and when you ask for more information, the person has done a drive-by and and doesn't even answer your questions anymore.
22:50
So, so that's the the real burnout issue is is I think there. Uh and now I forgot what the actual question was after [laughter] that aside.
23:06
I asked you if there are AI tools that I have to >> Oh yeah, we we do use AI tools and and uh I'd like to point out again, I actually work mostly with people. So for me as a top-level maintainer, I don't do a lot of coding. I do coding on my toy projects, on my guitar pedal, on my like I do coding for fun because I'm a programmer.
23:24
But my job is working with people and I do not use AI to work with people. >> Thank you. >> And I I should suggest you don't do that either. But we do have a lot of automated tools including AI.
23:40
We have uh like Google for example set up this patch tool thing that a lot of people are looking at. I forget the name, Shashiko?
23:55
Uh that looks at all the patches that go to the mailing list and does a review of them. And sometimes the review is not great.
24:11
Uh but quite often it finds issues and it asks questions and says, "Hey, what about this issue?" Uh and that's the public one and there's a lot of companies working on private tooling for inside inside companies, but actually these days a lot of the main developers uh do local AI which I actually would suggest uh everybody look into.
24:35
You don't want to be entirely at the mercy of of the big companies that at some point decide, "Oh, we need to make money, too." Yeah. Uh So you're showing your age because your your comment about tens of thousands of projects is really funny.
24:50
This week GitHub announced that they currently have 480 million repositories. And um in a recent study, 600,000 projects were considered of uh important or critical use for enterprise users of open source software.
25:07
So, it's literally hundreds of thousands and hundreds of millions overall. The the the space is so big, it's it's hard to wrap your mind around. But let's you say this is a social issue and you talk about working with people.
25:23
If you are at the beginning of your career right now, >> Yep. and a lot of people do these doom and gloom all code will be written by AI.
25:40
And on the flip side, those of you who were here on Monday for Jim's keynote, saw Jim's very optimistic note that said actually we are creating a lot of jobs in the tech industry and we're seeing hiring in the tech industry. So, do you have something that you would tell someone at the beginning of their career, here is where you should focus?
25:56
So, my opinion has always been that hey, AI is a great tool, but it's a tool. And and when I see people saying hey, 99% of our code is written by AI, I literally get angry.
26:13
Because those same people, I pretty much guarantee that 100% of their code is written by compilers. But they never say that, right? It's [cheering] it's exactly the same issue. Uh I grew up writing machine code.
26:30
And when I say machine code, I don't mean assembly language. I mean the numbers. And I mean, it leaves an imprint on you. I I >> [laughter] >> I can I still remember LDA is uh hex A9 on the 6502, right?
26:49
>> Correct. >> and I it took me a while to understand that writing down the numbers and calculating offsets for branches is kind of stupid. And people had come up with this tool called an assembler.
27:05
Uh and then later on I figured out compilers are good, too. And these days I'm figuring out AI tools are good, too. I'm still writing the code. I'm just not doing it the same way I did when I was basically typing in numbers in in data statements.
27:26
Uh and the I I'm personally 100% convinced that AI is changing programming, but it's not changing the fundamentals. Exactly the same way that you all use compilers to actually generate your code.
27:42
You will all use well, not maybe all of you, but a lot of people will use AI AI to generate the code that the compilers use to generate the code that the assemblers then use to generate the machine code.
28:00
This is revolutionary in the same sense that we've seen revolutions before. And AI will increase your productivity by a factor of 10. And I claim that compilers increase your productivity by a factor of 1,000.
28:15
So, AI is great, but AI is not changing programming. It may be changing other areas, don't get me wrong, but I'm a programmer, so I don't care.
28:30
Uh >> [laughter] >> And and one of the things Sorry, my voice is already fading. One of the things that I think has always been true when you look at the code written by developers. Uh good developers write good code. Other developers write code that then breaks.
28:46
And the same will be true for the people prompting the AI tool. People who know what they're doing, who understand systems, will be able to prompt tools to write good code. And people who don't understand the complexity of systems will also prompt systems and write code, and it will fail.
29:02
And I think you do want to understand how it all works in the end. I still like I don't program in machine code anymore. But I still look at the generated code.
29:19
So, when I use a compiler, even when I use AI for my pet toy projects, I will use AI to generate code, I will look at that code, I will actually still look at the assembly language end result because it's what I grew up with, it's kind of where my comfort zone.
29:36
And right [snorts] now I'm playing with microcontrollers anyway, so you really want to actually look at at the generated code and make sure it's it's doing what you really want it to do.
29:53
So, I think even when you use AI for coding, if you do a project that you actually maintain long-term, you need to understand not just your prompts, but you need to understand the end result, too, because that's the only way you can maintain it long-term.
30:08
And it's easy enough, and we see that all the time, where you uh [snorts] basically generate a new project, and it's a throwaway project, and it's a one one-time and it's over. And uh AI is great for that, and we call it vibe coding.
30:24
But if you want to make something serious, you're going to have to maintain it for 35 years or something. And and at that point uh it's it's a lot more than just writing the prompts to make somebody else generate the code. Well, unfortunately, we're out of time. I have a lot more questions.
30:40
I guess I'll have to ask them next year in Vancouver when we'll do the same session. I hope you'll all be there. Thank you very much. Thank you.
