>> Hi everybody.
Since we started a little bit late, Iím going to keep this nice and short unlike our guest,
Greg, who is quite tall.
And--yeah.
This actually--it rose out of a discussion we were having between some Kernel folks that
I know, and Bryan [INDISTINCT] and right over here, about sort of the nature of the embedded
system--what's that?
>> It might not [INDISTINCT] >> Well, no, I mean, like from outside the
company, inside the company like--donít--I'm speaking here.
All right?
>> Go ahead.
>> Anyway, and so, you know, at the end of this today, I want you to come out and speak.
It would be fun for us and fun for you and stuff.
So, here we are, Greg.
>> HARTMAN: Thanks.
Is this working?
All right.
I'm Greg Kroah Hartman, Iím a Linux Kernel developer.
Iím a sub-system maintainer of the Kernel for USB, driver Core, I used to do PCI, I
like to see all the weird things.
I'm one half of the stable Kernel maintainer group which I'll talk about later.
So here, we're going to talk about the Kernel, the process, who's doing it, how itís being
done, why you should care and why Google should care and things like that.
Please tackle--ask questions in the middle, I like it.
Make fun of me.
I'm going to make fun of you.
So, beat them.
So, first thing.
This is the most important thing, donít get anything.
Realize this.
This is what the Linux Kernel do for the past year and a half from the 2.6.20 to 2.6.25
Kernel.
This is what was changed per day in the Kernel.
This is a stable Kernel.
This is a stable development process.
This is freaking insane.
It is the fastest moving so--project--so--software project known to anybody.
Largest project of its kind known to anybody but it's also created a result thatís never
been done before and we have an operating system that supports more devices than any
of our operating system ever has.
We support more processors than any other operating system ever has in the history of
computing.
And we're doing it in an open manner, in a very chaotic manner.
Moving at a rate, that fast; shipping something thatís stable, that everybody, including
yourself, uses everyday.
We broke all the software development rules and we're continuing to break it.
If that doesnít make you guys scared, it should.
The biggest thing to that make you guys--should make you guys scared is, if you keep your
code outside the Kernel, thatís going to pass you by.
Do you think you can keep up with that kind of rate of change, figure out what's going
on with that rate of change?
If you get your code into the Kernel, that rate of change will happen to you automatically.
If you donít, it will blow you by.
So, get your code in the Kernel.
It's the main thing I want to talk about.
So, thatís it.
Thatís crazy.
And also, this is the first question people ask like, "Oh, thatís jus drivers, right?
Drivers are changing that fast.î No.
It's the whole Kernel.
I looked at it, Iíve broke it down by the proportion of the code--size of the Kernel.
The Kernel is about nine million lines of code.
The Core--the Kernelís five percent, driverís a little over 50%, 25%.
If you look at the rate of change at each individual portion, it's exactly proportion--itís
exactly proportional to the amount of code thatís in the Kernel for that section.
So, the Core of the Kernel is changing at 5% of that rate.
The driver section is changing at 55% of that rate.
The architecture specific spot is going up--what is that?
I donít know, it's going at 30% of that rate.
So, it's uniform.
It's flat across the whole thing.
So, donít think that just--that the drivers are changing, it's the whole thing; it's everything.
So, and along with that rate of change, it breaks down to this.
It's going really fast.
Three point sixty nine changes an hour, 24 hours a day, seven days a week.
During the 2.5, the 2.6 development cycle, we're running about 2.5 changes an hour.
And that was unheard of; no way that Iíd ever thought we could do that.
Since then, we've been going up.
I donít know why.
We're getting faster.
We're doing more rate of change and it's getting bigger.
So, this is some of itís staple.
So, here's our graph.
If you want to see all the--that lines of code that are added and modified, diluted
over time, I just did for the 26 [INDISTINCT] and long.
If you look--if you used that, it was a little bit less.
So, it's--itís really--itís kind of staying around the same but at lines out, it keeps
going up.
Thatís interesting.
Thatís not really trend worthy but it is continuously going up.
But itís trend worthy is the changes per hour.
It's continuing to go up.
2.6.25 changes per hour is amazing . Well, that rate of change does not included moving
files around.
Moving files around the Kernel doesnít affect it at all.
It does not change at all.
So, we're going faster, doing more rate of change per hour per release.
Everybody happy?
Nobodyís questioning that?
Oh, come on.
All right.
So, how we do this?
Hereís how we do this.
Extensively, it starts off at the developer on the bottom to make a change to a file,
to--they find a bug, they want to add a new driver, do something.
Then it takes that change, the path, should they send it off to the maintainer or the
driver of the file.
In the Kernel, we have about 600 unique maintainers that are listed on who owns what.
You canít find what--who owns what, you can look in the file using git and see who modified
it last.
Whoever touched the floppy driver last is the maintainer.
Never touch the floppy driver.
So, that personís responsible for looking at it and then they see it, they review it
and then they sign off by it.
They say, ìThis looks good to me.
I think itís okay.î And they add their signature called Sign Off.
And Iíll talk a little bit more about that later.
So, the developer says, the sign off, itís good.
They say itís good.
They send it off on to the subsystem maintainer.
Subsystem maintainer like USB, PCI, BFS, security, Core, things like that.
They consolidate all the patches from all data people and then they hold on to them.
They create this little mini trees everywhere.
These little mini trees are either in gits, using quilt, somebody--I think thatís all
everybody uses, some tarballs maybe.
And then all that, every night gets pulled into the Linux-next Tree.
Itís from Steven Rothwell in Australia.
Which--for IBM, the Aus labs.
He pulls it in every night, the Linux-next Tree merges them all together and sees if
it builds or not.
Sends you angry emails or you break the bill or you break the merge, itís great.
Itís awesome.
Itís testing all the subsystem maintainer stuff for all our development things.
Andrew at the same is creating--is now relying on Linux next and sucking everything in the
host tree, about once a week or so.
And then also, he pulls things from the Linux Kernel Mailing List, a sort of mailing list
that happen to get drop by subsystem maintainers or whatnot, he gets--he sucks those under
his tree and everybody--and then he does a release every couple of weeks.
So thatís testing, this is the developmental stuff.
So all the crazy stuff the subsystem maintainers are holding on to, all the stuff thatís coming
across the mailing list but Andrew picks it up, gets pulled to these releases.
Then when Linus says itís the merge tree--windows open.
Iíll talk more about that later.
All the subsystem maintainers hit Linus.
We got two weeks to hit Linus with all the changes you want.
We all go to Linus.
Andrew sends a bunch of stuff to Linus after the subsystem maintainers get there and then
we go on, the cycle releases.
Iíll talk more about releases later.
So, nice little tree structure.
So, Linus and Andrew and the next tree.
Does that make sense?
Nice hierarchy, right?
Nice and pretty.
So, Iíd try to graph this.
Looking at the relationship between people and the work--way patches flow to the tree.
I think itíd be a nice pyramid or whatnot.
It turned out, it isnít.
I made a graph that was about 40 feet long and 4 feet tall, of one Kernel release.
It was a mess.
Itís a big giant spaghetti mess.
It looks like an internet routing system.
And which is good because it means that, if this person doesnít pay attention, we route
the packets around them, Andrew picks it up and then sends it to subsystem maintainer,
it sends it to Linus or this person sends it directly to there and goes around the maintainer
or this person disappears for a week.
So, itís very good.
Itís lossy.
We can handle people disappearing, handle people come--reappearing and it works out
really well.
It isnít a strict hierarchical development system which is good and it shows how we can
keep up this massive rate of change so fast.
Itís kind of scary.
If you own a file or a driver, you need to accept the fact that other people are going
to be changing it, extensively, without you knowing it but hopefully you should be paying
attention and you just have to accept that so other people can do that.
You can always change it back if you didnít like it.
But thatís the way we get things done.
Thatís how we can work so fast.
Make sense?
Cool.
So, hereís a release cycle.
So, Linus does the release based on something.
We say itís good.
So then, we have two weeks.
We can throw anything we want at Linus.
All the subsystem maintainers throw everything we want.
Extensively, itís been tested in the Linux-next Tree, before that I have been testing the
MM3 from Andrew.
So, we hopefully expect that the code has been tested, it builds, it hopefully compiles
sometimes and all merges.
So, thatís when the hundreds of thousands of patches go in at once, two weeks.
Linux releases an RC1 Kernel.
Then the subsystem maintainers that--were on vacation or forgot about the merge window
being open, wake up and say, ìOh, no.
I forgot my stuff.î They send it to Linus again so, then goes an rc2 a couple of weeks
later.
Hopefully, thatís getting better.
People are slowing down.
Everything happens between rc1 and rc2.
Then, its bug fixes.
We start doing bug fixes, we track bug fixes, regressions, we do RC1, 3, 4, 5, whatever.
Until we think things are slowing down and good enough and boom when it says, ìOkay.
This oneís good.
Out the door.î And then we start the whole cycle over again.
There is no stable and unstable development trees anymore.
You step odd was unstable and development even was stable, not anymore.
For the past four years, weíve been doing this model.
When we finally realize that this model really worked good, we said, we should try and do
release every two to three months, right?
If you look, weíre releasing on an average of every two and three quarter months, for
the past three to four years, like Clockwork, you can trust that a new Kernel will come
out between two and three months.
This is a really, really good thing.
Unless the up stream just rose, figure out in their schedules when things are going to
be done within a--within a reasonable time frame that can plan when things are going
to come out.
The [INDISTINCT] can figure that out, developers can figure that out, ìI miss this merge window,
whenís the next one going to come around?î You don't have to worry that their selves
are going to get drop for forever.
It works really, really well.
One thing that happened though is we realized during these two to three month time, what's
happens about security bugs?
These are people that actually use the Kernel.org--Kernel.
What happens then?
Thereís bugs, security problems happen, what are we going to do?
So, we formed--two people formed the stable tree which is me and Chris Wright who works
for Red Hat and we do security updates or bug fixes; 0.1, 0.2, 0.3, 0.4, 0.5 and we
have a real strict list of rules of what we will accept, what we won't accept.
It's pretty easy.
It has to be bug fixes only and security fixes.
The patches have to be in the upstream tree already, unless thereís some really, really
odd thing, things like that.
It must be small, obvious.
We do those releases every couple of weeks, whenever we get around to it.
Sometimes we're slacking, sometimes we're not.
Security updates, we can do it usually if we got an inbox.
We can do it in couple of hours.
So, we release things as soon as possible if we can.
And that's working out really, really good.
The fun things is, after the--we drop them, after 2.6.20 comes out, it might be one more
2.6.1.9 stable and then we stop.
We don't have to do it.
Weíll have to maintain that for forever.
You can maintain it for forever if you want to.
Somebody picked up the 2.6.1.6 Kernel, decided to maintain it on his own for a long, long
time.
We all thought he was crazy.
He got a job out of it.
So, maybe he wasnít so crazy at all.
So that he could do things over time.
So, it worked out really well.
If you want--if anybody wants the 2.6.2.4 Kernel to maintain it for a while, let me
know.
Weíll give it to you.
We don't care.
So, that's how it works.
If itís stable, releases happen, development and releases happen and then we drop stable
trees.
So, we don't have to maintain them for forever, which given our rated change previously is
a good thing to do.
Makes sense?
Everybody happy with that?
Is it working?
All right.
>> So, you guys don't have an open analysis [INDISTINCT] every blogger really preferred
at least [INDISTINCT] >> HARTMAN: We don't have a notion of a preferred
release anymore, no.
Every release is a preferred one, the latest one.
Based on the rate of change, if you report a bug--a bug problem on 0.2--on 0.2 Kernels
ago, it's given how fast Kernelís often changing.
We're going to ask you, can you reproduce this on the current version?
It's the first thing we're going to ask you.
So, yeah.
So, everythingís preferred.
The last one is preferred.
Now, note when the distros do releases, we know what Kernels are going to release.
So, sometimes those Kernels hang around a little bit longer.
Fedora and OpenSUSE are based on 2.6.2.5 right now.
Odds are that Kernelís going to hang around for a little while because Chris Wright works
Red Hat and I work for SUSE.
So, weíll probably want to hold on to that Kernel for a little while longer, make sure
bug fixes get in there and it's good.
But, if you want the new features and things like that, you should go to the latest one.
[INDISTINCT] Yeah.
>> Why use the two-week merge window, why not the [INDISTINCT]
>> HARTMAN: Why two-week window, merge [INDISTINCT] what?
>> Why accept merges all the time?
>> HARTMAN: Why Linus accepts merges all the time?
Because we want to be able to throw a bunch of things and we wanted to shake things out
and stabilize them.
We want you to test and feel good that we can--at the very end, we don't want to accept
the massive bunch of USB changes that nobody has really tested or run for the past three
months--three weeks or so, you know, we want to at least some kind of software--good engineering
practices.
It looks like it something, we know what weíre doing here.
Yeah.
>> Thereís never a period of where thereís no merging or [INDISTINCT] their merging over
to something newer and then the RC what windows, youíre shutting off with, is that correct?
>> HARTMAN: So thereís no...
>> [INDISTINCT] gaps where...
>> HARTMAN: Thereís no gaps where were not accepting any merges, yes.
Well, we think of, yeah.
When we--we use gits to send things to Linus.
So, he pulls from the--the subsystem maintainers trees and put--merges them in.
So, that's where we merge things but it should be bug fixes only.
From RC2 on, unless it's a really odd thing.
We should have bug fixes only are going in, into the merges.
I think of merging as a new feature, I'm sorry.
Itís a wrong terminology.
I use--you can use git to send this stuff to Linus.
Linus came here and talked about git.
I use quilt.
Quiltís really good too.
So, yes.
>> So, how do we test them?
>> HARTMAN: How do we test?
>> Yes.
>> HARTMAN: One thing about the Kernel is, there is no good way to test it except running
it.
You canít run like unit test--you can write unit test that extensively test some of the
external features, such as the system call interface, it hasn't broken some odd things.
But really it is testing--the operating system is running on your machine.
There are so many hundreds of thousands of permutations or devices and interactions with
timing and everything else that we wanted to run on all your machines.
So, we rely on you to test.
We rely on the developers to test and that's the only way we can obviously test.
Like--classic example is, I had a problem when you were--you were plugging in two USB
devices at once and yanking one out, the data was being streamed out too.
You canít unit test that unless I want to put--I have a big box of USB devices that
I do test a lot, but it doesn't make for an automated system.
So, we ask you to test it yourself.
We rely on the development community to test.
That's all we can do.
There isn't a same way to write a unit test for an [INDISTINCT] it just doesn't work because
we're relying on so many external inputs.
People keep trying and that's great and we want as many test.
We want as many functional tests as we can get.
But like performance--and we performance test also.
People are trying to test it but it's a real hard thing to do that.
>> So there is no formal--letís say, we have between--like, demos in the 2.6.25 and the
last one out, [INDISTINCT] they were waiting for someone to do something for it.
Is this a formal procedure or does somebody does [INDISTINCT] and what is something that
you [INDISTINCT] >> HARTMAN: What if--so, what was holding
it up and what was the formal procedure?
We do track regressions.
So, like people record a regression, ìHey, this worked on 2620 and 2.6.21 RC whatever,
itís broken.î Regressions retract.
So, towards the end of the release cycle we say, have we fixed all the regressions?
Have we fixed all the bugs that have been reported?
Some people are trying to track the bugs that have been reported.
If it isnít a regression and it's just as a bug that always been there or something
that new happens or for something that's been added that's brand new and newer support [INDISTINCT].
We don't really treat that as important as a regression from the--actually machines working.
We want to make sure we never break a machine that was working previously.
We ripped whole features out because we broke the machine that was working properly.
Which is tough, ACPI and some interaction and some low level platform code is nasty.
So, we change that and it breaks whole models of laptop, thereís the IBM series of laptops
that was notorious for breaking whenever we change anything in ACPI.
Itís tough.
So, we try and track those and that's what's keeping us, ìLook, we know that thereís
this one class of bugs that a bunch of people are hitting, letís try and fix that up before
we do get between RC 7 and the final release.
So, we do keep track of that.
So, if you have a problem, let us know.
So, as a developer and as a user, if you want to help Linus out, test these RC releases.
Test them and let us know if there's problem, you have regressions or not and they're good.
I mean, all these are run on everybody's machines.
You don't have to worry too much about these--these things.
I wouldnít run it on something thatís production wise but definitely run it on your local machines--around
on my laptop, all the other ones.
If you want to test what's--if you want to help us test what's going to happen next,
test the Linux-next Tree, test Andrew Mortonís tree.
That shows that crazy developmental stuff thatís still under heavy development that
were trying to merge in and figure out what is good enough to go into the next merge window.
We got features in there that aren't really good enough to go.
There might be a couple releases away, things like that.
What's the question?
Okay.
All right.
Move on.
Like I said, we're doing this every two and three quarter months.
So, we're actually doing what we said we were going to do and that's a good thing, that's
a time-base release cycle; itís turned out to be a really nice thing.
It works really well.
A lot of Open Source projects are kind of switching towards it, it works.
You know that all of [INDISTINCT] this feature--this window.
Two and three quarter--three months later, itíll get in.
It's a good thing to do.
9.2 million lines, we've been growing at a rate of 10% increase in code size per year
since I started tracking it back to 260.
It's a linear graph.
Itís just pretty much straight which is amazing.
We're still growing.
We're still getting bigger.
That's a pretty big chunk of code.
Now, not all machines run at 9.2 million lines of code, that's funding.
You write in the Core and a few drivers around it, not working and things like that.
So, that's everything.
That's all the different platforms, everything else.
And in the last year and a half, this is the number that I find the most impressive.
We had 2,399 unique contributors to the Kernel and that's amazing.
That's showing we're doing really, really well.
Now, half of these contributors only contributed one patch.
Half the half contributed two, half the half the half contributed three; well, exponential
curve there.
But the top of the curve is getting flatter.
When we did the 2.6, the 2.5 or 2.6, the top 20 people did 80--8%--80% of the work.
Now, the top 30 people did 30% of the work.
Just by quantity of patches.
So, we are flattening out and weíre spreading the load out a lot more evenly.
I looked--it was 200 people--the top 200 people, do they--no.
Two hundred people did 8%? Something--I was thinking of the top 200 people that did a
lot of work.
I was spreading a lot better.
So, that's good.
I mean, here's the--the backgroundís graph.
I did the crazy graph.
It's on kernel.org, you can try and load it in your browser.
It's a good SVG test.
It broke [INDISTINCT] machine, crashed Mackintosh, it only--I got somebody from HP to print it
out, he had printers in his barn.
Big roller printers in his barn, he printed out that.
You can go through him.
So, thatís our number of developers, thatís good.
And our number of individual developers, per Kernel release is going up also.
A number of companies that are actually sponsoring developers is also going up which is a really
good thing, showing more companies are getting involved as a spread of Linux goes bigger.
So, letís talk about people, quantity.
We can track people like quantity, this is just quantity not quality.
I'm not going to talk about quality.
So, this is the number of changes that got into the Kernel.
Top 10 people for the past year and half; number one, Adrian Bunk, there's a lot of
janitorial patches.
Al Viro of BFS.
Thomas Gleixner, the X86--no, i386 next 86 arch maintainer along with Ingo Molnar.
David Miller networking Core.
Bart, IDE.
Paul Mundt is SH.
Ralf Baechle is MIPS maintainer.
Ingo again, Patrick McHardy is net filter and Tejun is Lab ATA, disk drive stuff.
So, two of these drive guys are top 10.
Thatís quantity.
Thatís a number of changes that they got in the Kernel.
Yes, [INDISTINCT] So, signed off by.
Remember how I mentioned every person along the hierarchal tree signed off on it actually
reviewed it.
Here's the top signed-off, this shows who are the gate keepers and how--who's doing
what.
Andrew is the top.
Linus is next.
Now, the funny thing is, Andrew has more than Linus, so that everything goes to Linus.
Linus doesnít sign off on patches he pulls from sub-maintainers, we're using git because
he isnít looking at them.
Heís trusting those maintainers to do the thing.
So, I make up a git tree for Linus to it pull from.
He says, ìGreat.î He pull from it and he doesnít sign off on those.
That's why his number is lower than Andrew's.
So actual the real number patches that got in is less than the sum of those two, itís
kind of funny.
So again, David Miller in networking.
Jeff Grazik is in network drivers.
Ingo, architecture for Intel.
Me, Thomas are in Intel.
Mauro does video for Linux, little webcam stuff.
He does that in his own spare time.
Works for a telecom company in Brazil, awesome stuff.
Paul Mackerras is Power PC.
John Linville is wireless drivers.
So, you can kind of see what areas are the code, Kernel is doing a lot of activity.
So again, video for Linux is a very active area, wireless is a very active area.
Interesting.
So, let's see who is funding these companies.
So, we can see the people, receive the number of changes it does.
So, who's doing this work?
So again, for the past year and half, Red Hat, IBM, Novell, Intel, Oracle, Consultants,
these are individuals working for consultancy, they are being paid by other companies or
working for consulting companies themselves.
Academia, I'm very happy to see Academiaís actually going up, at numbers going up.
And Renesas, Renesas?
I think there an SH maintainer--developer.
Which is really interesting, they just pop in the top 10.
So, you might want to guess what's number one to five?
Come on.
Canonical?
I'll tell you where Canonical is later.
Well, I'll tell you where they are now.
They're about 300th, I think.
Canonical had six changes in the past five years.
Canonical does not give back to the community, that's what they do.
>> Linux foundation.
>> HARTMAN: Linux Foundation, no.
Linux Foundation is not on the list anymore because they only have one developer.
>> They [INDISTINCT] dock book.
>> HARTMAN: They log to--I'm sorry log what?
Dock book?
Docks has to do with others and that goes in the Kernel.
Anyway, come on.
>> SUSE?
>> HARTMAN: SUSE is Novell.
So, Novell, SUSE same to go.
HP, no.
HP [INDISTINCT] all right, Iíll just go.
You guess.
Amateurs, number one.
These are people working for themselves.
We have asked them who they work for and say I'm a hobbyist and doing something on time.
And the number five is, unknown.
They are either, we have no idea who they're working for or that they wish to remain unknown.
Some people say, ìIím not allowed to say who I work for.î Hereís my [INDISTINCT]
So, we track this stuff.
So, between the--some of those two, itís almost 25, almost one quarter of the number
of changes that go into the Kernel.
Who's being done by people that are not being sponsored to do it.
And another thing about these unknown individuals, none of them--well, I think--I think about
five of them are contributing more than 10 patches.
So, there are lots and lots of people in the tail that we donít know who theyíre working
for.
So, itís not a large--they're not large contributors.
So--or you can look at the other way, 75% of the development thatís on the Kernel is
paid for, is sponsored.
Because these companies care about what happens to Linux.
If you want to see what's going to happen to the Kernel, what's going to work for you,
you need to influence it and realize what's going to be coming in Kernel?
What's next?
It's whatever you make it to be.
Whatever you want it to be lots and lots of companies are working to--are relying on Linux.
You guys rely on Linux.
Lots of other companies, Iíve given this talk to a number of companies.
Only one company Iíve ever given this talk to is actually on this list.
What?
Google's not here.
>> [INDISTINCT] >> HARTMAN: Where is Google?
Googleís 13th.
If you don't count Andrew; so, without Andrew...
>> We count Andrew.
>> HARTMAN: You count Andrew.
I count Andrew too.
Andrew does great work.
But--so, I was trying to think of a metric, do you--do you do the mean of a company's
developers, you know, the look of a massive body or do you count the net quantity but
in smaller companies of computer developers, how do you rank them?
I don't know how.
I'm trying to figure out some fun stuff with statistics.
My wifeís in Graduate School, she happens to be taking statistics class right now.
Since she has lots of ideas, I can use her.
So, yes.
Without Andrew, you guys are 40th.
You guys rely on Linux.
This either means one or two things; that the way Linux is going and the Kernelís going--and
working, itís just fine for you.
And youíll like it and youíll love it and you're happy.
That's great.
Lots of people are that way but the other thing is, do you trust these companies to
develop the Kernel in a way that matters to you.
I would say no.
>> Why?
>> HARTMAN: You trust these guys?
They don't know your business as well as you do.
You only know the loads that you do as well as you do, so if it's going as well then fine,
I'm happy and I'm not going to complain about it.
Yes, [INDISTINCT] >> No, I mean, one thing that happened to
us on the Andrew type thing is, they will move support for--I know a trial [INDISTINCT]
this is [INDISTINCT] >> HARTMAN: Yes.
>> If we invented more active earlier on, something might be, you know, and check it.
I think they mention only one thing in the Kernel that uses this and is just not working.
>> HARTMAN: Yes.
>> Maybe if we were in there, we might have been involved in this setting.
They might has still been removed but...
>> HARTMAN: At least you should know.
I mean, if there's things you care about, that's fine.
And that's fine.
If you care about--talk about this thing, that's great.
When I come is I'll talk to you and say, ìWell, company ranking so and so is doing our work
for us,î which is great too.
All these companies, I know IBM or Red Hat and Novell.
We both do consulting for other companies and do work for them.
But again, their overworked companies and there's no reason why the individual company
yourself can't get involved.
We make it very easy to get involved in the Kernel element process based on our--number
of people who are contributing, it's kind of working so there's no excuse for you guys
not to contribute.
So, that's my reason.
So, let's pick-out some and then weíll [INDISTINCT] so, 27 people are in the list.
Twenty-seven of you guys contributed a code [INDISTINCT] and here you are.
So, Andrew number 11, awesome.
Great.
Very happy.
David, number 149th, see here, that's almost coming.
So, he is a perfect example of what I like to say is a great person to have.
He started of working at [INDISTINCT] University doing janitorial Kernel work.
He looked at the Kernel, here's a fix, I can do, start running into it, got hired by you
guys, I don't know what he does, and has contributed now to the Core Kernel, the, BF--to the--what
does he do?
NM stuff, out of memory stuff, Core Kernel guy, he's contributing pretty well, 67.
That's in the top 8% of all Kernel developers like quantity out there right now, great job.
Paul, what does Paul do?
I looked at CPU scheduling, that's good.
Ken does what?
He used TLB and other sort of stuff.
Nice Core Kernel work, really good.
Then there's Matt, is Matt here?
Okay.
Matt puts everybody else in this company to shame.
Matt contributed spelling fixes to the Kernel, which is awesome, and I will not say--and
I love him and that's a really good to have but Matt beat out all your other Kernel guys,
he's number five out of the whole company contributing changes that fix spelling.
So if you're a developer, I donít--you should feel a little ashamed about that.
I mean, I gladly take the spelling change.
I hope they contribute to more in the future but come on.
All right, guys.
And then I didnít--I gave up on looking at [INDISTINCT] what you guys do.
>> How does the 27 compare to the other companies?
>> HARTMAN: On quantity?
Number of developers?
>> Number of developers.
>> HARTMAN: It's pretty good.
There's some companies with like 90 some, which is amazing like that--but then there's
smaller companies with like five developers that are very high in the list too.
So it's like I can't look at pure quantity of unique contributor.
I just can't figure out a number.
Do I look at the mean, the contributing one?
I look at the mean here, you're way down to seven, you know.
I think you guys want me to do that.
So Martin, where is Martin?
Martin disappeared.
So the joke is when anybody ever goes to work for Google, they enter the black hole.
Never to be seen from again, Martin, you still actually contribute to the Kernel.
I know Martin, so I can make fun of him.
Here's some more, two with patches, two patches, two patches and we start getting to one.
Al, thatís cool to see Al works here.
He used to contribute a lot; used to and then there's the rest.
>> Just that there are so many interesting things.
>> HARTMAN: There are so many interesting--and thatís great and thatís awesome and I have
nothing against that.
But if you can--if you--and I know you guys do a lot of development in-house and have
your own custom set-up and certain things like that, which is great and fine.
But if you think you can keep your changes outside the main Kernel tree and keep up with
that rate of change, I think you're going to have a problem.
It's really, really hard to do even with our good tools that handle merging, git and things
like that.
I know this really doesnít show and none of that Android stuff has landed, hopefully
there's Android shuffled in.
>> [INDISTINCT] know, your work is passion able.
>> HARTMAN: Okay.
Four [INDISTINCT] already, I can handle it.
>> Do we have a locker room?
I'm hoping to get that [INDISTINCT] >> HARTMAN: Hopefully, yeah.
I've heard--I've heard that before.
But yes, I trust you guys.
I trust you guys will gather the stuff but, so.
>> [INDISTINCT] trust yourself this [INDISTINCT] >> HARTMAN: Trust yourself just to maintain
it.
>> There's a lot of talk about how easy it is to contribute and your mileage can vary
sometimes.
>> HARTMAN: Your mileage varies sometimes and how--yes, and how easy it is.
If you ever have a problem with getting things in, talk to me, talk to Andrew, talk to the
other people.
We will help make this happen.
I have developers that are willing to help out with cleaning stuff up, we've take--we
take code from other companies that are not the proper coding style, we'll clean it up
for you.
We'll maintain and we'll get it in the Kernel.
We do this kind of stuff all the time so it shouldnít be here.
So, my main point here, you guys rely on Linux, there's no reason why you guys can't get involved.
You guys should contribute and help and drive it in a way that you feel is going to work
best for you.
Otherwise, it's going to work--good then I hope--and I hope it still works, so.
Everybody asks what's new.
Here are some of the things that happened in 2.6.26.
It's a joke to tell you everything that happened new.
It's a lot of stuff happened, a lot, a lot of the stuff.
So any questions?
That was fast.
In the back, yeah.
>> When you talk about the growth of the number of lines imposed with the Linux, I think they
probably have 95% of that is device drivers orÖ
>> HARTMAN: Well, maybe the talk of the code--95% of these drivers, no.
I think you missed in the beginning, 50% of the code in the Kernels is drivers--55% of
codes in the Kernels are drivers.
The rate of change is proport--is exactly proportional to the amount of code that is
in the Kernel.
So those 4,000 some lines added, 55% of that is in the driver's subsystem.
So 5% of that is in the core Kernel.
It's proportional across the way.
So the majority is drivers by a small bit, after driver's is architecture specific stuff
then networking but that's it, so it isnít all drivers.
But it's scary in a way, that means your Core of the Kernel is changing that fast.
The Core of the Kernel is changing at a rate that blows all other operating systems away
just by looking at the map or just touching the core or the whole subsystem, think about
that slurs.
Yes?
>> So what would you say is driving the change to the Core Kernel?
Is it because people just keep getting up better ways of doing things or because the
requirements are constantly changing?
>> HARTMAN: So, what's driving the change--the thing in the--what's driving the real change
in the core?
It is the fact that Linux is--I mean, Linux always jokes--Linux is not--is not intelligent
design, it's evolution which means we react to stimuli thatís happening in the world.
Just like, "No, we have to support this new architecture which means we have to change
the core to do this now."
We donít over plan things.
We take what's happening right now.
So, yes, new features happen and we add and we figure out how to change the core again
to work.
There's people who come up with, "Hey, I have good idea that we can get this--if we rewrite
these--we rewrite them--this locks to new [INDISTINCT] instead of spin locks, we can
get the RT stuff in there and get real-time in the core.
The real-time stuff shows that you can take these crazy out of the tree patches and merge
them into the Core Kernel and now we have a real-time where Kernel of Linux--some of
the people said that it can never be done, which is pretty amazing from the [INDISTINCT]
so, it's everything.
Itís just responding to external stimuli.
So we're not over planning, we're reacting at the time that it needs to be done.
[INDISTINCT] a lag a little bit, so some hardware companies would come up with a new device
and like, "Oh, crap, we you rewrite this."
Luckily we're getting better and these hardware companies are getting involved in the process.
Look at Intel in that list.
Intel's very high.
They're very big and Linux because Linux runs on all those chips they want to sell chips
they make sure work's really good on their chips.
[INDISTINCT] penguin picture.
Yeah.
>> MALE: So why donít you guys don't we pay more attention to consistency?
>> HARTMAN: Why don't we pay more attention to consistency...
>> MALE: [INDISTINCT] >> HARTMAN: Something's starting to change
rapidly and our interface would do change rapidly.
Hopefully, we never change the user space interface.
So, we have the boot line changing for them to do stuff.
I don't know maybe they had a really good reason to do that.
Hopefully they did and that sounds like a foolish thing to do.
We hopefully never change--we change the internal Kernel API all the time.
Everybody was like, "You have to have a frozen.
You have to have a state API."
Then you say, "What API?"
We have 8--9,000 symbols, we have 12,000 different structures.
I mean that are passed around inside of the Kernel.
You can't freeze that because it doesnít let's you ever change anything.
So inside the Kernel we change all the time, we change rapidly, we have really good reasons
for doing that.
But outside the Kernel user space API's, we should never change.
I can run binaries ever built for 0.696 just flowing today on the Kernel, 2.6 Kernel today,
cisco interfaces don't change.
For the most part, there's a little gray area of some interfaces that were on the Kernel
and user space like the proc file system that--end up changing every once in a while but hopefully
we catch all the--the users around that.
But we're trying to document that.
We're documenting that user space API now.
There's documentary [INDISTINCT] ABI, that we document that.
The command line stuff--we do document that although change was odd, I don't know why.
It shouldn't have.
You should complain to developers.
Maybe they didn't realize the broke it.
Sometimes that--like, "Oh, I didn't realize I broke that."
You know, like, "That's a book.
I need to go fix it."
Regressions are always going to happen and hopefully we fix this.
In the back.
>> MALE: Once a upon a time, you advocated the distribution [INDISTINCT]
>> HARTMAN: Yes.
>> MALE: [INDISTINCT] >> HARTMAN: Yes.
>> MALE: [INDISTINCT] >> HARTMAN: So the question is awhile ago,
I said that enterprise Kernel should just grab the Kernel all the time for there's enterprise
Kernels.
While Andrew keeps complaining that the user space that are quality going down the reporting.
I think Andrew's wrong.
I see every single Kernel bug that comes in the three different districts or to Kernel.org
and to two districts, gen2 and [INDISTINCT].
And over time that number reported is flat.
Red Hats bugs, their bugs require for the Kernel--is actually going down.
Now, either that means we have a fixed number of testers and people reporting problems,
this can be true.
But we're increasing our size of our code; we're increasing a number of users on our
code.
So, obstensively, I'd like to think that we are doing better because our code size is
going in a number of bugs reported is flat.
Andrew sees a lot of--a lot of bugger reports.
He sees a lot of stuff and a lot of stuff all the time.
I see a lot of stuff all the time too.
But if you look at our rate of change for what's happening I would argue that we are
doing better or doing good.
If we break the user space API, let us know and we'll fix it.
I mean, regressions always happen.
I can say that.
So as far as the enterprise version, I've said it before publicly I still think the
enterprise Kernel model is wrong.
I get paid by enterprise Kernel Company.
I think it's wrong.
Because if you look at what they do so, take SUSE or their [INDISTINCT] SLED 10--SUSE or
enterprise ten.
You take 2616 Kernel froze it in time and so, okay, here's what we're going to do.
And they're fixing bugs and make sure they pass all regression test and ship it.
A year later companies come along and sort of say, "We're going to do a service pack
up date because we need to support new hardware."
Problem is new hardware comes out that changes the guts of the Kernel based on the fact that
we didn't know those things are going to happen.
We didn't know that ATA was going to happen.
We didn't know these new devices were going to happen.
We know the wireless stack, what was going to happen there.
So you go back and you kind of shoehorn all these stuff into there.
If you look at what we have--we have I think 3000, maybe 4000 patches to 2616.
It's not a 20616 Kernel anymore.
It's 2616 with a bunch of bug fixes that we know were reported to it.
And that's scary because we know a lot of bugs have been fixed between 2616 and what
we have today, 26.5 that werenít reported, that werenít fix or that we didnít know
about it and we didnít [INDISTINCT] so, extensively the Kernel, the user interface should be stable
and stay the same.
We donít change our version number to make people feel good that are used to the old
Unix Model, things moving very slowly.
The windows model things move very slowly that I think, "Theyíll stay with API."
no, they just kind of slow API.
All those Kernels--all the other operating system, they change, they break their API,
they break their driver API.
I'm on the windows driver development mail list.
It's the best mailing and stuff.
You just see the quotes for the Microsoft developers.
Oh, yeah, we did something change there so we donít remember what change are we going
to have to test, a lot of angry developers all the time.
So, just like users, it's just like Linux.
We change our APIs all the time.
They donít have any fixed API.
So, for inside the Kernel, outside the Kernel they do, hopefully.
Outside of our Kernel we do.
So, I still think that you should move forward.
Now there's a model people that take a box, an enterprise box, put it on it and throw
in the corner and donít touch it for five years.
And then enterprise model that only gets security fixes and backwards to that that's great but
only about 20% of the market out there because if you buy a box to shove in the corner and
you're like a year from now you want to buy a new box, the new box is new hardware, and
yet this [INDISTINCT] a new hardware on that old version of Kernel and that doesnít work,
for desktop it just doesn't work.
So, I still think the model isnít going to move faster.
So, inside I'm trying to argue that, [INDISTINCT] is trying to argue that, see what works, you
know, it's an interesting dilemma.
So, now we're still making money so I'm happy.
That's it?
You guys are easy.
>> MALE: So, where are you going to go next, let take...
>> HARTMAN: Where are we next?
Well, what's next?
Yes, well, like I say is whatever you wanted to be.
Things that I think are cool, KVM.
KVM, is really, really cool.
It's a running a virtual machine it's a hyper visor, turns Linux into a hyper visor.
I think it's going to win other of this and model, two different or the VMware model.
Like, what VMware does it's actually Linux running there.
So all the big embedded or big hyper visor guys are really running Linux--cant find a
watch.
But so KVM School, KVM can afford the S390 main frames, port IO 64 for the arms coming
and there're some other accessory on port too.
So KVM is really cool.
It's a really neat technology, just new virtual machines.
So there's another way, if you have an enterprise [INDISTINCT] did you want to run fixed, you
can run on top of KVM or a virtualized machine, you would change the hardware, you get the
new drivers for your virtualized machine for your low level stuff than your virtualized
machine and you can still render old version window just fine because at API between the
two is fixed.
And that's a new idea.
So, you can run your applications to run your old app--the old things, they're running on
this old API especially you're C++ libraries enterprise have a certain C++ library and
it never want to change that--things like that.
I know of some big shops, Linux shops, they're stuck using [INDISTINCT] and [INDISTINCT]
three because the C++ developers do not want to ever change the compiler.
It's what those guys do as they strip out the Kernel underneath and replace he Kernel
with the latest enterprise for the latest Kernel--old kernel and then usually above
that now even knows it happens to use them, they just know a raw metal stuff.
I give [INDISTINCT].
PSP3 is coming hopefully sometime in the future and of all without, that should be good [INDISTINCT].
Yeah.
>> MALE: So, I'm wondering [INDISTINCT] has a great [INDISTINCT] pretty dynamic thing.
I was, kind of, curious, you know, like the [INDISTINCT] on the particular subsystem they
[INDISTINCT] with that and it comes to like on particular individual.
And what's your plan at certain [INDISTINCT]?
>> HARTMAN: So, as the load as we...
>> MALE: Or clients want to settle down.
>> MALE: [INDISTINCT].
>> MALE: You know what I mean?
>> HARTMAN: Yes, [INDISTINCT], resettle down.
>> MALE: Settle more.
>> HARTMAN: Settle more.
The substance and maintenance what's our load and as originally the--our load has gone out.
I know my personal load I dropped, I donít maintain the PCI, I mean, [INDISTINCT] more
because I wasnít paying enough attention to it.
I know some of this is going to get more attention is now taking over.
So, hopefully we can or smart enough to recognize among ourselves that when things are going
wrong and we can change.
We are nothing but introspective--I love talking about this.
Every Kernel, someway we always talk about the process.
On the middle is we argue about the process.
As our process good or not and we changed things all the time, we have this table tree
because people like our process really isnít covering up.
Let's see how that works.
The Linux next off, hey this process isnít working some system maintainers are burning
out.
Let's try this.
So, we're nothing if not keep trying things.
So, hopefully we just keep trying things and as soon as you're working really, really well,
well better than we ever thought it would be.
We havenít hitting max limits yet so far.
Every time we do a Kernel is like I was the fastest ever.
The 2625 Kernel blows away all those--those are average number.
She's a 25 as the fastest, most amazing rated change, most developers, most individual companies
and anything we've ever done.
It's crazy and now, probably the next one will be more faster.
So I don't know.
Hopefully--I don't know where we're going top stop.
We've already broken all the records so I don't know.
We're making up as we go.
Don't feel good about Linux now.
Here.
>> So I have a sort of strange question--I mean, the set of layers kind of suck right
now.
How is it going with that?
>> Set of layers of suck you, that's news.
>> [INDISTINCT] >> HARTMAN: They use more suits, then let
the developers know.
>> Okay.
>> HARTMAN: I mean that--that's, you know, new to me so let them know.
If they don't know these problems then that's a big thing.
If you find the problem let us know that's the best thing about our development model.
You can find the person responsible for the code and make fun of him.
>> Or give me a patch.
>> HARTMAN Or give me a patch.
Well, that's sign off by everything.
So that's really--we like it--call it the path of blame.
So you see a bug, "Hey look, all you guys miss this, fix it."
So, yes if it's using too much CPU let us know.
We'll work on it.
Tashion is really good, really responsive [INDISTINCT] really good at that.
So, they're good at this stuff.
All the stuff has really weird hardware.
Some of the hardwares written so that it's requiring more CPU power to do some of these
things just because some of those manufacturers are also CPU manufacturers.
So, they want you to buy new CPU.
I don't know.
One thing about KBM it doesn't make you buy a new CPU that's why AMD and Intel.
I think itís a virtualization stuff.
>> Does KBM [INDISTINCT] to [INDISTINCT] >> HARTMAN: KBM is the--or KBM makes Linux
be the hyper visor.
The way it has a Kernel developer it is the hyper visor.
If you look at the way Zen model is, they have to have to their own scheduling, their
own memory management or their power management so and then have Linux running on the side
to handle the drivers because nobody wants to write drivers.
Linux is always used for drivers.
Herd uses Linux for drivers.
Yeah, I'll take that.
So, Linux--so why not do--so Zen is having a duplicate all the numerous stuff, all the
power management never run in--on the laptop, it will suck your power dry.
So, why not just use Linux as the core, because the core of the Linux is equal the size of
the core of Zen.
Actually--Linux Kernel is actually developed by more people.
Maybe they know more things.
They can do more things faster so use--so KBM models reuse Linux and then instance of
the--the guest operating system runs in there's a little driver.
There's KBM driver and just runs up there as a--it runs as a process.
So, the really cool thing about that is real time.
You can now real time schedule these guest operating systems and the real time is a real
new thing.
Some people can measure network latency and latency of processes in dollars.
And they can do that in Dollars and then--so, if they can reduce that.
They can make it predictable.
They can throw some really, really big machines at it and make their cost go down and make
more money.
So real time really matters for those guys.
Real time might matter for you guys for network latency.
Other big companies would do a lot of web services really care about the network latency
they want to be able to schedule.
They donít want to be waiting for another process to come in and finish thing.
They want to be able to know what's going to happen to determine it holistically.
So, anyway--so KBM can handle that.
KBM can then real time schedule your guest operating systems Zen can't do that.
Zen also did their development often in the side of the corner and never played nice with
the Kernel developers.
They're trying to catch up now.
I don't know if they're ever going to catch up which is sad.
That's it.
You guys are easy.
>> Will there be a 2.7 Kernel?
>> HARTMAN: Will there be a 2.7 Kernel?
We said a long time ago, "Oh, if we ever do four little page tables or if we ever replace
the timers of the systems, maybe we'll need a development branch that we're here to do
this."
We've done those things.
The really cool thing about that development model and the one problem of the developed
models we burn engineer's time.
We waste engineers.
We make you break your changes up into tiny discrete units that can be tested and obviously
fit--obviously reviewed.
So we make you take your, "Oh, I'm going to rewrite the timer sub system into a lot of
individuals steps over time so which you'll never know if things go in."
So, we've done these major things we thought we're going to have to happen already and
people didn't notice.
Well, some of the performance guy notice four label page tables [INDISTINCT].
[INDISTINCT] reformats some place.
They fix that since then.
So, I don't know.
We don't need a two set.
I mean, the numbers are marketing.
You know, so I don't think so.
Anyway I think we're running out of time.
>> I just want to--let me--put the mic here.
Is this mic active?
Good.
I just want to remind all of you that our outgoing open source patch process is very,
very low ceremony.
If you got a patch right now you can ship it out within the hour.
So we're not holding you back as a company, so...
>> HARTMAN: Spelling fixes this all.
>> Consider spelling fix.
Anyone in this crowd can do a spelling fix so I'll encourage you to make Linux the best
spelled happening system in the world.
Oh, yeah.
Now, we're certainly fast when it comes to shipping out patches.
We only ask--will you look at your first couple and that's mostly to make sure that you're
not insane and not from a--not from a technology stand point but actually from a craziness
stand point, so.
And after that we ended up trusting you and that's how it works so please do it.
>> Okay.
Well, thank you all for having me talk.
>> Not that I think that 13 is bad.
I think it's pretty good for a company of our type.
>> HARTMAN: It is so.
>> It's a decent place to start.
>> HARTMAN: It's okay.
>> But it'd be great if we were nine.
>> HARTMAN: Yes.
>> I could make that first slide.
>> HARTMAN: Yes, everyone wants to be on that first slide.
>> Great.
>> HARTMAN: Great.
Well, thank you very much.