Larry McVoy on BitKeeper, kernel development, Linus Torvalds & Bruce Perens

Not long after the second part of the most recent Perens interview went online, we got email from Larry McVoy. He said there were some factual errors in the Perens piece and asked what our policy was on corrections. We assured him that we always tried to make corrections when they were needed and asked what the errors had been. He pointed out two: Perens' claims that BitMover (McVoy's company, which produces BitKeeper,) had trouble making payroll and that McVoy was constantly threatening to sue him.

Thinking that a story on BitKeeper — the controversial proprietary source management tool currently in use by Linus and others for Linux kernel development — that also carried McVoy's corrections would be more interesting to our readers than simply fanning the flames of an expectorating contest (to mix a metaphor), we took advantage of the opportunity to talk to McVoy and learn what we could of the BitKeeper saga. It's a fascinating story of the tensions between some closed and open source developers.

About McVoy

First, a little background on McVoy. If you've been a regular reader of the Linux-Kernel Mailing List (LKML) the past few years you probably know a lot of this anyway, but if you're like me and feel that daily reading of the LKML is like trying to swim against a constantly increasing volume of water flowing down a hill, sit back, grab a drink, and enjoy this Cliffs Notes version.

McVoy has been a kernel hacker for a long time. He believes he first got involved with Linux at about the 0.97 release level. He told me that "Alan Cox once described me as 'the old man who sits around and says no.'"

He has been a kernel developer for more than 15 years. He worked in the systems group at Sun, focusing on performance issues with file system and VM. He also worked on Sun's first cluster. But that's not all. McVoy said, "Andy Bechtolsheim and I were the guys who worked out that you really can signal at 100 megabits over copper." That work became 100 megabit Ethernet while everyone else was working on FDDI and ATM. Then he left for SGI and did more systems work there.

Given McVoy's experience with proprietary operating systems and systems programmers, I was curious about how he viewed the gifted amateur leading the Linux development effort, Linus Torvalds. McVoy said after "years and years of doing operating system work, including at Sun, which I still think is the best operating systems development team in the world, I've never met anybody like Linus. The guy is amazing." McVoy says Torvalds possesses two attributes that he has never seen together in the same person before. He adds, "I'm not sure I ever will see them (in one person) again."

Those attributes are exquisite taste and managerial skill. As to his taste, or perhaps technical instincts would be a better descriptor, McVoy told me, "He just sees the pictures in his head... and if he doesn't know what the picture should be, he's got good instincts about what it should be." McVoy notes he has seen similar taste in senior systems engineers at Sun and elsewhere. What sets Torvalds apart in McVoy's mind is the second attribute, his managerial skill. "He has the ability to get good work out of jerks," McVoy said. In my mind I was thinking, "OK, cool. He is skilled and experienced on the closed-source side. He has a great deal of respect for the architect of the Linux kernel. But what contributions has he made for free software and open source?" So I asked him.

To my surprise, it turns out he has made several. While still with Sun, McVoy wrote a paper on why Sun should open source the Sun OS. Although Sun didn't heed his words, Bob Young (founder of Red Hat, Inc.) did. McVoy said Bob Young has told him that paper was "very influential in his decision making process," both in his decision to start Red Hat and how they went about doing that after making the decision.

He also wrote a suite of benchmarking tools called lmbench. I wasn't familiar with lmbench so McVoy explained it is "a collection of micro benchmarks that measure bandwidth and latency of everything in your system." He added "It was actually very popular both with operating systems people and with hardware people. The hardware people liked it because it could tell you memory latency down to the nanosecond."

Software people liked it too, and in fact it is in use (and available for download, see Resources for a link) to this day. If you have previously written McVoy off as a money grubbing proprietary/closed source kind of guy who doesn't know how to share, consider this: lmbench is licensed under the GPL.

McVoy said he and Linus Torvalds used to argue about what should and shouldn't be measured in the suite. That's an important topic for debate. He pointed out why that is true, although anyone who has seen benchmarks like those used by Microsoft marketing knows this already. McVoy notes "You could send a very skewed view of a system by measuring the wrong things."

BitKeeper backstory

McVoy's biggest contribution to free software may be BitKeeper, his proprietary source management system. The story of how BitKeeper has come to be Torvalds' (and many other kernel hackers) tool of choice in maintaining the Linux development tree is worthy of a book. It's not just an unlikely outcome, given the animosity that often flares up when proprietary and open source types gather in the same space, it has been a frustratingly painful one. McVoy tells me that it was his desire to help Linus that has resulted, to use his own words, in "a miserable last five years."

Long ago on a mailing list far, far, away, trouble was brewing. The Linux kernel was about to fork big time. Kernel hackers were angry and frustrated. Patches were being dropped with alarming regularity. Tempers were at the boiling point. By the way, I'm not talking about the most recent brouhaha over the same thing, the one that Rob Landley and ESR (Eric Raymond) brought to most of the community's attention with insistent harping on a "modest proposal" about how Linus should be doing things. What McVoy spoke of first happened two or three years earlier. It was McVoy, actually, not Landley, who coined the phrase "Linus doesn't scale."

Part of the misery stems from the fact that for a systems programmer to step down a level and do applications development can be hard on the ego. Old applications programmers like myself know all too well how OS jocks view themselves as the fighter pilots of IT while at the same time viewing user space programming more like making milk runs shuttling passengers for Gonzo Consolidated Air. When my brother Allen tells me that (he being an old IBM and Tandem mainframe systems programmer), I tell him that's just an excuse to stare at the same line of code for a week. Nonetheless, McVoy told me that the drop is status is so bad that "You're embarrassed to go hang out and drink beers with your friends."

Another part is that while many of McVoy's peers were becoming — at least on paper — very rich as a result of the dot com boom, he was slaving away on the complexities inherent in taming source code management problems. Long days. Little money. Walking away from lucrative deals at Google and Cobalt that could have been worth millions.

Those are just small parts of the whole. McVoy said, "The most miserable aspect of it was just over and over and over again in the kernel group (the LKML) having people question our motivation; question whether or not we were really trying to help the kernel." So why did he do it? According to McVoy it was because of his desire to do the right thing, to help Linus (and Linux kernel development) avoid the train wreck it seemed heading for.

"I had been involved with the kernel for a long time, knew who all the players were," McVoy said. "Behind the scene when the patches started getting dropped, there was a lot of muttering, and a lot of people going 'Well, screw this, I am going to start my own tree.'"

He added that Dave Miller was probably the "hottest head of the bunch." Miller had worked for McVoy at SGI and Cobalt, so McVoy could talk to him about the problems and help to talk him out of forking. However, Alan Cox was angry too, and McVoy noted that Cox is "a much more even-headed guy." Cox and McVoy exchanged a lot of email as McVoy sought to prevent that fork, too. At the same time, he was talking with Torvalds about the possibility of a new source management tool to help carry the load. Finally, with tensions at the snapping point, McVoy got Richard Henderson, Dave Miller, and Torvalds to come to his house for dinner. McVoy said "We had dinner and then sat on the floor for about four or five hours and drew pictures. I finally got it into Linus' brain, here is what I can give you. And at the end of that he said ok, if you build that, and it works like that, I'll use it."

McVoy's first estimate was that he could create BitKeeper in six months, working by himself. The number of programmers increased to two, then four, then eight. The time grew from six months to three years. McVoy has been told that it would cost $12 million to create BitKeeper, paying going wages for a team working five-day weeks.

Eventually another firestorm of controversy swept the mailing list, again the problem was an overloaded lead and dropped patches. McVoy says Torvalds works like TCP in that "If he gets overloaded, he just drops packets." Landley began a LKML PR effort on behalf of Eric Raymond, who was unhappy about his new kernel configuration tool not being included in Linus' tree. Landley was beating the drum for Linus to adopt a new methodology. Linus said he worked the way he worked and that it wasn't likely to change. It just so happened that the way Torvalds said he worked and what McVoy envisioned for BitKeeper are remarkably similar.

"I had this picture in my mind of sort of this star topology, peer to peer system, with Linus at the center. Then the next ring of people out are his lieutenants, like Dave and Alan. Then beyond that was another ring of people, and then beyond that was another ring of people. What I visualized was that everyone would kind of hang on to somebody else, but it would be a smaller and smaller set of people the closer you got to Linus," McVoy said.

Suddenly, without warning or advance notice to McVoy, Torvalds began using BitKeeper. That happened almost exactly a year ago, on February 5. It was a slow beginning. Torvalds noted in a post to the mailing list that he was testing BitKeeper. Torvalds wrote, "Quite frankly, so far it has definitely made me slower — it took me basically a week to get about 50 patches applied, but most of that time by far was writing scripts and just getting used to the thing. Thanks to Larry and Wayne for helping out with the problems I had."

Scarcely a month later the clamor on the list about the use of the proprietary tool had become palpable. A petition was begun to stop it before it could get started. Torvalds responded with a withering reply, saying among other things, that "If people in the open-source SCM community wake up and notice that the current open-source SCM systems aren't cutting it, that's good. But it's absolutely NOT an excuse to use them today. Sorry. I use CVS at work, and I could never use it for Linux. I took a look at subversion, and it doesn't even come close to what I wanted." He added that it was "stupid" to use inferior tools for ideological reasons.

McVoy responds to Meet the Perens 2: Secret preloads, BitKeeper & TCO
The downside is that the flaming on the list has not subsided, and in fact it was the "debate" between McVoy and Richard Stallman that prompted me to ask Perens what he thought of the BitKeeper wars. It was Perens comments that prompted McVoy to contact us to ask about correcting a couple of statements. Before I got any further, let me get those corrections out of the way.

Perens said of McVoy that "Every time we talk he threatens to sue me".

McVoy responds: "I'm not sure this is even worth responding to or fixing, but it's simply not true. I've found Bruce to be a very reasonable guy in private, he's actually spent time working on the BitKeeper free-use license, he had useful comments and corrections even though he hates the idea of a non-free product. I'm not about to sue someone who is helping us. Neither I nor BitMover has ever filed suit against Bruce or any other person or entity."

Perens also said "Larry very definitely feels that he cannot support his family with this product as an open-source product. In fact, he's hardly able to support his family with it as a 'not-open-source' product. He claims that making payroll every week is a real pull."

To which McVoy replies: "Bruce has no knowledge of our financial situation, this is pure fabrication on his part and it's not the first time. He tried the same thing on a Linux Weekly News article, it wasn't true then and it isn't true now. Our company has been profitable for all fiscal years of the corporation's existence (we incorporated in 1998). Our two-week payroll is more than Bruce makes in a year and we have more than six months of cash in the bank, with another eight months in upcoming renewals (the vast majority of our customers lease and we have almost 100 percent renewal rate). Most young companies would love to be in our situation, we're about as healthy as you can get."

Today BitKeeper is available for use free (as in beer) for open source projects, but it is sold for non-open use. The primary difference between the two versions is that there is no privacy of code for open source work.

McVoy talked about the development of the business model/licensing scheme for BitKeeper. He told me they wanted a model that would give value to all three areas of concern: the vendor, the free software community, and the commercial/proprietary customer. According to McVoy, the GPL gives very high value to the free software community, but very little to the commercial world, and none to the vendor. A pure closed license gives good value to the vendor and commercial customer, but none to the free software world. The BitKeeper model strives to give value to each. McVoy said, "The light bulb that went on was that you need to find something or other that one group wants or doesn't want, and take it away or give it to them. What we found was privacy."

Torvalds has said that what he likes best about BitKeeper is its peer-to-peer design. The fact it is has now been proven to be scalable to thousands of developers around the globe doesn't hurt either when tough-minded businessmen evaluate BitKeeper for their own shops. BitMover hosts a server called bkbits. McVoy describes it this way: "We have a server here called, which hosts both branches of the Linux kernel plus a couple hundred branches. That is one dinky, 750 MHz Athlon that's serving. We've counted up, I think it is something like 3,500 distinct users. You would need to spend close to a million dollars to get a Sun/Clearcase combination that could do the same thing." I asked McVoy if the flak he gets from zealots on the LKML is bad enough to make him do what Perens and others have suggested he might do, which is to take it all back and not allow open source developers free use of the product. McVoy thought for a few moments and we talked about other things before responding fully. "To answer your earlier question, will we ever take it away? McVoy said. "I don't think we will ever take it away, but I may very well take me away."

McVoy admits that he responds to flames too much. While this is not at all unusual in the kernel hacking community, it might be the best thing for McVoy's own personal peace of mind. He describes himself as "one of these sort of egotistical but insecure people that will rise to any challenge. I don't care who you are, if you say we are doing the wrong thing I am going to try real hard to explain to you 'No, we're not. We're actually good guys. We're trying to do the right thing.'"

To date, he says, he has been on the LKML almost every waking hour since Linus began using BitKeeper, watching for problems. McVoy told me, "It is exceedingly unlikely that there will be an eight hour delay between the time the problem is reported and the time it gets fixed. In general it's a little shorter than that but I have to sleep some nights." He adds, "A lot of people in the open source world would probably say that's a good thing, because I flame too much." For all the reasons noted, McVoy is thinking strongly about putting someone between him and the list.

As to his ongoing debate with RMS on the list, McVoy says "The only reason I end up railing against RMS is he is really trying to hijack the Linux effort for his own purposes... The amount of source code that the Free Software Foundation has funded is almost zero. The amount of source code that RMS himself has written is also very small. He did write the first version of GCC and did a lot of work on that, but by far the vast majority of the code in GCC was done by Cygnus as contracting work to make other chips go. I am personal friends with all three founders of Cygnus, so I know what I am talking about."

In closing, McVoy wanted to be sure to say thanks to Cort Dugan of FSM Labs and the Linux PowerPC Group for their help with BitKeeper. He said, "For a couple of years before Linus ever touched it, Cort and his team used it. They jumped on it and used it when it was very early (in the development process) and were just incredibly helpful helping us debug problems. If there was some issue, they always saved the repository so we could go poking through it and try to figure out what was going on. Those guys you know they have done more for us than anybody else."

© 2008 SYS-CON Media