Staying on the cutting edge
October 8, 2004
Winter has ended in Melbourne but I'm not the sort to venture out without at least a sweater. However, the cold doesn't seem to bother Theo de Raadt as we sit in the lobby of the city's Duxton Hotel, a day after the end of the annual summit of the Australian Unix and Open Source User Group.
But then de Raadt is used to the cold - he lives in Calgary, a place where minus temperatures aren't exactly uncommon. And despite the impression generally given out that the founder of the OpenBSD project is a person who is inclined to be anti-social, I find him to be nothing but warm and friendly - a sharper contrast to the weather could not be imagined.
It's unusual for him to sit talking about himself to a journalist - all the queries he faces from the media are usually technical ones. But he waxes voluble all the same. There's often nervous animation in his voice, the words tumbling over each other with an energy I've encountered in developers who are way up there in the stratosphere.
De Raadt was born in Pretoria in 1968. His father had moved there from the Netherlands soon after the war and his mother's side had already been there for a generation. "An interesting thing that I like to always reflect on is that my grandfather moved to South Africa when he was nine; my father moved to South Africa when he was nine, and I moved out of South Africa when I was nine," he says.
De Raadt Senior was keen to leave the country due to the "great fear of stuff that was happening and, of course, the conscription, the mandatory two-year conscription." His parents went around the world once when he was four and again when he was six. "They checked out New Zealand, they checked out Australia. Spain was checked out. But Canada really sort of stuck with them."
The eldest of four children - he has two sisters and a brother -
de Raadt can still recall the day in 1977 when he landed in Canada.
"I remember that we left on the hottest day which we had
experienced in our years in South Africa and we arrived in Canada,
in Calgary, on the coldest day of November that had been
recorded... something like minus 39 degrees," he said.
His father was a highway engineer and started working in Calgary. There were some hard times as the amount of money the family could take out of South Africa was restricted.
A recession in the early 80s put his father out of work and when jobs did surface again, there was a choice - Bolivia or the Yukon. The latter was the option that the de Raadt family took and Theo developed an interest in computers just before the move took place. "Just before we moved to Yukon I became interested in computers. And I managed to get my hands on a Vic 20. I started programming the Vic 20. I was very interested in the way that electronics functioned, (I) got myself my computer, and very, very quickly went down to lower levels of how the architecture worked, and I was writing tiny little video games for the Vic 20 in 3.5 k of memory," he said.
From the Vic 20, he moved on to a Commodore and then to an Amiga. "The IBM PC had already shown up by then, but it was really a business machine, not really a machine that people had at home, not really a tinkering machine. The Amiga, even though it cost more, well, let's be honest, it was priced about the same as a regular PC was, but someone would buy the PC because it had all the fancy stuff, it could run games and everything, it could do all your home productivity things. All the tinkering friends I had had the Amiga instead of the PC. My tinkering friends who had not bought computers that particular year, bought PCs the next year and started tinkering with PCs."
There were a couple of others who were as interested as he was but "I was a little more hardcore in the way that I was approaching programming on a computer," he said. "With them it was more learning little things, and learning how things worked. For me, I was really interested in the application, and the consequences of the application of ideas.
"For example, one of the little games I wrote on the Vic 20 very early on, I ran out of memory. I was 80 bytes short. I actually managed to generate a tiny little piece of code. Each character on the screen had a colour, and the colour was stored in a memory higher up, where it only used low 4 bits, so the upper 4 bits of this memory were available for use. And I actually went and stored my code in the upper 4 bits, and I would copy it back and forth to be able to execute it. This is a ridiculous type of thing to do. But if you're out of memory, it's what you have to do."
The learning process continued as he progressed through school but had no effect on his grades. "I've never really had to work at school," de Raadt said. "School has always been very easy, and this continued into my university years. The courses I chose that were difficult, they were the ones I was interested in, and therefore I would work hard at them, but I wouldn't work hard at the other ones, and I would basically just fill my options up - go to the minimum amount of classes, and as long as I had a B grade I was happy."
His father and mother encouraged him to take up electrical engineering when he went to university which he found "very, very boring. It has nothing to do with electronics at all, just the basics - just stresses and loads and mathematical things which are not interesting to me."
De Raadt's first exposure to BSD came soon thereafter. The excitement is evident in his voice when he talks about it even after all these years - and at times like these, he begins to talk in the present tense. "I meet someone who is in the computer science department. And I get access to his account. It's running 4.2 BSD. And I am immediately entranced with this system that has multiple users on it, and there are these barriers to access between processes. You can open up sockets and do intra-process communication, but you cannot acccess memory to each other. You can't access the memory of another process, and if your program crashes, it generates a core file and you can see what happened. This is completely different from the Amiga, where if your program crashes, it takes the machine down with it. And I'm just flabbergasted and astounded by this."
He transferred to computer science even though his parents weren't exactly happy about it. "...because of the course load I had taken, because the first two years of computer science were so strictly structured, and packed with intradependent courses, it took me an additional year to recover and realign myself with the program."
In the meantime, de Raadt just played and played and played. And when his second summer came, he managed to get a systems administration job at one of the departments of mechanical engineering.
He found kindred souls there: "The reason I got it is because there was a fellow there, and a friend of his, with whom I became very good friends and they were the same type of people I was. They were low level addicts. They wanted to take a machine and rewrite an entire component, understand how it works, and then make improvements to it. "
"One of the guy's names was Caveh Jalali, the other guy's name was Mike Price. And we proceeded to do something completely ridiculous. We managed to get our hands on the early Minix source code from Andrew Tannenbaum - at that time it was only running on PCs and i386s. It wasn't ported to the Atari STs, which are an m68k-based machine... We decided, for better or for worse, to port this Minix source code to the Sun 3/50. Minix at the time had very poor management control, and the Sun 3/50 had a management unit, and we proceeded to start doing this - take a free Unix type thing - well, it wasn't really free, but it was kind of - and make it run on this thing. That was a year's effort for us. And about halfway along, when it was partially running, we bailed on that and moved across to try and port BSD 2.9, which is PDP 11 original BSD Unix, across. That faltered as well. But we learnt a lot from the attempts."
Jalali was hired by Stanford Research Institute. When he moved on de Raadt became the systems administrator. "I did this while I was a student for about a year. And then when the summer came, having been a student administrator there, the mechanical engineering faculty forced the department to hire a full time systems administrator who was not a student. So I was replaced. I became a systems administrator at Clinical Neurosciences, which is a laboratory attached to a hospital, where they were cutting cats open and putting electrodes in their bodies."
The carcasses of dead felines put him off after some time. "About a month into this, I was so disgusted by all the cats, that I left that place. I wandered for about two years, and then the computer science department (at the University of Calgary) hired me to be a systems administrator. Now I am a student at the computer science department, and suddenly, I'm an administrator. If I wanted to check all the exams I could have, because I had access to all the professors' files. I even had the keys to the entire office base."
This fuelled his interest further. "I suppose this really changed things for me - to be suddenly in an environment with Sun computers and SGI computers. And here we're talking like 150 Unix computers connected together. And about six months into this, I was one of the people who got signed in on a SunOS source code licence. So being the inquisitive type, I started reading SunOS source code, and finding security holes in it. Or just regular bugs.
"I just used components of the source which are recompiled, specific binaries with patches for the systems that are inside the entire department. We tried to build a cohesive network that requires less system administration. With more system administration, you don't get a chance to improve or fix bugs. So I prefer to actually just fix the problems - that's more interesting to me. I"m not a person who wants to run around helping users with buggy software. I'd rather just fix the buggy software, then I won't have to help the user. And then I have time. I can go and play and find the next bug that might bite someone up front."
When de Raadt left university, he wrote applications for a company for a year and a half. But around a month or two after he joined the workforce, the BSD source code became available. Even though his first employer had not paid him a great deal - in comparison to what he had made by selling the code which de Raadt had written - he made a big contribution to the turn which de Raadt's life would eventually take.
"He sent me to one conference, and that conference was in San Diego," said de Raadt. "I guess that was probably about 14 years ago. Probably '92 or '93. This was the conference (held at about the time) when UNIX System Laboratories had suddenly decided to sue the University of California. And they were claiming that anyone who had read the UNIX source code was mentally infected. This conference was packed with fantastic papers presented by people. And the very last slide on everybody's talk was, 'I would love to give you the source code, but it would mentally affect you'."
He found the whole discussion there gripping. "Keith Bostic presented. And Chris Torek presented. All these people were involved in the UNIX source code, which you have to understand I'd already been reading for four years on a regular basis. All of a sudden they were all there, we were going to the hot tub and we all sat there and discussed the way things work."
He also met Chris Demetriou who told him that the FreeBSD people "had already sort of started modifying their source code very, very slightly, but they were basically doing a patch kit. So they had the base source code that came out from the Jolitzes, and they had a patch kit, that had at the time about 185 patches, and it was impossible to maintain."
Both de Raadt aand Demetriou thought it was the wrong way to develop software. "Everybody thought that Jolitz would become - well, the FreeBSD people were still of the mindset that Jolitz would become open and friendly and would want to work with other people and create a community, but that was not something they wanted. They were people who thought that after this lawsuit was over, the University would somehow reform into an organisation that would play with this stuff, and allow us to get involved."
But the pair had a different opinion. "We really knew that these would just be impediments in the dialogue, in the improvement cycle. Chris had a bit of money, so after he got back to California after the conference, he set up a machine and he sent me a mail and I was the second person who did a commit to the NetBSD source tree."
The team began to grow. A friend of Demetriou's, Adam Glass, got involved. And then Charles Hannum from MIT got involved. According to de Raadt, Hannum started putting in some very impressive changes. "He knew what he was doing. So now we had a team of four people. And we started aggregating developers. I guess it was about 4 or 5 weeks when the visibility of what we were doing hit the FreeBSD mailing list. And at this point, it was not FreeBSD. It was one guy who had a machine with an FTP server, and he had a patch set, and people would send patches to him and his buddy and they would put the patches up. It was all discussed on mailing lists.
"They suddenly realised, 'wait a second, somebody else is doing this.' And they were very angry. And then, somehow, in about a week of discussion, they all realised, 'Hey, that is the way to do software development.' They created their own repository and started working from that. They did this rather than joining forces with, at this point, six of us, who had started this NetBSD project."
Even though de Raadt was aware of Linux, he says not a single person at the San Diego conference spoke about it. "There wasn't one single person, I believe, who ever mentioned Linux at that thing. So we are really saying that we were here before the Linux people were. Now, on some other mailing lists, he (Linus Torvalds) was already starting to build - Linux was already a thing that you could download, install and run. But in our mindset, it was a very weak System V. It was a Minix derivative at this point. At the time of this conference, Linux still used this Minix file system code. Well, the same file system layout, and half the code was borrowed. And you have to remember that Linux was infected by the fact that it used Minix source code.
"Linus started off an entire kernel, but used the Minix file system code and sent messages through it, and then he started replacing the entire way he called it but he still had the back end block allocation functions. Then he eventually removed and replaced the block allocation functions. Now EXT fs is just a BSD file system. But whereas a BSD file system maintains a block bit mask for which blocks are allocated, the EXT file system remembers the starts and ends of zones. But other than that, the entire way the file system is structured is the same. Everything else on the disk is the same, Everything else in the allocation is the same, except for which blocks are free and which ones are not. So Linux wasn't really there yet."
The FreeBSD group grew faster than the budding NetBSD project, but this de Raadt says was because they were not particular about the calibre of the developers who joined. When de Raadt and the others got together, there were about 180 patches for the code. "Of the 180 patches, I believe we integrated 80. The other 100 were ones that were less critical, hard to judge or entirely flawed. Of those 100 that remained, we threw about 50 out. The other 50 we worked on. For about two of them, it was probably 2 or 3 years before we could find a proper fix."
By now de Raadt was around 28 but had never thought of becoming rich. When he went to university, he had lived for two years in Yukon. As there was no university in Yukon, the local government provided a grant of $2500 Canadian for each year of university study.
"It doesn't sound like a great deal of money, but when you're a student, you live in residence and your parents, maybe, toss in $1000, then you're okay for the first year. And the second year you get the $2500, and you've worked for the summer, then you're okay too. When the third year comes along and the $2500 comes along, and you're system administrator while you're a student, and you're getting paid $9 an hour, then you're okay too. And I never hit a point of scarcity. And I never really wanted anything except computers. I spent five years in university and I haven't wanted a computer, because they're there. I didn't go to university so I could learn something. I went there because they have resources and interesting toys to play with. I don't know how to leave. So all I want now, having left university, is to continue playing. I don't want money."
The name NetBSD came about as a result of discussion within the group. "I said, it's not only that it's free, because the Linux people had started to call their stuff free, and the GNU people started to call their stuff free, and I don't want to be associated with this entire thing that's 'free', because they've just taken the word free, and they've shredded it - it's free, but here's your restrictions, and that's not free at all. So I said, it began at the Usenix conference, it's an operating system that can be used for networking, so why not call it something to do with the net? And so the name NetBSD was chosen. And I don't remember the rationalisation. That's kind of silly. But it was really a reaction against Stallman calling it the Free Software Foundation. Like free, but with restrictions, software. And already at this point there were some of us in the community who were already kind of insulted by them saying free when it's not really free. But what can you do?"
The development continued but after some time, de Raadt says he and Hannum started disliking each other intensely. Both took strong positions and had entirely different ways of dealing with such situations. "His way of dealing with a strong position is just to say a single sentence and not talk to you anymore. And I just didn't find that very communicative, it was very annoying. As the group expanded even more, we started getting some developers in the group who were not really developers. They were kind of lame. And this is something that I've always seen in FreeBSD where they end up with their developer ranks swelled, but if you actually look at what the people do, they became developers because they didn't want to do small little things. And once they have commit access, they just sit and they don't do anything anymore. Whereas the real process to follow in such a thing like that is that someone mails in a patch, and he works on a certain area and you sort of encourage him, and if you suddenly discover that he's a complete addict and he cannot stop, then he's a developer. But if he's not a complete addict, it's just a person who found a thing that he wanted to fix. Then you take the thing from him, put it in, and you wait to see when he provides something else again."
De Raadt said Hannum claimed he was insulting to users. "He goes behind my back. He builds a group of people who want me gone. They demand that Demetriou drop me. Hannum builds a position of 'he's gone or I'm gone'. And Demetriou shuts off my account. And that's it. I then sit for about seven months trying to find a way to get back to repository modification status."
There was a process of negotiation after this to see if de Raadt could get back. In the meantime, he continued working on the Sparc code which was his area of responsibility. "And I make such large modifications to this code base, I make hundreds of things work because I was the guy who worked on the code base. And I wouldn't give it to them. Because if they didn't respect me enough to allow me to be part of this community, by allowing me to make the changes, I'm not going to give them the changes and be an outside developer. No way."
OpenBSD was about to be born. "And so on a certain day, having exhausted all of my options, and exhausted the community, and having found other people who, like me, had struggled with the NetBSD people, to get me back in, I create a repository, and we start committing like a storm. And all these other people who had been disenfranchised by these NetBSD developers while I was there, join up immediately."
The name OpenBSD was picked, in some ways, as an affront to the community. "Because it's open to the community to participate. And, yes we do have barriers, but those barriers only exist to protect the thing. They don't exist to protect the animosity. That's the most ridiculous thing. Because NetBSD was closed to my participation. OpenBSD is open to anybody's participation."
The OpenBSD project was ignored by the other two BSDs for nearly two years - during which time de Raadt pulled the best bits from both those projects and incorporated them into his own project. The first release came about three months after he began, around August 1996. Then de Raadt was contacted by a company called Secure Networks which wanted him to help them write some tests for scanning networks.
"It basically was intended to take all the security problems and probe for them. But unlike trying to do a version check, this thing would actually attempt to do a break in. And then, it would give you an assessment that was scary, because it would mean that there was a product that actually had 50 ways of breaking in."
De Raadt collaborated with the programmers to give them insight into what he knew. And in return, he got a whole lot of feedback, all about security vulnerabilities which he could fix in his project. Then someone broke in to his own machine - which he says was later traced to a NetBSD developer - and this helped him to fix the flaw that permitted the intrusion. This set him and the others in the team on a security quest; as the feedback spread back to the Secure Networks it also brought in some income for the fledgling project.
The relationship ended after some time. Says de Raadt: "...at this point - we were basically about a year in - we were basically just finding the bugs on our own, and it didn't matter so much that we had this relationship with them so they could write a checker. And some of the things we were finding were extremely hard for them to write checkers. So the latency became greater between the thing that we would find and the test. And it wasn't really as much fun anymore."
The first OpenBSD release sold around 300 CDs. In 1998, when de Raadt made it to Defcon, he sold around 1000 CDs. As he continued showing up at conferences and OpenBSD started making its appearance more and more in network infrastructure, the exposure increased. One release - there is one every six months - sold 9000 CDs. Now the number is less because most users utilise FTP to install.
Residing in Canada has given him one advantage - the cryptographic export controls in the US don't apply. OpenSSH, now five years old, is the best known application to come out of the OpenBSD project and is widely used all over the world.
At 36, de Raadt is probably a bit more mellow than he was a few years back. The presence of a woman in his life for the last two years - "she's a medical doctor, she helps provide stability" - has arguably helped this process. But his approach is still much the same as it was.
Asked about mailing list posts referring to him as a dictator, he says: "I think that's just the way that it is. People just say really, really dumb things and you have to basically burn it down to their own level otherwise they won't stop. Sometimes you have to do that, sometimes you can't do that."
He doesn't believe in sugar coating an unpleasant message: "If I'm in a good mood, I'm going to ignore it (bad advice offered to people on mailing lists). And if I'm in a bad mood, I'm going to go tell someone that he's doing it. Because I don't want it to become a recurring thing that people are always saying ridiculous things."
Last year, the Defense Advanced Research Projects Agency (DARPA) pulled funding for the OpenBSD project due to some remarks which de Raadt made against the invasion of Iraq. He has no regrets about this as he says the grant was going to run out after two and a half months anyway and it would not have been renewed.
He believes the OpenBSD project is financially secure. The only large donations or grants have come from Darpa, NLNet and Usenix. Apart from that all the contributions have been from individuals. The same is the case with hardware. "Hardware donations do not come from vendors who use OpenSSH on parts of their stuff. They come from individuals. The hardware vendors who use OpenSSH on all of their products have given us a total of one laptop since we developed OpenSSH five years ago. And asking them for that laptop took a year. That was IBM. It took a year of negotiation and I had to talk to 15 people and I had the right person from the beginning but she had to get okays from other people and I had to write letters to say why. It was astounding."
During his trip to Melbourne, de Raadt was no less forthright. "When I gave my talk (at the annual conference of the the Australian UNIX and Open Source User Group) I was introduced by the guy from (anti-virus software vendor) Sophos. He introduced me by telling a little story - that all the OpenSSH out there, all that is worth one laptop. And I told the entire conference that HP Finland had only sold two Itanium machines all of last year and one of those machines was sold for $100 to one of our devleopers because HP Finland was prohibited by Intel from giving any more test machines away. Because when Itanium 1 was out, Intel basically discovered at the end that of the 6000 machines that had been sold, 4500 had been given away. So the numbers were different when it came to Itanium 2 machines.
"So the HP guy comes up to me (at the Melbourne conference) and he says, 'If you say nasty things like that to vendors you're not going to get anything'. I said 'no, in eight years of saying nothing, we've got nothing, and I'm going to start saying nasty things, in the hope that some of these vendors will start giving me money so I'll shut up'. And I said that's the way it's going to be. And wait till I give my next talk. And I said 'what are you going to do when I end up doing a talk in front of 4000 people, and finish by inserting into my slide an Itanium sales chart, and basically tell the entire audience 'do not buy an Itanium and if you're going to buy it, don't buy it from HP'? And here's why, I can explain why."
The attitude of hardware vendors obviously rankles. "Suddenly their entire model is - Linux, Linux, Linux, services, services services - and they'll sell you garbage. And no support for the open source community. Not anything. And yet they're all saying 'we're helping the vendors'. No, what they're doing is they're helping themselves. They're trying to stay alive, because they're afraid that all the Linux sales vendors are going to eat their lunch and provide the services direct to the customer. And everyone's going to find the real reason why people buy a HP box or an IBM box today is pretty much one piece of equipment - the power supply. Why? Because the grey box vendors haven't discovered that when you make a 1U rackmount machine, you have to put in a power supply that can run 24/7 for four years. And if you can do that, they'll buy your box. That's what HP and IBM and Sun do. They just put a power supply in it, with a good fan so it doesn't break. Otherwise the machines are identical."
De Raadt is one open source developer who owns his own home. The DARPA grant gave him the finance to pay up the mortgage. He has a ridiculously high electricity bill, he says - $500 Canadian - but cannot help it as needs to keep his infrastructure going. The money is enough to keep things going at the moment and he has no doubts that donations will continue to come when they are needed.
For him the excitement is still there, as much as it was when the project began. And he's still driven to be at the forefront of the technology. He can't see himself working for a company or as a consultant. "What's so exciting is to be able to just take something and polish it so much that hopefully in the future people will start borrowing things from it."