Bill Hilf recalls an early day in his career at Microsoft. He had been hired away from IBM, where he led Big Blue's Linux/Open Source Software technical strategy-a job description that at first glance wouldn't land him anywhere near the Redmond, Washington campus. But there he was, standing in the room that would become the Microsoft Open Source Software Lab, a place where 300 servers would run some 50 Linux distributions. Hilf watched as some IT guys threaded a network cable through a hole in the ceiling. He was still staring at the piece of fiber when one of the the men came downstairs to inform him they had finished. Completely finished. Hilf would have to build the rest of the lab from scratch.

It wasn't that the company was being stingy with its resources, but rather that Microsoft IT wasn't exactly a hot bed of expertise when it came to supporting Linux and other fruits of the open source movement. Indeed, for Hilf and Microsoft both, that was the point. Open source wasn't going away. Some Microsoft customers also ran Linux servers and other open source tools. If you want to know what's going on, bring in the equipment and expertise and find out.

While Microsoft executives have sometimes lashed out at open source, the company has also admitted that community development can produce competitive products. "We have to compete with free software, on value, but in a smart way," Steve Balmer told ZDNet News back in 2002. "We cannot price at zero, so we need to justify our posture and pricing. Linux isn't going to go away-our job is to provide a better product in the marketplace."

So maybe the idea of an open source laboratory funded by Microsoft isn't all that surprising: the company would at least want to check out the competition. But instead of hiding the lab away as a secret, off-campus project, the company has been quite open about what it is up to. The Lab has documented and promoted its work at http://port25.technet.com/, while inviting blog-like feedback from readers. Hilf, whose official title is general manager of platform strategy, still travels the world talking about the lab. (When I spoke with him, he had just returned from Brazil.) And in building the lab, he has created an open source environment of dizzying heterogeneity. In an ocean of Windows servers, Hilf and his group have put together an island of open source: lots of different hardware running lots of distributions, including obscure ones like Asianux, CentOS and NetBSD.

The staff includes both full-time employees and part-time contractors. Some have been immersed in Linux, such as Gentoo Linux founder Daniel Robbins, who worked in the lab in the second half of 2005. Other staffers have written UNIX books and tools, worked on open source software security, embedded development, cluster computing, and desktop interfaces line GNOME.

Microsoft shows no outward signs of building its own Linux distribution or releasing the source code for Microsoft Office-either of which would give industry analysts a coronary. Rather, the aim is to accommodate of the obvious reality: in some sectors, open source is a serious force, as well as one of the few technologies that Microsoft cannot purchase outright. Microsoft uses the term "coopetition"-the melding of "cooperation" with "competition."

The Lab has made a virtue out of its island-like status, arguing that it mirrors the kind of environment where open source sometimes can find itself within a primarily Microsoft shop. Researchers deal with platform integration, software deployment, and connection to the Microsoft world. Tools used by the lab include Vintela VMX, Kickstart, Red Carpet, Portage, and Red Hat Network. Remote management is done with SSH, VNC, X-Windows Tunneling and Windows Terminal Services. Hilf doesn't claim this array is typical of a customer's setup, but says that the diversity of the tools yields more information and has yielded some techniques and outright discoveries-such as the ability to extend Microsoft Systems Management Server (SMS) to manage non-Windows OSs. Another Lab accomplishment: running all its 50 Linux distributions as guest operating systems on a single machine running Microsoft Virtual Server 2005, Service Pack 1.

The lab does comparative benchmark testing between open source applications and their Microsoft equivalents. For example, the lab compared Microsoft's high-performance computing entry- Windows Compute Cluster Server 2003-with others in the HPC sector, most of which run Linux. The Lab has also done some interesting work in measuring how far commercial Linux distributions have strayed from the original distributions. Using data from kernel.org, the Lab found that the changes have grown considerably over time, with just 411 changes for Red Hat 7, released in September 2000, versus 3971 for Red Hat 9, released just six months later. SUSE 9.2, released October 2004, had 5153. Hilf says the tracking is useful in comparing open versus commercial development, as well as in deciding what distributions to support. Since August 2005, the Lab has also been tracking how many PCs and servers are supported by two of the most widely used distributions-Red Hat and SUSE-compared with those of Windows. Aside from competitive analysis, the tracking shows what hardware configurations might be needed in the lab. Another part of the Lab's mandate is to act as a kind of technology ambassador between the open source community and Microsoft. It was in that ambassador role that Bill Hilf spoke with Software Design.

Microsoft shows no signs of becoming an open source company. So what is the core purpose of the lab?

In many ways, we're trying to understand what customers really want when it comes to open source and commercial software from Microsoft. Not so much what the pundits, philosophers and prophets want, but what do the real customers want? At the end of the day we're a public company, we're a business, and we want to make things people value so we can sell it to them. That's our model. So what we try to simulate at the lab is what people are really interested in. We found there's a tremendous need for interoperability, for example, particularly in the data center environment, between Linux and open source, on one hand, and Microsoft software, on the other.

Or take an organization like JBoss [a division of Red Hat that is the nucleus for a number of open source projects]. Half of their customers use Windows Server-so they are our customers as well. So it's less about: "hey, we love open source" and more about "if we can build the right bridges, there's more business opportunities for us as a company." That's the difference between the philosophy of open source and the business reality of open source, which is where I spend much of my time. Because at the end of the day I'm employed by a company that makes money from selling software.

When you arrived at Microsoft to build the lab, it sounded like you did everything but the wiring. How did the Lab get built?

When I took on the role, there wasn't much definition about how to get things done. So I just did them myself. I already knew how to build a data center and get it up and running. So I told the people that I worked for to just give me enough room to run and I'll get the job done-but let me do what I need to do. They said: 'fine'

One of the hardest parts was just finding space on the Microsoft campus. Redmond is a pretty small area. It's not like being in the Silicon Valley-the campus is confined by geography. I finally found a room where they had been storing some old equipment, so I re-architected it to support a data center. The fiber optics was the easy part. The hard part was first getting 220-volt power laid into that room with enough capacity to support up to a thousand servers. We had to hire a bunch of electricians from all over to reconfigure the room. Microsoft IT is a great idea organization, but it is heavily focused on Windows.

For example, because I was running complete non-Microsoft software in my environment, the Exchange Server testing lab didn't have the infrastructure or people to support me. So we had to do it all on our own. Which was the way I wanted to do it anyway because the whole goal of the lab is to emulate customer environments as much as possible. That includes the operation and management of that environment, not just having a server with an operating system. So this was all part of the experience. People always love to come by and see what it looks like and take a peek inside.

Did Bill Gates stop by?

No, he didn't stop by.

How about Linus Torvalds? Was he invited?

I think I invited Linus one time, but he's in Portland. He actually only barely goes to his own work at the Open Source Development Lab. He works from home a lot.

But I've had lots of customers come in-big customers, small customers, developers, and certainly different open source groups like Jobs, Sugar CRM, and a variety of others. It's a fun environment to be in. I even put an Xbox 360 in the lab a couple of weeks ago.

Within the Microsoft campus, where is the lab?

It's a room inside of a building right on what we call Main Campus, which is where most of the core product groups like Windows and Office are located. It's in the same building as the rest of my team, so it's pretty handy.

When you put the lab together was it just you, or was there a small group?

At first it was just me. I just had a bunch of Linux machines in my own office. In time, I had so many machines and it got so hot, that the air conditioning facility people had to come in and modify the office. So finally, I realized I should just build a lab because I was eventually going to have even more machines. I hired a guy to help me out until we got set up, and the lab just grew from there. Now we have a pretty sizable group.

When you approach open source people to work at the lab, do they think they are entering the enemy fortress?

There's so much idolatry around Microsoft and open source. To put it in context, I've been hiring open source people for a long time at IBM, eToys, CNET and elsewhere. Over the course of my career, I've probably hired 300 to 400 different open source developers. So I knew what characteristics I was looking for, but I also knew what appeals to them-the environment that they like, the flexibility, and the ability to be both a hacker and an administrator. It's often a different dynamic than is found elsewhere.

We hire typically on a yearly contract. That gives us a constant rotation of new talent from the community. They can come in and have an interesting experience-not just with Linux, but with many types of open source applications we research. I've got guys who have left my team because they did their year and went on to open source companies around this area - Motorola and Amazon, for example. And they are always asking me when they can come back: because they've never been in an environment where they can work with so many different types of technology in one spot. That's quite different than being, say, the mail administrator in a big company, or someone who is just configuring file or print servers all day.

The only comparable experience I can think of would be other open source labs.

Even they are very focused on a few key projects, Linux being one of them. I always tell the guys in the Open Source Labs, both at Oregon State and at the Open Source Development Lab, that I'm pretty sure I run more Linux configurations than most people on the planet, because we run everything-embedded, desktop, server, output devices-and across each of those we'll run 30, 40, even 50 different Linux distributions and other open source applications. The projects vary greatly. One day, a researcher might be working on virtualization technologies, the next day he might be working on wireless support for a Linux desktop.

Where did the lab's Port 25 website come in?

When I built the lab, I went all around the world talking to different customers about it. But eventually I figured there has to be a better way than logging miles on every airline on the planet. The site does that-and in a way that allows more people to give more feedback.

I'm sure you've heard of the essay "The Cathedral and the Bazaar." Are you running a bazaar in the middle of a cathedral? And if so, are you each learning from the other?

That's an interesting way to say it. Eric Raymond's Cathedral and the Bazaar was more about software marketplaces than labs. But in some ways, we run a very non-Microsoft environment, and not just in terms of the software, but also in the way we manage and use it. In many ways, we're both inside and outside the fishbowl. People can look inside and see how Linux and UNIX people might go about administering a server room. And we can jump back and forth to try different things out. So when we do analysis projects, we try to get a real sense of what it's like to be in a Linux-oriented environment.

Microsoft is full of very technical people, much more so than any place I've worked in my life. There's always this wanton lust and curiosity for technology that is new and different. Sometimes I have a hard time getting people out of the lab - they want to keep experimenting, exploring, and trying different things.

Do you give presentations within the company? Are you an ambassador for open source within Microsoft?

Yes, we do it in different ways. Sometimes, we do executive briefings, where we'll bring a customer to Redmond for a couple of days. I do that a lot. I also go around to different product groups at all different levels of the company to discuss how open source works or whether we are building the right product for this audience. It ranges, but most of my time is spent out of my office-at other people's offices talking to them.

One lab project was to track the changes on different distributions over time. Why?

The way I try to approach the entire research process with the lab is to focus on the technical realities of open source, not the perception. So we do very deep analysis in the software, and we do it historically. Over time we'll look at how the software has or hasn't changed so we can understand the real dynamics of open source development. The data you are referring to shows that as open source addresses broader user bases, more complex challenges, and more difficult functionality, the code becomes more complex and difficult to work with. A lot of these projects plateau at that point, because it's very hard when you have a large user base.

One of the unrecognized capabilities of Windows is that we make sure that many, many devices and software applications actually work with it. That doesn't happen by magic. It's because we pay some guy to come in every day to test that a digital camera can plug into Windows. So when you start getting big user bases, it takes a while to make sure that everyone's apps and devices are going to work long-term. And we have a lot of users: 400 million using Office today and probably over a billion users of Windows, when you include both licensed and pirated copies. So the level of accountability is pretty significant. That's what I've seen in the growth of a lot of open source software - as larger user bases have more demands, the complexity grows and everything becomes harder. Building a kernel to run on a single processor machine is one level of complexity. Building it to run on 64 processors efficiently is a different level of complexity entirely and requires different skills.

You have argued that complexity leads to increasing commercialization of open source software.

That's inevitable because what is "good enough" no longer is when you hit a critical mass. A given open source app might be good enough for a very technical person, because they know how to get it to run. But for my mother, who got her first computer just a few years ago, "good enough" has a different definition. It's not good enough if she has a horrible experience and is calling me for technical support. Nor is it good enough if she doesn't get done what she wants to get done, be it write email or browse the Web.

These projects are also getting larger because they are now heavily staffed with people who work for commercial companies. Linus was just quoted in an Oregon newspaper saying that 80 percent of all the work in the Linux kernel is done by about 50 people. Those 50 people are typically employed by IT companies that make money from Linux. So these companies are taking senior talent and having them work on the Linux kernel so they can have a better capability for whatever relates to their business. In other words, the people working on complex things in Linux are accustomed to working on complex things in other operating systems. That's not surprising. They are engineers who are disciplined in these areas because some of these problems are now very difficult.