Last year, the normally staid Apache Software Foundation did something out of character: it “⁠lit up⁠”⁠ its track record of open source development by advertising on New York’s Times Square and in Las Vegas. The LED signs featured the organizations red feather logo and proclaimed: “⁠10 Years of The ASF: Celebrating a Decade of Open Source Leadership.”

Jim Jagielski was among those celebrating. The chairman of ASF co-founded the organization, has been a key developer on Apache Tomcat and the Apache Portable Runtime Project, and is the longest active committer on the flagship Apache HTTPd Server Project. Jagielski had his first taste of open source as an electrical engineering/computer science student at Johns Hopkins University. After earning advanced degrees, he joined NASA's Goddard Space Flight Center where he developed simulations of spacecraft power systems. Later, he became chief technology officer at Covalent Technologies, which merged with SpringSource, which, in turn, was recently acquired by VMware. Jagielski’s ever shifting job title is now chief architect of VMware’s SpringSource Division.

The ASF is not the oldest open source organization, but has become one of the most influential and productive, due in part to its cohesion. Developers working on the ASF’s many projects have not always gotten along, but they’ve mostly figured out how to work together. After 10 years, that feat alone is worth a sign on Times Square.

How did the Apache Software Foundation come into being?

Originally, we were a volunteer, loose-knit group of developers called The Apache Group―the people who created the Apache Web Server. The technology began attracting companies, which we encouraged. For example, IBM wanted make the code the core of their own Web server project. But they needed a legal entity on our end to deal with, not just a bunch of guys doing things in their spare time. We also realized that The Apache Group gave us, the volunteer developers, no legal protection. So for both those reasons, we formed The Apache Software Foundation, which transformed us from a ragtag development project to a more formal organization.

But that formality didn’t change the dynamics of the organization. The way we work together and how projects are developed: that’s still the same.

Aren’t software engineers supposed to be better at working with code than with people and lawyers?

Even though we are a bunch of hackers, we have a lot of other talents. For example, as an engineer with NASA, I needed to know about the legal aspects of running large projects. I was able to bring some of that into ASF. Another co-founder, Roy Fielding, did some of the formative work defining the HTTP and HTML protocols. That’s very exacting work, requiring a precision similar to how the legal system works. Roy brought a lot of that talent when we drafted the actual wording of the licenses.

As far as collaboration, the situation required it. The success of the Apache Web Server came from having many different kinds of people work together from different locations, with a diversity of ideas. The whole theory of evolution centers on diversity―on species that are fantastically adapted to their environment. We wanted to maintain our diversity. By not disenfranchising anyone, we’ve wound up with a much larger talent pool contributing ideas and insights. That’s what generates fantastic code.

This structure remains pretty unique. Many software projects have a benevolent dictator, an ultimate authority who decides what goes into the code base and where the project is ultimately headed. We don’t have anything like that. We actively resist it.

A benevolent dictator can keep projects from getting bogged down in internal disagreement. How have you avoided getting stuck?

We believe that if there is a healthy community around a code base, then the community will thrive and flourish. And conversely, if you are just depending on one person to drive development, you run a real risk. What happens if that key person changes jobs, or his wife has a baby, or he just gets burned out? With too much reliance on one person, then the code and the project will stagnate and die. The best thing to do is depend on a group of people. That way, if someone does get burned out, nobody needs to hurry and ramp up to become the lead developer. You’ve already got a core of lead developers who are always ready to keep things going.

How do you reach consensus when there doesn’t seem to be any?

We’ve certainly had our share of serious disagreements. But with our approach, anyone can raise a veto on code―and that veto cannot be overruled. Of course you must justify your veto on technical grounds: the way a piece of code has been written or how it is implemented. But if you do that, the person who submitted the code needs to address the issue. An advantage of this veto process is that it forces the entire development community to rethink what the project direction. Sometimes the project management committee gets involved. This is the core group of people with the vision for the project. They must at least try to achieve consensus among themselves.

So consensus is “baked” into the organization.

Consensus is in our DNA. That, in turn, has attracted a certain kind of developer: someone who can listen to the opposing view and, at times, give way without holding a grudge. If you are not that kind of person, then you’ll probably get weeded out of the ASF development process. ASF has given presentations about the “⁠poisonous person syndrome⁠”―referring to how just one inflexible person can poison a software project. It’s important to know how to encourage those people to rethink that mindset. And it’s important to realize that even the best developer in the world is not worth that kind of damage to the community and the project.

So forget about the stereotypical reclusive software developer. The job description for coding open source requires social skills and high human intelligence.

Right. That’s especially true during peer review. If you are working on closed source software, only your team reviews your code . Which means that if you make a bone-headed coding mistake, only a small number of people actually know it. But make that same mistake in an open source community, and it’s permanently recorded on the Web―there for anyone to Google it. That takes a special kind of developer, one who can handle that kind of open review and potentially public criticism. But that’s how peer review works in the scientific and medical communities. They’ve shown that by opening things up as widely as possible, developments happen quicker. And people themselves develop their own personal skills and programming talents much more quickly when a larger group reviews their work.

I would be nowhere near the level of programmer I am today were it not for the 20-plus years of putting my code on the line and getting really good feedback. And I’ve had the opportunity to do that for other people.

Your career spans both closed and open source development. Are we starting to see some of these advantages playing out in commercial development?

I first got my experience in programming getting my bachelors degree at Johns Hopkins. Back then, we were encouraged to download BSD code, which of course was all open source, and write our own software and donate it back. This was not a commercial exercise, but an academic one―only a relatively a small group of people were doing this. It was only with the advent of the Web, that open source development became more widely known.

Now, open source has changed the how most code is developed. Even if you don’t distribute your code in an open source license, you are still using a lot of the lessons. You are using the concepts of peer review, and using email to make sure that your collaboration remains free-flowing. The focus even in traditional developments is now more focused on consensus―and on what users are interested in, not just what the project manager thinks people want. Even Microsoft is using many of the techniques in open source in their development environment.

Microsoft actually contributes financially to the ASF.

Yes, in fact they renewed for the second year and are platinum sponsor. They don’t agree 100 percent with what we do. We don’t agree with 100 percent of what they do. But for companies like Microsoft, the ASF is the Switzerland of open source community. Some other organizations have taken a more energetic stance toward open source adherence, but we’re not that religious. We don’t feel that you must have an Apache license or you are the ultimate evil. We live in the real world. We all have day jobs, and sometimes, that means we ourselves sometimes do closed source development. By being a bridge between the open source and traditional closed source worlds, the ASF is in the sweet spot.

The ASF is now more than a decade old. What’s on the horizon?

The scary thing about the ASF is that because we are so community focused, it’s really hard to tell what’s coming next. You never know when the next big project is going to happen. Three years ago, would we have thought that Lucene or Hadoop would be as prevalent as they are now? I don’t recall anyone at ASF saying we need to have an implementation for really fast search or map reduction software. And we don’t sit around a table trying to determine where the ASF needs to go. We don’t have a six-month plan, let alone an 18-month plan. Instead, projects are driven by demand from the outside world.

One trend clearly on the horizon is cloud computing. People are getting used to the idea that you can add and remove servers and other services on the fly, without having to manually configure them. The project I’m working on right now will enable Apache to “⁠know⁠”⁠ when additional proxy servers are added, without requiring administrator intervention. People will definitely be tackling those kinds of technology puzzles.

The other clear trend is that we are still getting a constant influx of new projects, committers and developers. So we while we as an organization have our “⁠people⁠”⁠ infrastructure in place, our physical infrastructure needs to grow. We’ve been adding new servers and expanding our co-location facilities. We just accepted the Subversion version control system into the Apache Incubator, leading to it’s becoming a full ASF project. Subversion is a huge code base with a large number of developers, which will require more bandwidth and disk storage.

In your picture, why is Gumby sitting on your shoulder?

I am a Gumby fan. I have action figures all over my office, which I’ve had since I was working at NASA. It’s a traditional, stereotypical nerd office―it’s kind of messy, but it works for me. I’ve been collecting these figures for 30 years or so. And I’m still shopping.