Two of the largest commercial websites, Google and Amazon, have been quietly experimenting with ways to give programmers access to their search engines: Google with its Google Web APIs service and Amazon with its Amazon Web Services Program. In both cases, you sign up for a free account, download a software development kit, and start coding. Significantly, both SDKs make use of three standards that have become the building blocks for Web services: XML, SOAP and WSDL-the Web Service Definition Language.

Google's and Amazon's APIs are what you might call "Web services for the masses." Other than time, they require no investment. If you want to experiment with Web services, they are a good place to start. But for the vendors pushing Web services development platforms, the real impact of Web services is yet to come. The practicality of XML as a medium of communications between programs has been proven beyond a doubt. The next step, as the vendors I spoke with candidly admit, is for the industry to agree on Web services standards that will allow serious business applications.

The software industry has at times been guilty of over-promoting new ideas. Still, the amount of work being done in the area of Web services tools and standards development is substantial. For software developers contemplating a Web services implementation, the key will be in figuring out what's real, what's under discussion, what's inflated hype-and most importantly-what you actually can and cannot do with Web services as they mature.

Simon Phipps, Sun Microsystems' chief technology evangelist, believes that Web services are still in their infancy. "All the standardization so far has been on the plumbing, not about what's flying through the pipes," he says. "SOAP is about connecting computers to each other, WSGL is about explaining the nature of the connection. Web services were first thought of as little applications that grabbed stock tickers and weather forecasts. But the real money is going to be in business-to-business transactions, exchanging purchase orders, invoices and payments. I've written an invoice using an XML schema. Now: how do I get it to you? With Web services, I have a way to get that invoice straight into your accounting system, where it gets automatically processed."

Phipps helped introduce both Java and XML to IBM when he worked there in the 1990s. He says that both technologies are about de-coupling the tiers of development-and that loosely coupled applications are one of the industry's meta-trends, good for at least 10 years of research, development and influence. "Java was about loosely coupling applications to platforms. And XML was about loosely coupling data to software. And now Web services is about loosely coupling computers over networks."

The complaint about traditional application distribution methods like RPC and CORBA is that the pieces are too interlocked, and hence, too rigid. "So when I change my operating system in my department, your department is forced to rewrite some of your applications so that they can continue to work," says Phipps. "There may be business value for me in my upgrade, but there's no business value to you in your maintenance." The problem is solved by loose coupling of the applications-that is linking them strictly by an exchange of XML-formatted information. As long as both applications have a mutual understanding of what that data is, how to get it and use it, and what it represents-they can be said to be integrated. And that's all the integration you really need.

Bob Sutor, IBM's director of Web services technology says that loosely coupled services are the ultimate black box technology for application integration. "Having a loosely coupled services means that I know how to talk to it, how to tell it to do something, how to ask it something-without having a clue how that piece of software is built," he says. "If the service moves from a Windows box running on a PC to Linux running on a mainframe, nobody should see any difference, other than speed."

Moving up the software stack

Web services are really nothing more than a set of standards, still evolving, for linking applications via XML. The foundation standards are now carved in stone. XML-tagged files are the content. SOAP is the envelope for sending the content. WSDL provides the rules for how two applications talk to each other. Two organizations are involved: the Organization for the Advancement of Structured Information Standards (OASIS) and the World Wide Web Consortium (W3C). According to Phipps, OASIS leans toward XML data services, W3C toward Web infrastructure. The process has had the usual assortment of skirmishes, with vendors teaming up to support one standard while fighting each other on another.

Some observers worry that there are so many standards under consideration that Web services will get buried in a mountain of specifications. "That's bull, of course" says Sutor. "Looking back after the standardization period, you will see that Web services will be absolutely encompassed by the standards that define the technology. In the end, there will probably be 25 to 30 of them-not that you have to learn them all."

Additional standards will be needed to produce what Phipps describes as "business-strength Web services"-standards that take into account the nature of the information being transferred. The most immediate need is that the information be both secure and authenticated-so that you know for certain it is accurate and where it originated. "Today, all the heavy lifting is being done by technology which is out of bounds, so to speak," says Phipps. "The technology is not yet a part of the Web services discussion. For example, there is no way in the Web services standards discussion that you can do authentication. You have to go out of bounds and use other technology."

Phipps draws a distinction between Web services as a connective glue and Web services as a means to building business-class systems. "So far it has only been the former: a way that engineers can connect programs with each other across networks. But as soon as they want to do transaction rollback, they have to use a real database. As soon as they want to do reliable messaging they have to use a message queuing system. The things you need for business-strength Web services are proprietary or are a standard that isn't a part of the Web services discussion at the moment. And while that's changing-particularly in the area of security-Web services essentially remains a tool in the engineers' toolbox. It is a powerful tool, a widely used tool, but it's still young."

IBM's Sutor divides Web services development into three big phases. The first was the connection phase, which ended around the beginning of 2002 with the release of the SOAP. The second phase, where we are now, centers around security. Sutor calls the third phase "choreography" or "orchestration"-referring to the sequence of tasks that accomplish a given business transaction. Today, people make those choices. In the future, says Sutor, they will increasingly be made by software. "There are a couple of different proposals that are out there, including one published by IBM, Microsoft and BEA last August. You'll soon be hearing more."

In watching Web services move forward, Sutor points to another organization that he thinks will exert a lot of influence: the Web Services Interoperability Organization (WSI.org). "WSI doesn't create standards, but considers how they should be used and what subsets of the standards will actually get the job done. People hear the word "standard" and they think it refers to a set of perfect documents. But standards aren't handed down from heaven. They are the result of negotiations, of people trying to come to some sort of agreement. And that's an imperfect process. You may have decided to include six different options, only to find out that in real life, people only use two of them. Or you may discover that the document you wrote up so carefully is ambiguous or under-specified."

Sutor says that WSI's work will, by definition, lag behind the standardization process, waiting until there are practical applications-then look at the standards and figure out what are the best practices involved in using them. Specifications will come from different places, but WSI will ultimately define what we consider Web services to be and how to use them in the best way.

But before WSI can determine the best practices for a given standard, that standard must be in place. In the case of security, the standard is proving hard to pin down because the demands are so high. What's needed says Sutor, is a particularly fine-grain level of security. "At the low end, that means encryption down to the individual elements and digital signatures that ensure authentication. At a higher level, it also means the ability to communicate privacy policy. I need to be able to say: 'I'm sending you information about one of my customers, but you're not allowed to do anything with it-he's still my customer, not yours." Privacy policies spell out whether you are free to use information in its entirety, or just parts of it, whether you can disseminate it or keep it confidential. This is a tall order, and Sutor thinks a couple of years will be needed before a complete security stack for Web services emerges.

Application development

A purist might argue that the only true Web services applications are those like Amazon's and Google's search engine APIs, which make use only of the agreed-upon "plumbing." For business applications requiring more, each vendor has filled in the gaps with its own product toolset. Hence IBM talks about applications built with WebSphere, BEA talks about its WebLogic platform, Microsoft about .Net, and Oracle about Oracle9i. All of these platforms are really supersets based on a common XML nucleus. Applications built with them will share that core, but it's unlikely they will precisely match the Web services specifications as they evolve. Those specifications are still too much a moving target.

And so the Web services applications vendors point to are really a hybrid. But they all make use of the fundamentals, and given the state of the art, they remain the best examples of what could be possible in the future. As is often the case, some of the earliest adopters are in the financial services sector. Microsoft points to the Bear Sterns and Denmark's Danske Bank, while BEA talks about an internal information "portal" developed for Oppenheimer funds managers. "It serves as a dashboard, an instrument panel, providing market research information, allowing them to execute trades and manage their portfolios," says John Kiger, BEA's director of Web services strategy.

Of course, developers have been building portals long before Web services came on the scene, but Kiger argues that the advantages of Web services are substantial. "Before, you might have implemented a proprietary interface from the portal to the back-end data source, using some existing EAI (enterprise application integration) technology or code developed in-house," he says. "Web services provides a less costly, less complex, more flexible way to make that connection. It enables them to more easily manage additions and deletions to those data sources."

But why should XML and SOAP, the foundation technologies of Web services, work any better at this integration chore than the traditional choices? One reason, says Kiger, is the comparative cost. "In this case, Oppenheimer uses WebLogic's portal, which runs on WebLogic's server, all of which include a comprehensive Web service stack. There's no need to procure any additional EAI infrastructure to connect the portal to the backend data source."

Kiger says it is easier to find programmers with XML and SOAP experience, than with any given EAI solution-and that will become even more true as Web services take hold. "And there is the inherent simplicity of Web services, which draws on a simple, but powerful idea from the Web, itself. The Web allows any user to talk to any server without having to know anything about the application at the other end. You don't need to know the operating system, programming language, or data model. All you need to know is the data format and the wire protocol. Web services applies this user-to-user model to application-to-application communications. Again, all you need to know is a wire protocol and the data format, in this case, SOAP and XML."

For some Web services applications, performance may also be a problem. The overhead for Web services messaging is considerably higher than the binary message format associated with EAI and other proprietary messaging infrastructures.," says Kiger. That overhead is due in part to longer messages-more bits that need to be sent over the network. Faster CPUs will take care of part of the problem, but ultimately, more network bandwidth may be needed. Kiger predicts that over the next year, 1000 messages per second are possible. "By contrast, Tuxedo over a binary connection has provided 10,000 messages/second in a credit card implementation. But that's an example of a high-end messaging application and represents the extreme. For most enterprise integration scenarios, supporting hundreds or thousand of transactions is more than is required."

Sidebar: The View From Microsoft

In the market for Web services development tools, there's Microsoft's .Net and then there's the rest of the industry, whose tools are based on Sun's J2EE (Java 2 Enterprise Edition). Microsoft has gone its own way by largely ignoring the J2EE specification, introducing C# as a substitute for Java, and emphasizing Windows over a more platform-neutral approach.

Microsoft did not start out on the best footing with .Net. It used the name in so many products, that its initial marketing efforts have become a case study in how not to promote a brand. Some developers believe that Microsoft's architecture has not withstood the trial-by-fire of Java, and is thus not as mature.

On the other hand, Microsoft's vast installed base and tenacious product refinement has made it a major player. A Gartner Dataquest survey taken last year found that .Net was favored over J2EE products by companies planning to use Web services. Another Gartner Dataquest survey of systems integrators named Microsoft's .Net, IBM WebSphere, and Oracle's Oracle's 9i as the top three development products. .Net in the lead.

And Microsoft has put .Net front and center. "It's one of our key initiatives; it is part of our long term strategy," says Rebecca Dias, product manager for advanced Web services at Microsoft. "There's a huge challenge for building service-oriented architectures within and beyond the enterprise. .Net is core to solving that problem." Dias is a self-described ".Net convert," having come from the J2EE world, where she worked on Ionoa Technologies XMLbus platform.

I spoke with her by phone. Here are some excerpts:

People say that Web services are good for connecting applications, but not yet for full-blown business class applications. What's your take?

I believe that the needed specifications are getting nailed down, and that our architecture already provides it. For example, we're showing great amounts of performance using Web services in the .Net Framework, for instance.

What about security? WS-Security is not yet an official standard.

We have customers going into production with WS-Security implementations today. So I don't think security concerns should be holding people back. But while WS-Security is a critical part of the overall infrastructure, it's not the only one. If you want to have conversations between multiple parties where you are sharing security context information, for instance, there's a new specification called Secure Conversation and Trust.

What about another proposed specification, WS-Reliability?

WS-Reliability is interesting in that it attempts to address the problems in terms of doing reliable messaging over the Web. But it doesn't fit with the rest of the architecture. Microsoft and some partners [BEA, IBM, TIBCO] have released an alternative, WS-ReliableMessaging that is designed to address the needs of a broad service-oriented architecture, not just business-to-business.

Microsoft has been accused of being late to the Web services market. Is that a fair assessment?

I beg to differ. We didn't throw some garbage code over the wall. We're spending a lot of time doing very robust integration, testing to make sure that Web services are core to the infrastructure and facilitates you actually use. If you want to have Microsoft Outlook talk to your back-end CRM system, we can do that for you.

Outlook is a Windows program, and some have argued that .Net is Windows-based and is not UNIX focused. Is that a problem?

The beauty of Web services is that it facilitates integration regardless of the underlying platform. We've demonstrated time and time again our ability to provide solutions to integrate with UNIX systems and mainframes. In a three month development cycle at Danske Bank, we helped save a million dollars a year by offloading transactions to the mainframe. Of course, we're providing tools and functionality for our platform, but we play well with others.

Gartner Dataquest contends that higher-end mission critical applications are usually done on UNIX and that's where you see J2EE doing better. Is that J2EE's strength?

I'm not sure I completely agree that J2EE has a lot of mission critical systems. If you do more research into what Gartner and Forrester say about J2EE, the predominant usage scenario is using a servelet engine to facilitate portals. As far as mission critical systems go, the UNIX based systems are C++, some Java, and often CORBA. It's a heterogeneous world out there: mission critical applications are not on one platform or another-not J2EE, not UNIX, not just the mainframe, and not Microsoft Enterprise servers.

In terms of Web services, does the name ".Net" equate with Microsoft's Web services offering?

Think of .Net as a strategy for delivering software. It's the vehicle by which we help the customer gain interoperability, performance and other advantages from the Windows platform by integrating it in a heterogeneous world. The .Net Framework is the techno geeky core foundational elements by which somebody can build applications that are ".Net connected." Web services are a subset of the .Net Framework.

Where are Web services headed?

If you look at the protocol stack, I believe that history has been fundamentally changed, but I don't think people realize it just yet-and won't for another five years or more. The fact that we are standardizing the protocols for doing secure, reliable, transacted communication is a pretty phenomenal event.

Why five years?

It's always in retrospect that people go "Wow!" It was the same with the Internet. Once people build services and are able to reuse those services in a business context, they are going to say "Wow-this just wasn't possible ten years ago."