Last March, the research firm Venture Development Corporation had a surprise for the embedded operating systems industry. In VDC's rankings for 2002, Wind River Systems-the long-time embedded OS leader-came in at a virtual dead heat for overall market share with none other than Microsoft. This come-from-behind feat has been in the making for a few year now. Microsoft offers two embedded operating systems: Windows CE .Net (formerly called Windows CE) and a modular version of XP. The company has aggressively priced both, bundling them with integrated tools and add-on components that don't require additional licensing. As a result, development costs under Microsoft embedded operating systems are significantly lower than under Linux, according to a recent study by Jerry Krasner of Embedded Market Forecasters.
Microsoft's successful foray into the embedded space casts further light on an industry that many observers believe will be the future of software development. There are only so many desktop PCs in this world: even if you are a fanatic road warrior, you only really need a couple. By comparison, the opportunities for embedded systems applications, running on everything from microwaves to automobiles, seems unlimited. And the embedded OS vendors are fighting it out for market share.
"The market is extremely competitive," says VDC analyst Stephen Balacco. "Just a few years ago, there were just a handful of suppliers, primarily with real-time operating systems. Today you have companies like MontaVista providing Linux distributions, as well as Microsoft and Symbian." He says the top concerns for developers choosing among these OSs is to control development costs and get to market fast. "The vendor strategy we're seeing is exemplified by Wind River, who is providing integrated platforms targeted at specific domains: telecom, automotive, consumer electronics, etc. The platforms include all of the components an OEM would require, integrated so that they can accelerate their development process."
Nonetheless, says Balacco, Wind River has been challenged over the last two years. The telecommunications sector, where the company has been strong, is slumping. Linux is making inroads. And then there is Microsoft. "Our research shows that Microsoft is gaining momentum. A little over two years ago, the company refocused its attention on the embedded space, establishing the Embedded Appliance and Platform Group [which, in the latest reorganization, is now the Mobile and Embedded Devices Division]. That was a signal to the market that they were serious about the embedded space, and they have been gaining momentum ever since."
Balacco says that Linux in the embedded space has also had an impact. "When Linux came on the scene, it had some of the traditional embedded operating system companies like Wind River, Green Hills Software, and now Microsoft taking a look at not just their products, but their business models. It's as if these companies looked at open source and realized it has some attractive things to offer, like releasing source code, being more flexible in how they price their products, and building a collaborative community of developers."
Microsoft, for example, now reveals some 2 million lines of CE code to developers on a read-only basis. The idea is that you can better optimize your code if you understand CE's inner-workings. On a select basis, Microsoft will also grant OEMs the right to modify and redistribute code. None of this is pure open source development, of course, but considering this is Microsoft, it's a large step.
The embedded OS universe
Embedded OSs fall roughly into three categories:
- Proprietary real-time operating systems (RTOSs) like Wind River's VxWorks and Green Hills Software's Integrity. This category includes the first wave of embedded OSs to come on the market after OEMs stopped building their own in-house.
- Non-proprietary operating systems like Linux (from companies like MetroWorks, LynuxWorks, and TimeSys) and ITRON.
- Single-platform/
single-sector operating systems like Symbian, which mostly runs on mobile phones (Fujitsu, Nokia, and Sony); as well as Psion handheld organizers; and Palm OS, which runs primarily on Palm PDAs and PDA/ phone combinations. Symbian and Palm rank number three and four on Venture's 2002 embedded software list.
Where does Microsoft fit? Here the lines get blurred. While Microsoft CE .Net is best known for its use on palmtops, Microsoft argues that CE is in fact a true, real-time operating system, competing directly with VxWorks. And VxWorks-the best known RTOS-runs on many applications that don't require real-time performance. As for Windows XP Embedded, it is Windows XP divided into some 10,000 parts. Developers select the functionality and a Microsoft-provided tool pulls the necessary components to deliver it. The resulting OS footprint is typically 64-120MB, including browser, shell and networking stack. (Microsoft provides a similar tool for CE.) Hence, Windows XP Embedded is a proprietary cousin of embedded Linux, another desktop operating system scaled for embedded use.
Windows XP Embedded only runs on X86 architecture CPUs and tends to be used in systems like kiosks, thin clients, and point-of-sale terminals-platforms that resemble a desktop. By contrast, CE runs on four processor architectures: X86, ARM, MIPS, and the Hitachi-developed SuperH. CE's footprint is much smaller than Embedded XP, typically 8-20MB (1-2MB, say, for a residential gateway, 20MB for a settop box), while affording four-to-five times the battery life. As a result, it is CE, not Embedded XP, which competes in the high-volume consumer electronics market.
Anywhere but a PC
Jerry Fiddler, co-founder and chairman of Wind River Systems, defines an embedded OS as "the operating system controlling the computer inside something that's not called a computer. It does what a desktop operating system does, which is to manage the hardware and provide a platform on which applications can run." VxWorks may well be the most widely implemented OS in terms of the variety of platforms it runs on. The OS is part of Honda's ASIMO robot, the European Space Station's PROBA satellite, Apple's 802.
A stricter definition of the term "embedded" is suggested by Dan O'Dowd, CEO and company founder of Green Hills Software, a Wind River competitor. "What embedded means is that there is no user interface to the computer," he says, stating what has become a minority view in the industry. "That leaves out consumer devices of all sorts. The primary purpose of most devices that use Linux is to communicate with the user, whereas, in an embedded system, the primary purpose of the operating system is to communicate with mechanical equipment. Most of the devices on the market are just small computers-preprogrammable devices with a display and a human interface." Green Hills Software, which ranks number seven on VDCs vendor list, got into the RTOS business in 1996 as an adjunct to its developer tool business. Its first customer was Boeing for the B1 bomber.
The first embedded OSs were built in-house, but as microprocessors grew more powerful and embedded applications grew more complex, embedded OSs became a specialty business, much the way desktop operating systems have. And the financial expectations for embedded operating systems has also approached their desktop counterparts. If you could get your embedded OS on half the mobile phones in the world, charging a dollar a unit royalty, you would replicate the success of Windows. The problem, though, was that high-volume manufacturers balked at paying high licensing fees, and now, the more typical pricing model is to charge by the developer seat, perhaps on a subscription basis for tools and service. That's often how embedded Linux and traditional RTOS distributors make their money. In the case of Symbian, the SDK is free, with free and for-fee support offered by Ericsson and Nokia, both of which are interested in attracting developers. Microsoft, on the other hand-a company that understands the value of royalties-charges a relatively low flat fee for bundled OS, development tools and applications, as well as run-time fee per unit.
One of the biggest differences between an embedded and a conventional operating system is the footprint (the size of the final OS image residing on the microprocessor), which varies according to the platform and application. "You want exactly what you need and no more," says Fiddler. "Anything extra costs you money, battery time, heat dissipation and processing power. Hence, every embedded OS implementation is different.
According to research firm VDC, Wind River, despite the competition with Microsoft, "maintains a commanding market leadership position" for the sale for RTOSs-real-time operating systems. "Most operating systems you can't make any guarantees about time," says Wind River's Fiddler. But if you're piloting a jet and pull back on the control wheel to dodge a mountain, you expect an immediate response from the aileron. Similarly, the firing of a sparkplug in an automobile and the robotic devices on an assembly line all require events happening at the right millisecond. "Basically, a real-time operating system makes guarantees about timing. An RTOS gives you a worst case scenario about how long it takes to get sensor input, or an interrupt, as well as how long it will take to provide a response."
Fiddler says that historically, the terms "embedded operating system" and "RTOS" have been used interchangeably. But in fact, true real-time performance may not be an application requirement, and most of the time-argue the embedded Linux vendors-absolute real-time performance is not an absolute requirement. Indeed, the industry has begun using the term "hard real-time" for a traditional RTOS and "soft real-time" for an embedded OS that is, well, something less, but good enough for most applications.
Green River's O'Dowd thinks that's a contradiction in terms. "The definition of real-time is the ability to always respond within a deadline," he says "'Soft' real-time means that sometimes the system doesn't respond within deadline. And that's like being slightly pregnant. For that reason, nobody puts Linux in engine controllers or navigation systems. What they are mostly doing is putting it on digital recorder recorders-and that's just a computer."
Scott Hedrick, senior product marketing manager for embedded Linux developer MontaVista Software says Linux is catching up in terms of throughput. "A few years ago, there was a huge gap between where Linux was and where it needed to be," he says. "We have collaborated within the open source community to close that gap. We're now seeing real-time performance that is so close to the legacy RTOSs that most customers are satisfied. We've had Japanese customers developing high-speed networked printers, cell phone companies and others have done extensive benchmarking." Of course, none of these examples would qualify as "hard" real-time.
Monetarist's largest presence outside the U.
ITRON is known as a lean, dense OS with a small footprint. Linux, with its desktop heritage, was developed for the expansive space of a hard disk. "Linux does require a minimum memory size and CPU speed: a 32-bit microprocessor with MMU [memory management unit] and at least a meg or two of RAM or ROM," Hedrick says. "But we have customers not only developing highly reliable consumer electronic devices, but medical devices, and military aerospace devices. Nokia and Alcatel have developed rack-mounted telecommunications carrier boxes providing mobile phone services."
As for Microsoft, it argues that Windows CE .Net is in fact a hard real-time OS, meeting the requirements of most real-time applications. But while the OS is used in some manufacturing applications, its status as a working RTOS in real applications pales in comparison with the traditional RTOS vendors. In terms of market share, Wind River remains the dominant RTOS vendor.
Linux and Embedded Windows-two factions fight it out
Consumer electronics is the high-volume market for embedded systems, the place where most OS vendors dream of success. It is here where Linux and Windows are fighting it out. Last July, embedded Linux got a strong endorsement from a newly formed consortium of consumer electronics manufacturers called the CE Linux Forum, or CELF. The eight founding companies are Matsushita, Sony, Hitachi, NEC, Royal Philips Electronics, Samsung Electronics, Sharp and Toshiba-with IBM, at this writing, planning to join.
CELF will largely be in the business of setting member requirements and pushing the embedded Linux to deliver them. Initial goals include speeding the startup and shutdown times, improving real-time capabilities, reducing ROM/
"I hope it will be a big advancement for Linux in the consumer electronics space," says Tim Bird, who co-chairs CELF's architecture group and is a senior staff engineer at Sony's Silicon Valley facility. "Linux has already started to make some inroads." Sony has used it on the Sony Cocoon digital video recorder, as well as the Sony StorStation tape backup system and in a video switching box. "Some of these are not high volume consumer products, but we're certainly moving in the direction of putting Linux in more things."
But not everyone is convinced that Linux will become the de facto standard for embedded systems-or that it is the best solution for smaller developers. A report by Jerry Krasner, a long-time observer of the embedded market, argues that the cost-of-development under embedded Linux is substantially higher than under embedded Windows. (Krasner has admitted that Microsoft paid him to develop the framework and analyze the data gathered by a "very reputable, very large, third party." He said Microsoft approached him because he has already been writing that, based on conversations with developers, embedded Linux came with a high hidden price. This seeming conflict of interest does not seemed to have tainted his findings, however.)
Evaluating data that included a survey of 456 embedded developers, Krasner concluded that the cost of development using Windows embedded platforms is roughly four times less than with embedded Linux. "The average Total Cost of Development for OEMs using Windows XP Embedded, Windows CE.
Why the discrepancy? Krasner argues that a big missing ingredient under Linux is a truly integrated application development environment or IDE. Windows, he says, offers "a completely integrated IDE for operating system development and Visual Studio .Net for application development." Tools include optimizing compilers for multiple CPU architectures for C and C++, which can optimize the tradeoff between code size and execution speed. Also included, integrated language-sensitive editors for code development, a graphical user interface, graphical debugging capability and automatic application building tools.
Linux, he writes, has nothing comparable. If you purchase the OS from a vendor like Red Hat, you must install the GNU Pro development environment-a fully integrated IDE-as well as the Red Hat Linux environment. But the code editing and build tools are separate from the GNU Pro tools, and must be assembled and maintained by the design team. "The a-la-carte Linux development environment would consist of downloading everything from the Free Software Foundation, or other Linux distribution, web site and assembling them into a team development environment. This would include downloading the GNU/
Moreover, the Windows IDE is substantially cheaper: $995 per seat versus an average of $3,899 for embedded Linux, with maintenance included. Indeed some vendors license their tools instead of selling them, and here the price is even steeper, averaging $17,222 per year. In addition to tools, Microsoft also bundles applications, real-time support, services and codecs. Most Linux distributions do not. Krasner estimates that the only way an OEM could compensate for the cost discrepancy is in volume, shipping more than 389,000 units to make up the higher up front cost.
Todd Brix, a lead product manager for Microsoft's Mobile and Embedded Devices Division, says that its pricing is a deliberate strategy to move costs to the post-development back-end. "We have a more vested interest than our competition in getting our OEMs to market quickly. If a vendor bills on a subscription basis, the longer the project goes, the more money a vendor makes. We have an incentive to get our customers to get to market quickly, and ship in high volumes. We only make money if OEMs are successful."
Both Microsoft embedded OSs come in modular form, with "pre-tested, footprint-reduced components" that you put together for a given application. Embedded Linux does not, and developers must work harder to reduce the size, writes Krasner. And Embedded Windows does a better job of encouraging serial development-enabling developers to build the OS, application and hardware in parallel, due in part to emulation tools. Under Linux, "serial development is more the rule than the exception."
Tom Williams, editor of RTC Magazine, which focuses on embedded industrial systems, thinks Krasner's report shows that Linux's perceived lower cost is largely a myth in the embedded space. "Do you think there's no run-time charge for Linux? Wrong. There's no runtime charge for the kernel," he says. "But if you want to build in something like an MP3 player or an encryption engine-someone has written that to sell, and you may pay a dollar or two per module to license it from a third party or the Linux vendor." By contrast, he says, "when you buy a distribution of CE, you get all this stuff for the price of the distribution. It's all certified to work with the kernel. And if you decide not to pay runtime royalties, but instead pull distribution off the Web, you spend precious time testing and integrating, rather than clicking on it and building it into the operating system image."
Williams says that Krasner's report explains why the big companies behind CELF can embrace embedded Linux, while smaller firms should take a closer look. "These companies are going to sell in large volume. If they put embedded Linux into a settop box and cranked out a million of them, they'd save money. They'd save money if they bought their own RTOS. But the average shop doing products for the embedded market will get killed by these price differences."
Even in the real-time space, Krasner argues, embedded Windows has an advantage. Linux has no unified real-time Linux model, and that's keeping the OS out of sectors where the proprietary real-time operating systems shine-embedded mission critical systems, aviation, and some factory automation." Embedded Windows is not a big player here yet, but that could change, Krasner says. "If you are building factory controllers, or instruments-you really have to take a look at the economics of your own business models, rather than that of the RTOS vendors." And what about consumer electronics devices? "I'd probably use CE, and not because I'm an advocate of Microsoft," he says. If a CFO of a mid-size company read Krasner's report, then heard his engineering manager plead that the developers still wanted to go with Linux-"he'd chop his head off. And that's the polite part of the anatomy."
Sidebar About CELF: A conversation with Sony's Tim Bird
Tim Bird is co-chair of the Consumer Electronics Linux Forum architecture group, and a senior staff engineer in the Linux Architecture and Standards Group of the Platform Technology Center of America for Sony Electronics, based in San Jose, California. He was joined by his colleague, Thomas Swidler, CELF's executive director.
Why a consortium for embedded Linux?
The biggest reason is that as Sony and Matsushita looked around the industry, we saw that other consumer electronics companies were working with Linux. A lot of the time, these companies will work internally on projects to extend Linux, but they will eventually publish anyway because of open source licenses. We thought it would be a good idea to collaborate up front in terms of the drafting of specifications and reducing duplication of effort. We're not competing to improve the operating system, we're cooperating to improve the operating system.
There's been some grumbling about a lack of standardization for embedded Linux.
How will you work with Linux vendors?
Members will work together to develop specifications-a list of requirements, and possibly definitions of APIs. Then we'll look for existing open source projects. We'll see if a member company has done or is willing to do some of the work, Then we'll solicit an implementation. It could be from an existing open source project--it doesn't have to come from a member. The by-laws say that if there isn't an existing implementation, we may ask one of the member companies if they'd be willing to produce one. We put no limits on how that might be done. The member company could produce it themselves, or they could go to an outside company and have it contracted out. I view this as an opportunity for the expansion of the embedded Linux market, which is a real opportunity for the Linux vendors. And there probably will be opportunities for contract work that come out of Forum activities.
Will the expansion take place under the same licensing as mainstream Linux does?
Yes. It's all under the GPL and LGPL. In the actual membership agreement we say "under a license that meets the terms of the open source definition as published by the Open Source Initiative, including but not limited to the GPL or the LGPL." We're a bit more broad in the language because every once in a while something does come up that's not GPL. There are major components like Apache and X that are not GPL, so we wanted to make sure that we were inclusive of Open Source licenses. But we're not trying to change any license.
Does CELF represent a de-emphasis on ITRON and an increasing emphasis on Linux?
Yes, it's a natural evolution. I don't think you're going to see a sudden dramatic drop in ITRON-based products. But if you look at the capabilities of the OSs, Linux is really a more fully functional OS. It's got memory protection, very high levels of abstractions in terms of interprocess communication mechanisms, virtual memory, and good separation between user space and kernel space. Linux is essentially a multi-user, multi-processor system. That's important, because as embedded consumer electronics devices grow in complexity, you need a much more full-blown, full-featured OS to handle the requirements.
Is Linux boot-up time a concern for consumer devices?
You of course gain some speed when you don't boot from a hard disk. But there's are other ways we can save time, as well. In the desktop space, Linux and other operating systems do a lot of probing of their environment. They have to configure the buses, probe for attached hardware, see if new hardware is attached, and do lots of calibrations. That's because, as general purpose operating systems, they have to adapt to a myriad of different configurations. The embedded world is much more of a fixed solution. So one of things we're doing is finding places where we can cut out auto-probing to shorten the amount of time it takes for Linux to come up. Our initial goal is to get under one second boot up time.
How confident are you that you'll actually be able to achieve it?
We have made some progress. There will be some code fed into the Forum right at the start from some work that Sony and Matsushita have done previously. We are seeing some times that are very encouraging. The bootstrap times are broken up into several different phases. There's the amount of time for the firmware, the OS, and the initial applications. In that middle phase we're seeing the total amount of time for the OS to boot up to under 500 milliseconds.
Thomas Swidler: I want to throw in the caveat that consumer products are becoming more complex and more networked together, so they will also require more PC-like features. Therefore, it is a big task for us, and its where the Forum can be of help.
What about power management?
Bird: There's a range of issues. There's how much power the device takes on when it's active; how often it goes to sleep. There are a lot of interesting techniques being used now to lengthen the amount of time that you can get out of your battery charge. Most of the modern processors now have frequency scaling, where they adjust the speed of the CPU clock so the processor is running faster or slower depending on the power requirements at the time.
Do members view Japan and North America as one big market? Or are there differences?
Bird: For the purposes of implementation I don't think that geographic difference is a big factor. A lot of these changes are fundamental and independent of whatever language or product category you are talking about. If you're trying to speed up boot up, there are some certain technical issues that don't matter which market you are playing in.
Swidler: The initial member companies may be primarily Japanese, but they have many subsidiaries in different countries. Plus we have a lot of membership requests coming in from all over the world. I don't see this as a strictly Asian initiative over the long run. It's a global effort.
Are you predicting that Linux, as opposed to anything that Microsoft produces, will become the de facto standard for consumer electronics?
Swidler: Everybody wants us to quote on that.
Bird: The goal of the Forum is to make Linux as useful as possible for consumer electronics products.