More than 10 years have passed since the New Zealand-born Mark Billinghurst met Japan’s Hirokazu Kato at the University of Washington in Seattle. The two researchers had different academic backgrounds, and were separated by culture and to some extent, by language. But they also had converging interests in the field of augmented reality, or AR, in which virtual imagery is overlaid on real world objects. Their work together includes a 2002 paper: “⁠Collaborative Augmented Reality,⁠”⁠ whose title also touches on their relationship―a collaborative friendship that has advanced the careers of both men.

Among software developers, Kato and Billinghurst are best known for ARToolKit, which was largely written by Kato, and ported to Windows and documented by Billinghurst. The library solves one of the key problems of augmented reality by enabling an AR application to determine where users are looking in the real world. This year on the tenth anniversary of ARToolKit, the IEEE virtual reality conference gave Hirokazu Kato its Virtual Reality Technical Achievement Award in recognition of the widespread impact of the library. The two men also worked together on the ideas behind Magic Book, an AR project in the form of a children’s book, which won Billinghurst a 2001 Innovation Award in the entertainment category from Discover magazine.

Billinghurst now directs the Human Interface Technology Laboratory New Zealand (“⁠HIT Lab NZ⁠”) at the University of Canterbury in Christchurch. Kato heads the Interactive Media Design Laboratory at the Nara Institute of Science and Technology. But they have not let the kilometers separate them. Kato spent six weeks visiting Billinghurst in New Zealand and Billinghurst is now preparing to spend three months in Nara as part of a sabbatical.

I spoke with Billinghurst by phone from his office at the lab.

How did your collaboration with Hirokazu Kato begin?

We got started in 1998. I was working on my PhD at the University of Washington, while he was a visiting faculty member from Osaka University. I wanted to work with him because we both had an interest in computer vision, but he was also quite interested in augmented reality. We had a great collaboration, including developing the first version of the ARToolKit tracking software, which came out of research into a augmented reality system employing computer vision. The following year, we demonstrated it at the 1999 SIGGRAPH conference, as well as making the first open source version available.

You are from New Zealand, but were studying in the United States and collaborating with a Japanese professor. What was that like?

We had complementary skills. He has fantastic skills in computer vision research and software development, and I was strong in interface design. We were both hard workers, willing to put in the hours needed to get everything done. The cultural differences were not as great as you might think. New Zealand has some strong connections with Japan, and since that time, he and his family have come to visit a couple of times, which I’ve enjoyed very much. I think Japanese people like visiting New Zealand because our country is somewhat like Japan. We too are a country of long islands with mountains and a strong connection to the sea―but without so many people.

How did Kato-san’s interests mesh with yours?

When he first arrived, he had no experience with augmented reality but wanted to do some work with computer interfaces. I was doing augmented reality, which he was interested in because of the computer vision components. So I helped him enter a whole new area of research, which in turn helped his career in Japan. Two years ago, he started his own lab at the Nara Institute of Science and Technology. I also helped him publish a number of papers in English, which is sometimes difficult for Japanese researchers to do.

Generally speaking, the Japanese are very strong at building devices and doing systems engineering, whereas the West is sometimes stronger at interface and experience design. Japan has amazing hardware and robotics that plug into AR applications. That generality applied to our collaboration. He has been strong on the software engineering side of things, and I’ve been able to help with innovative interfaces and applications.

Your PhD research was on the topic of “collaborative augmented reality.” What does that mean?

I began with an interest in medical virtual reality, but was soon drawn to augmented reality and computer-supported collaborative work. I noticed that at that time nobody had combined those two areas of research. So collaborative augmented reality seemed like a great opportunity for my PhD thesis. The basic idea is that you have people sitting around a display table, seeing a virtual image, and doing collaborative work. We developed ten or so applications that showed different ways to do this, both face-to-face and remotely.

We began with a card game of “⁠concentration⁠”⁠ where virtual is displayed is on top of actual cards. If you put, say, a virtual cowboy and a horse, you’d see an animation. To make this work, the game needed to “⁠know⁠”⁠ the location. That’s where ARToolKit comes in: each of the cards had the ARToolKit tracking symbol. The main point of the game was collaboration: you could play with people you didn’t know. It was a simple idea, but when we showed it at SIGGRAPH, about 2,500 people stopped by to play.

You and Kato-san also collaborated on the ideas behind Magic Book. Do you see play as an intermediate step to more serious applications?

People can learn a lot about AR through play, though there are indeed more serious applications, as well. For example, architects work together on a 3D model of a house. AR technology hasn’t been picked up by the commercial sector as quickly as I would have expected, but games themselves are a huge potential market. There’s at least one good example: the Eye of Judgment title on the Sony PlayStation 3 is the first console game employing augmented reality. The interface is actually a set of physical cards, each with its own associated virtual reality monster. You stage fights by putting the cards together. Sony has sold about 300,000 copies since releasing the game in October 2007. There will be at least two more commercial AR games entering the market this year.

Maybe Japanese companies will be the first to seize AR commercial opportunities. I noticed that an ActionScript 3 version FLARToolKit tracking library, which supports Flash, was also created by Japanese developers.

About a year or so ago, Ryo Iizuka and others created NyARToolkit library, which ported an optimized version of ARToolKit to Java and C#. Tomohiko “⁠Saqoosha⁠”⁠ Koyama derived FLARToolkit from there. The software has become quite popular because of interest in using AR for online marketing applications via a browser. With FLARToolKit, millions of people can have their first-ever AR experience.

Are you surprised how widely the ARToolKit has traveled?

One of the benefits of doing open source development is that people can do things you never expected, such as making a Flash version. When we first worked on ARToolkit, we weren’t sure whether to make it open source―we just decided to try it. That turned out to be a good decision, because the library has spawned a huge number of applications. The AR community owes a deep debt to Hirokazu Kato for doing that and for making AR technology easily available.

You are also involved in a commercial venture: ARToolworks.

Yes, I am one of the founders of ARToolworks, which has the commercial rights to ARToolKit. The company’s president is Thomas Furness, who was one of my two supervising professors at the University of Washington and is a founding father of virtual reality. I’m part of the management team; Hirokazu Kato is a friend of the company and does research for us. How this all works is that we use a dual-license model. We make ARToolKit available for free under a GNU Public License for non-commercial use, with a proprietary version for commercial, closed-source development.

That seems like a good combination for many open source projects.

ARToolworks is smaller than some other companies founded on the same model, but we not receivied any venture capital or investor funds. However the company has been around for almost 10 years now and is doing quite well. So I do think that a testament to the value of building commercial companies off of open source.

Where is augmented reality headed?

It’s headed in a number of directions. Primarily because of FLARToolKit, we’re seeing huge growth for online marketing. And we’re also going to see strong growth in the mobile phone space, because with mobile phones, you already have all the hardware necessary to provide an augmented reality experience―including graphics, display screen, processor, and a camera. With a billion mobile phones sold every year, there’s a huge opportunity. We’re also seeing several interesting niche commercial areas: particularly medical and architectural design. For example, it is possible to superimpose medical data, so that, for example, a neurosurgeon could look into a patient’s skull and see the actual tumor superimposed with 3D medical data.

That sounds to me like a virtual reality application, which was a media darling for a while before people dismissed it as over-hyped. Is AR becoming “Virtual Reality 2”?

I was around back then when everyone was predicting that in five years, we would all be walking around in the real world with head-mounted displays. And of course, in hindsight, that was all rubbish. But as it turned out, the interactive 3D technologies that started from virtual reality have been commercialized in other ways―in game consoles and 3D graphics applications. Even the Nintendo Wii with its motion controller can be tracked back to the controls used for early virtual reality systems used 10 years ago. And of course, there’s a thriving market for high-end visualization systems that combine large, immersive projection walls and 3D graphics.

I think the same thing will happen with augmented reality. The initial applications won’t necessarily be the ones that become commercialized, but in five or ten years⁠’⁠ time, there could well be a multi-billion dollar market that could be traced back to the research we’re doing now.

You are planning to resume your face-to-face collaboration with Kato-san by spending three months in Nara. What will you be doing?

I want to look at enhancements to ARToolKit as well as natural feature tracking software, which does AR tracking from normal printed images without requiring a physical marker, as is now the case. ARToolKit is getting quite old now, though we release new versions every year or so. Even so, over the last five years, we’ve had 195,000 downloads―which is a huge number for a software this specialized. I’m looking forward to the visit and doing some great research with Hirokazu Kato.

You’ll be in Japan for a while. What do you think of the food?

I love Japanese food. Not all of it, but most of it: some of the Japanese candies are a bit strange for me.

But otherwise, you seem to like the ingredients of your collaboration with Hirokazu Kato.

I think our collaboration is a great example of what can happen when you mix together people with different cultures and expertise. You can get some wonderful creative synergy and serendipitous results that might not have otherwise surfaced. When we met each other 10 years ago, neither of us set out to build an open source tool that would be used by hundreds of thousands of people. But we had a great research and personal relationship, and because of that, we were able to generate and explore interesting ideas, ultimately creating a software library that is the most widely used tool in field of augmented reality.