• by Steve Smith , Staff Writer @popeyesm, July 14, 2009

Is augmented reality the next step in the post Wii world of gaming?

From the first days of Pong, the video game industry has been on a relentless hunt to make virtual worlds look, sound and move more like the world we experience - or at least fantasize about. Now, a new coterie of computer scientists and avant-garde marketers may reverse the polarity of that trend and give those virtual worlds a dose of reality.

Decades after Pac-Man was a barroom hit on antediluvian "video game machines," researchers at the Mixed Reality Lab at the University of Singapore play the venerable game physically on the sidewalks of the campus (do not mix with alcohol). Backpack-mounted pcs calculate their positions on a game board that is overlaid onto the streets, and 3-D images of the game's cartoon ghosts and munchable cookies are superimposed on their view of the world through head-mounted displays as they move in real time and space to capture or evade one another.

Computer scientists at Georgia Tech's Augmented Environments Lab have put live sequences and scenes from the virtual world Second Life into a live camera view of an ordinary room, so that rendered 3-D avatars can be seen walking through real environments, and actions in the online world can be seen occurring on a tabletop. This fall, Sony will try to marry some of these concepts into a marketable pet sim for the Playstation 3. Using the Eye Toy cam accessory on the game console, Pet Toy will insert a 3-D interactive pet onto your living room table and register your hand movements as you interact with the creature. Sims ... meet world.

Reality: The Game
The world of Augmented Reality (AR in geek-speak) may be at the bleeding edge of computer science and game development, or it could just be off the edge. ar combines a host of still-evolving technologies like motion detection, geo-location, and object recognition in an effort to blend real time 3-D renderings into a video stream so that people in real time and space can seem to interact with them. "In an ar application, one of the key things is to have virtual images combine with the real world in such a way that you feel the computer graphics are part of the real world," says Mark Billinghurst, director of the Human Interface Technology Laboratory at the University of New Zealand and developer of ARToolworks, the leading software tool kit for building AR projects. In most cases, AR is achieved with a combination of video camera and software that can recognize and track a real-world object. The software uses a four-cornered marker on the object, or points on the image to determine its position, and then superimposes in the video stream 3-D objects that will move and reorient in tandem with the markers or the camera view.

AR would seem to be a natural area of inquiry for the traditional video game industry, but early attempts have been mixed. Eye of Judgment (2007) for the PS3 was a battle card game that used the Eye Toy video cam to overlay animated creatures on the cam's view of a tabletop card battle game. It failed to gain much traction in the market, though it garnered fair reviews from critics. Companies like Billinghurst's, along with Total Immersion in Los Angeles and metaio in Germany have been deploying AR for the past year in a series of high-profile marketing campaigns for toy and auto companies, but surprisingly the traditional game companies have been slow to embrace the concept.

Part of the problem until now has been the technology itself. "I haven't seen anything yet that doesn't feel buggy," says Josh Lovison, gaming and mobile lead at ipg Emerging Media Lab. "There is nothing like the software not working the way it was intended. That is what has held it back the most."

But the technologies are evolving at a fast pace. AR promo campaigns for the new Star Trek film and in-store ar animations that pop up on Lego and K'NEX toy boxes have gotten major video game companies cautiously interested again. "They are still living in this 3-D immersive world," says Bruno Uzzan, CEO of Total Immersion. "They don't want to let the real world in."

Working with Topps sports trading cards, Uzzan's company implemented probably the most successful AR integration thus far in the entertainment and gaming space. Baseball card collectors can hold a player's card in view of their Web cam and see on screen a 3-D rendering of the star floating atop the card, ready to play rudimentary batting and catching games online. Uzzan says that the makers of real-world toys have been quicker to see the possibilities of AR than virtual-bound game companies. "Video games don't like the real world; everything is fully virtual," he says. "Toys need to get into digital. They need to find a way to bridge their real products with digital components."

The Future's in Your Hands
But after decades of complete immersion in synthetic realms, shouldn't electronic gaming start exploring the world it left behind - old-fashioned physical interaction? Of course, despite their reputation as isolated activities for poorly socialized loners, most of us now understand that video gaming is a very social activity. We tend to play with others in the room or across the Web in massive multiplayer contests.

"But what is the essence of board games that people aren't getting from computer games?" asks Blair MacIntyre, associate professor of computing at Georgia Tech and a leading ar researcher. "It is a social experience that is very different from being heads down, fast-paced and competitive." MacIntyre has developed experimental tabletop games like BragFish, where people use handheld PCs and mobile-phone cams to pilot virtual boats and bring their fishing catch back to shore, while all of them view the virtual activity on a common real-world plane. "They get a sense of what other people are doing on the game board. They are chatting and socializing, and the pace of the games is different," he says. ar games have the potential to expand the universe of entertainment experiences and knit virtual and physical, Internet data and real-world objects in groundbreaking new ways.

Independent game developers have been talking to MacIntyre, because they see AR as a way to innovate and stand out. But he admits that the major game manufacturers "really are not interested in anything that can't be delivered on a huge scale." For now, ar games require extra steps of connecting Web cams to consoles and using game pieces that bear oddly-shaped markers the cam can track. To borrow geek-speak, AR is still kludgy.

The increased power of smartphones could change everything, however. All of the elements AR needs - a camera, a processor and a display - are integrated on a single device that has the added advantage of being portable in the real world. "This is going to matter hugely in mobile gaming," says Lovison. He foresees the phone acting as a technology hub that gives the real-world game GPS data for positioning, pulls game and multiplayer information from the Internet and drives virtual imagery into eyeglasses with AR screens. "Where kids used to play with Nerf guns, if you have any phone camera and glasses you can turn any world that is augmented into a game to play. I do think it will get there."

Billinghurst agrees that handhelds may be the frontier on which AR will prove itself. His ARToolworks development kit has been demoed for iPhone applications, but the company had been waiting for Apple to announce support for video so applications like his can apply virtual graphics onto the phone's live view. "When Apple announces support for video, that will allow ar apps, and after a few months you will see AR there," Billinghurst says.

MacIntyre's research shows that players get involved when their mobile devices become small windows onto a tabletop or gaming space with virtual imagery. Current implementations like Eye of Judgment and the Topps 3-D baseball cards create an unnatural third-person view of yourself manipulating real and virtual objects. "With handhelds you get a first-person view and have the illusion of looking through it," MacIntyre says. The mobile phone allows us to get closer to those head-mounted display experiences that AR researchers consider ideal for the format.

Back to Reality
Uzzan admits that previous gaming implementations of ar have been disappointing and may have dissuaded the industry from pursuing the augmentation line of development. "But everything is ready for AR now," he says. Multiplayer gaming is in vogue, from casual online gaming communities to massive live World of Warcraft and Halo bashfests. The technology is moving beyond those ugly visual markers that looked like Klingon iconography. Image detection and tracking can now superimpose virtual objects onto faces, and natural features in a video scene. "The other thing helping us a lot is the Wii console," says Uzzan. "The Wii-mote is part of it. For the first time you can be part of the game. It is unlocking some doors that have been locked so far."

And like the Wii remote, AR could help build a long-awaited bridge between virtual and real worlds. Internet data, 3-D graphics, and computational power have spent decades on desktops where the possibilities for their use were locked (just as we were) to the desk. Just as mobile phones are bringing the Internet and all of its rich data out into the physical world where it can interact in new ways with locations and objects, AR knits together digital and physical realms in ways we couldn't have imagined earlier. After decades of staying cooped up inside, tethered to a world of pixels, video gaming may finally be ready to come outside to play.