Mobile AR Experiences

Apple adding augmented reality to iOS (IPhone & IPad) maps

A recent patent submission by Apple describes a technique to improve iOS (IPhone & iPad) maps using augmented reality. It proposes to integrate additional information by overlaying a camera video stream with supplemental information about the surrounding environment.

This approach seems to be very close to techniques used already by Layar or Wikitude. I’m really curious how Apple will implement this approach in a final product. Yet it remains unclear where in which way this technique is innovative!

“According to Apple’s patent application, street names, addresses and landmarks can all be added to a scene simply by pointing the camera at an area in the vicinity.”

Patents Show Major Changes Coming to Apple iOS Maps | The iPhone FAQ

My take on WebGL

Hi there,

in recent news about WebGL, where Microsoft addressed concerns about browser security in its blog post WebGL considered harmful. Basically, they say that due to several reasons, like the inevitable access of WebGL code to the hardware (the graphics card), buggy drivers and other issues, WebGL represents a significant risk to browser security. Therefore,  the MS Internet Explorer will not support WebGL in the foreseeable future.

Who doesn’t know WebGL yet, “WebGL is a cross-platform, royalty-free web standard for a low-level 3D graphics API based on OpenGL ES 2.0, exposed through the HTML5 Canvas element as Document Object Model interfaces. Developers familiar with OpenGL ES 2.0 will recognize WebGL as a Shader-based API using GLSL, with constructs that are semantically similar to those of the underlying OpenGL ES 2.0 API. “, which is the definition of WebGL by the Khronos Group.

Of course, that created a some turmoil in the WebGL community and Gregg Tavares from the Google Chrome Dev team came up with a more than clear blog response. Basically, he is saying that Microsoft is all wrong. Every mentioned security issue can be adressed so that browsers can implement WebGL in a secure way.

In any way, Google will continue to push forward Chromes WebGL development (Chrome experiments), as Firefox will do (Mozilla Demos). I predict that MS will finally align, when it will become inevitable for them. This point could be reached, when popular games will be published using WebGL technology. You could play as you go in your browser, no install, no plugin (Flash). But thats still a long way to go.

Since I work for more then 10 years with OpenGL, of course I would like to see WebGL succeed. There are significant challenges on the way, but I clearly follow the position of Gregg Tavares that these problems can be tackled, solutions already exist or will be developed in the near future.

If you are new to WebGL, here is a small introduction and if you want to have more details on the topic do not hesitate to take a look at the Google IO WebGL Techniques And Performance presentation.

Of course, I’m especially interested in WebGL since this technology will probably allow mobile AR developers, in the not so far future, to create applications that won’t be dependent on many different plugins or special applications for each mobile plattform.

You can get a good impression what I’m talking when you read previous blogpost on Remixing Reality by Mozilla presenting an AR application written in webGl and flartoolkit (flash) that even runs on Android Firefox beta.

I just played around with some WebGL and came up with the example below my other blog. Regards to learningwebgl.com. If you do not see a turning triangle than try another browser like Chrome or Firefox.

The CARE project

“The CARE project proposes to apply Augmented Reality techniques to the cultural domain. 

CARE is a French research project funded by the French research agency (ANR). The project brings together 7 partners, whose goal is to integrate, adapt and develop tools to enhance interactive systems devoted to cultural experiences. These enhancements are based on Augmented Reality and Emotions. 

Besides the enhancement of the two specific scenarios considered in the project (Ballet of Biarritz and Museum of Toulouse), CARE results will include improved conception methods as well as new generic interaction techniques based on augmented reality technologies.”

Reality Fighters for PlayStation Vita at GamesCom

“Fight as yourself anywhere and any time” is the motto of Reality Fighters for PS Vita give its debut at GamesCom in Cologne Germany these days.
You can texture your fighter with your photo and play on real (back)grounds of your choice.

Tracking seems to be relatively robust, for a planar surface of course, with sufficient lighting conditions. The projected shadows on the detected surface add very much value to the entire AR experience

Google acquires Motorola Mobile (and Spock is not impressed)

Google acquires Motorola Mobility (and Spock is not impressed) from NC

Only indirectly related to mobile augmented reality but still massive news. Hope this helps to drive innovation in the mobile sector.

More details on Googles Blog.

KinectFusion: Real-Time Dynamic 3D Surface Reconstruction and Interaction - Microsoft Research

KinectFusion: Real-Time Dynamic 3D Surface Reconstruction and Interaction - Microsoft Research presented at SIGGRAPH 2011 in Vancouver Canada.

 

KinectFusion, a system that takes live depth data from a moving Kinect camera and in real-time creates high-quality, geometrically accurate, 3D models. Our system allows a user holding a Kinect camera to move quickly within any indoor space, and rapidly scan and create a fused 3D model of the whole room and its contents within seconds. Even small motions, caused for example by camera shake, lead to new viewpoints of the scene and thus refinements of the 3D model, similar to the effect of image superresolution. As the camera is moved closer to objects in the scene more detail can be added to the acquired 3D model.

Remixing Reality - Mozillas WebGL Augmented Reality Demo Rocks

Remixing Reality is a WebGL Demo by by ILMARIHEI published at Mozilla Labs that uses, as far as I can tell, Flashs Flartoolkit to create an amazing AR experience inside a website.

Flash with Flartoolkit allows you to integrate a prerecorded video presenting special markers. Flartoolkit computes then 3D positions of these 2D markers inside the video. These information are used by WebGL to render a 3D object or animation at the markers 3D position. Or in between two markers, in case of the jumping basketball animation.

In the next step they should get rid of Flash and Flartoolkit, so that no plugins are needed anymore to render the website. But this will take again a lot of time until some kind of augmented reality web standard will be established and most of all implemented.

Actually, I’m not 100% sure if they use Flartoolkit, but if you disable Flash the demo doesn’t work anymore. So they use Flash at least to render the video. I suppose the tracking is done with Flartoolkit.

I just borrowed their demo ;). But only because I think its so totally cool and well, because it’s so easy. If you like the demo, you should go to the Mozilla Developer site and share it on FB and Twitter with your friends. Switch to Chrome or Firefox if you cannot see a jumping basketball between two moving sheets of papers.

Unity Track Day at GDC Europe in Cologne

As part of European Game Developer Conference (GDC) in Cologne Germany, Unity3D will present a full day tutorial known as Unity Track Day. The different talks will give details about cross platform games, performance optimization and better social integration. Registration is already closed, but tickets are still available on site. More information here.

Here is the detailed program.

Hand’s demonstration on making cross platform games with Unity, Part 1Time: 9:30am – 10:20amSpeaker: Carl CallewaertA hands-on demonstration of the development of a multiplatform game starting with prototyping the game to the final version running on Android, iOS, the web, Facebook and as a standalone app. The steps include but are not limited to: programming, art assets import and updating, particles, GUI, AI and building for multiplatform deployment.

Hand’s demonstration on making cross platform games with Unity: Part 2Time: 10:50am – 11:40amSpeaker: Carl Callewaert

Performance optimization tips and tricks for UnityTime: 12:40pm – 1:30pmSpeaker: Jonas EchterhoffThis session will demonstrate optimization techniques on a sample Unity project, show how to use the Unity profiler to find performance bottlenecks, and will show some common pitfalls to avoid for optimal performance in Unity projects.

Shader Authoring in UnityTime: 2:00pm –   2:50pmSpeaker: Shawn WhiteIt’s difficult to make a game today without writing at least one shader. Whether you’re indie, mainstream, or somewhere in the middle, having the ability to write your own shaders will give you what you need to make your game POP! Unity provides a suite of built-in shaders to get you started, but in this talk we’ll get you up to speed on what you need to know about writing your very own shaders in Unity.

Being Social: How Games Can Leverage Facebook on the Road to SuccessTime: 3:20pm –   4:10pmSpeaker: Tom HigginsThe explosive growth of social networking is changing the ways in which people of every type interact with each other in some incredible ways. That’s easy to say but it takes a bit more effort to properly understand. What are those changes? Are they limited to those creating browser-based content or do they extend to the desktop and mobile apps as well? And most importantly, how can you and your team take advantage of them in your games? This presentation will discuss some of the most prominent questions facing developers targeting social networking sites and describe how developers can put that information to use right away. The discussion will focus on Facebook and the Unity game engine in particular as they offer developers unique opportunities for success. But in the end the strategies that will be shared can easily be applied anywhere, regardless of what social networking site or game engine is being used. Developers of all types can attend this session and learn how to incorporate social networking features into their own efforts, from the early design phases all the way through to release, giving themselves a much greater chance of success.

Using and publishing content and extensions on the Unity Asset StoreTime: 4:40pm –   5:30pmSpeaker: Tom HigginsThis session will start with an introduction to the Asset Store, showing what is available and demonstrate how to search for and use content in your own projects. After that he will go through how to submit your own content and extensions to the store, how it should be formatted and go through common mistakes to avoid and other hints for speeding up the submission process.

Distribution with UnionTime: 5:40pm –   6:30pmSpeaker: Brian BruningThere’s never been a better time to be a game developer.  Huge audiences are now reachable without the aid of publishers or retail outlets.  But what are the odds of getting noticed out there today in the new digital landscape?  What tactics are most effective?  What should you be doing to get noticed and find customers?

Talk on mobile Augmented Reality at Paris Android User Group (Paug)

Hi there,

here is the video of our mobile augmented reality talk given at Paris Android User Group in june.

In the first part Atef Haouari presented a technique to create a mobile AR application for Android using position (GPS) and orientation sensors.

There is a recap of the entire talk here. In the second part (from 1:07h), I present some of our work in the area of vision based mobile augmented reality for Android using the Qualcomm AR SDK (QCAR) and Unity3D.

The presentation was held in french. Sorry for all the non-french speakers.

Watch live streaming video from parisandroidug at livestream.com

The Augmented City

Metaio just released a new video of their recent The Augmented City demo. Looks like they finally implemented a quite stable markerless visual tracking (SLAM??). The 3D rendering and animation is very fluid and appealing as well. Will be interesting what else they will announce on their upcoming insideAR conference in september in Munich.