The key to full-scale popularization of mobile AR is "AR cloud" | Mogura VR

With the release of ARCore by Google following Apple's ARKit, AR (Augmented Reality) apps have become more accessible all at once. Both are free, cool, and work anyway.

However, at present, even if it is called "popularity", it seems that it is limited to generating a large amount of YouTube video playback among AR developers. From now on, developers must prove that AR apps are not just a novelty trendy thing, but something that can spread widely to the general public.

Some mega-hits (such as Pokemon Go) will benefit from having a massive channel to distribute AR apps to iPhones and high-end Android phones. But I doubt that hundreds of millions of people will use AR apps on a daily basis.

ARKit and ARCore-based AR apps are like surfing the net without friends. It's too 1996.

Don't get me wrong, the arrival of ARKit is the best news ever in the industry. However, in order for AR applications to begin to spread in earnest, another key must be needed. It is the "AR cloud", an environment that enables a consistent AR experience across time, space and devices.

I expected the iPhone X to fill that role. Tens of millions of hands with depth-sensing rear cameras scan the world around them and create rich, detailed 3D maps that they can share with others in the AR cloud. I wonder if it will be like this. That expectation didn't come true this time, but I'm sure it will come true in the next version. Even if the arrival of the AR cloud is delayed, the demand will not disappear. The iPhone X's depth camera is only on the front. Where you want to mount on the back

2017 is a major turning point

Over the past two years, as the awareness of AR has increased and the specifications of mobile devices have caught up with the standards required for the realization of AR, many startups have I was developing an AR tracking feature similar to ARKit and ARCore. However, in response to the one-two punch announced by Apple and Google, we are in a situation where we have to change our positioning as soon as possible in order to continue riding the waves of the times. Therefore, it may be a natural conclusion to steer towards cross-platform functionality. But how much future is there in this thin layer of additional functionality? And Unity already provides the ability to deploy your creations to multiple platforms. In order for startups that enter the same field later to establish their presence, it is necessary to provide tools, services, and content that are not second nature.

When I make a dramatic announcement, I think about the positive impact it has. The first thing that came to my mind about ARKit and ARCore was that it freed AR tech startups from building a difficult foundation, and the smarter ones would be aiming higher up the value chain, where it tastes better. was. That is the AR cloud.

AR researchers and industry insiders have long predicted this. Someday in the future, a real-time 3D (environmental) map of the entire world, or "AR cloud", will be the only and most valuable software infrastructure in computing, far more valuable than Facebook's social graph or Google's page rank index. I thought it would be

I have been working in the AR industry for almost 10 years now. The company he co-founded in 2009 (Ogmento, now Flyby) was acquired by Apple and became the foundation for ARKit. Incidentally, the same technology was also licensed to Google Tango before the acquisition. I am also the co-founder of AWE (an AR conference) and Super Ventures (an AR-focused fund), so I have a wide range of knowledge about the AR industry, and I am looking for areas where AR can create great value. I am always searching.

I wrote this article to draw attention to the value AR Cloud can create on top of ARKit and ARCore.

For an in-depth look at ARKit, ARCore, and what they imply, my partner at Super Ventures, Matt Miesnieks, has written a great series of articles (very hot topic). It's been a long time since I've been in the middle of a long time since I've been in the middle of a long time).

For more articles on collaborating in AR, see Mark Billinghurst, another of my partners at Super Ventures and one of the world's leading experts in the field.

Why do we need an AR cloud?

In a nutshell, the AR cloud turns the whole world into a stereoscopic shared screen, allowing multiple users to interact and collaborate. It's from

Most apps built on ARKit or ARCore are single-user AR experiences, which takes away from that appeal. It's true that YouTube is now flooded with videos of special effects and spectacles using AR, but sharing videos of AR experiences is different from sharing *the AR experience itself*. increase. In the words of Magritte, "This is not a pipe."

1928 work by Rene Magritte. By writing "This is not a pipe" under the drawing of the pipe, the viewer is made to think that this is a picture of a pipe and not the real thing.

The AR cloud is a shared memory of the physical world, enabling sharing of experiences rather than just videos and messages. This means you can collaborate on games, designs, research, and problem-solving on just about anything in the real world.

Achievement of involvement by multiple users is one of the great attractions of the AR cloud, but the hidden potential beyond that is the accumulation of information in the real world.

Concept of collaborative AR space in shared space (2011) *Translation of balloons in the image Top left: Common AR play space (real and virtual objects are mixed) Top right: Restricted to the same experience even remotely Lower left: Display augmented reality on the screen

We are about to enter an era in which the way information is structured will fundamentally change. Today, most of the world's information is stored on servers in the form of digital documents, videos, and snippets of information that can be retrieved from anywhere over the internet. But it requires some form of search and discovery.

According to a recent study by Google, over 50% of searches are done while on the move. There is an increasing need to retrieve the necessary information at the necessary place and at the necessary time.

The AR cloud acts as a soft 3D copy of the world, capturing information at its source (scientist-speak in Latin, in situ, meaning "in its place") -- the physical Allows you to reorganize in the world.

With the AR cloud, you can look at things to see how they are used, see places to see their history, and see people to see their history.

And whoever controls the AR cloud will be in control of how the world's information is sorted and accessed.

When will the AR cloud come to fruition?

I fear that when startups start working on the AR cloud, it will become a commodity before it has a chance, just like ARKit. Some people will.

AR Cloud is not for the faint of heart. It will take some time to realize. That's how industry leaders describe it to developers, professionals, and consumers. But for investors in AR and advanced tech, now is the time to see who is likely to win this long-distance race. One of the first early collaborative AR demos in architecture, 2004

For reference, let's take a look at the history of AR tracking technology, which has only recently been commoditized with the advent of ARKit and ARCore.

- Point clouds first invented: 19th century – First real-time SLAM using computer vision by Andrew Davidson: 1998 – Monocular SLAM demo on iPhone by George Klein (*): 2008 – Project Tango First AR-focused mobile device announced by: 2012– SLAM on iPad with Occipital Structure Sensor: 2013– Flyby demoing features close to current ARKit: 2013– Announcement of ARKit and ARCore: 2017

Editor's note: SLAM stands for Simultaneous Localization and Mapping. Simultaneous self-localization and environment mapping. In AR using SLAM, 3D models can be literally "placed" in the real world, and characters can "walk on the ground" and recognize real terrain.

At this pace of technology advancement, it will be more than three years before a mature AR cloud service is generally available.

In the meantime, startups can work on parts of the overall vision, go for first-mover advantage, or build services tailored to specialized needs (eg, enterprise).

Has anyone ever tried to create an AR cloud?

There have been a number of companies that have provided AR services from the cloud in the last decade. Starting with Wikitude in 2008, Layar, Metaio (Junaio), and later Vuforia, Blippar, Catchoom. More companies are now entering (see AR industry map). But most fall into one of the following categories:

– Accumulation of information tied to GPS or location information (such as displaying a message in a balloon inside a restaurant)

– Cloud Image Recognition Service to Launch AR Experiences

These cloud services don't actually know the situation on the ground or the geometry of the physical environment. Without such information, virtual content cannot convincingly blend into physical reality. Moreover, sharing the experience itself (not just a good-looking video) with others is simply impossible.

Pokémon GO is gaining popularity among a wide range of people?

Pokémon GO's astounding popularity is one of those rare exceptions, and a stroke of luck. It would be extremely difficult, if not impossible, to achieve similar success by copying game mechanics. The game's servers accumulate geolocation information, highly detailed local imagery, and player activity histories, but do not share a perception of the physical space in which the 65 million monthly active users have played. Therefore, no sharing of experience in the physical world occurs either.

For that you need AR Cloud.

What exactly is an AR cloud?

Since the 1990s, many scientists have been thinking about what elements an AR cloud should have. It has not become So, here's my version that's loosely summarized from the context of the AR industry.

What is required for an AR cloud system: 1) Persistent point cloud linked to real-world coordinates (shared soft copy of the world) 2) Instant localization for multiple devices anywhere (real-world environment and world 3) the ability to place virtual things in the world's softcopy and interact with it in real-time, on-device, and remotely

Let's take it one step at a time.

1) Persistent point cloud bound to real-world coordinates

A point cloud is defined by Wikipedia as "a collection of data points in some coordinate system, each point is usually described as (x, y, z) in a three-dimensional coordinate system, and is a technique that is now fairly commonly used in industry and military, including 3D mapping and reconstruction, surveying, and inspection. (Translation: The Wikipedia definition quoted by the author differs in some details from the description as of November 1, 2017. The translation was made by supplementing the original article from the English version of Wikipedia, so the Japanese version (Not exactly as written) Collecting point cloud data from the physical world is itself a "solved problem," as engineers put it. Many hardware and software solutions have already been developed for the creation and processing of point cloud data. Some use active laser scanners like LiDAR, some use depth cameras and stereo cameras like Kinect, some use monocular camera photos, drone images and satellite photos with photogrammetry, Vayyar and others. Some use synthetic aperture radar systems (radio waves) and satellite radars.

By the way, photogrammetry was invented at about the same time as photography, so the concept of point clouds can be considered to have originated in the 19th century.

A game that uses point clouds generated by HoloLens and is displayed according to the shape of the living room

To ensure that persistent point cloud data always reflects the most recent state of the real world to its fullest extent, an extra layer of complexity is required. In other words, point clouds acquired from various sources, including mobile devices, must be stored in a database in a unified format, and the data must be available to many users in real time.

Although scanning and tracking are limited to specific devices with advanced support (ARKit, ARCore, Tango, ZED, Occipital, etc.), the point cloud data is stored in a cross-platform database. is required to be

Individuals' motivations for sharing acquired point cloud data will be similar to current Waze users. Waze is a car navigation app that shares traffic information with the community. While receiving valuable services (optimal route guidance), users can share information collected from their devices (moving speed on the current road section) in the background to provide services to other users (information updates). frequency).

But in the AR cloud, this can pose serious security and privacy concerns. This is because we are dealing with the recording of an environment that is much more private than a public road or highway: the space around an individual. For startups, there seems to be a big business opportunity in point cloud data encryption.

Depth cameras are a great help in creating 3D maps. To put it bluntly, it's such an important part that a normal smartphone without a depth camera is not very useful.

A depth camera allows you to see more of the geometry of the surrounding space. The algorithm used is the same as a smartphone equipped with a normal camera, but since point cloud data with higher density and accuracy can be obtained, it is possible to create a more precise AR cloud.

There have been some devices equipped with depth cameras so far, but they have not been widely used. Thousands of Tango's experimental developments were sold. Occipital's depth sensor also sold only a few thousand units. The Tango-based Lenovo Phab 2 Pro, launched in 2017, was the first smartphone on the market to include a depth camera, but it hasn't caught on in a big way, and the new ASUS ZenFone AR is still not mainstream. yeah.

The iPhone X's depth camera was only installed on the front, but if a full-fledged one is installed in the future, it will bring dramatic value. For countless users.

As a requirement for the AR cloud, I mentioned multi-user real-time access, which in some ways is similar to the role of servers at the core of MMO (massively multiplayer online) games. That means enabling millions of users to access information from anywhere and collaborate in real time. However, in MMO, content such as characters and stories are placed in a permanent artificial world stored in the server, but in AR Cloud, the real world itself becomes the stage, and point cloud data with augmented information is added. perfectly aligned.

3D maps are already widely used in self-driving cars, robots, drones, and more. Even robot vacuums like Roomba use 3D maps to navigate around rooms, and are even considering sharing maps they create in the future.

However, the point cloud for the AR cloud must be different from the general one and suitable for AR. If the 3D map is too detailed, it will be too heavy for mobile devices to handle, and will not give users the instant response they want. If there is a gap in the map, it will not be possible to do localization depending on the angle, which will hurt the user experience. In other words, an AR cloud is a special point cloud with its own set of parameters.

Google Tango calls this "spatial memory". It is the ability of a device to recognize visual features of physical space (contours, corners, and other landmarks), remember them, and recognize them later when they return to the same location.

What you need for your AR cloud will be different for indoors and outdoors, and for enterprise versus consumer use.

But most important is fast localization.

2) Ability to instantly localize multiple devices anywhere

Localization is a function that is the basis of all AR experiences by estimating the position and orientation of the camera.

Flyby's localizer superimposing the point cloud onto a real kitchen

In order for an AR app to place virtual objects in real space in the way the creator intended, it needs to know where and in what direction the device's camera is looking at the geometry of the real environment. I have.

It does this by matching visual feature points captured by the camera with feature points recorded in the AR cloud to find matching positions and angles. For instant localization, the AR Cloud localizer must refine search results based on device orientation and GPS location (as well as triangulation using other signals such as Wi-Fi and cell towers). must be With more contextual data and AI, you get even more precise results.

Once localization is complete, ARKit or ARCore will take over (using IMU and computer vision) to provide stable positional tracking. As I said earlier, this issue has already been resolved and commoditized.

There are already many solutions such as Google Tango and Occipital Sensor, and ARKit and ARCore are one of them. However, these "out of the box" means can only localize to one local point cloud at a time. Microsoft HoloLens can localize HoloLens-generated point cloud sets, but not "out of the box" or point clouds generated by other devices.

The search continues for the "ultimate localizer" that can localize a huge local point cloud set from all angles and share the point cloud cross-platform to multiple devices. However, any localizer is ultimately dependent on the quality of the persistent point cloud, so it makes a lot of sense to try to build both.

3) The ability to visualize virtual objects in 3D and interact with them in real-time, on-device, and remotely

Create persistent point clouds and the ultimate localizer Even so, in order to complete the AR cloud system, it is necessary to be able to arrange and visualize virtual contents registered in 3D. "Registered in 3D" simply means "aligned in the real world as if it were really there". This virtual content should be interactive. Multiple users should be able to observe the same slice of the real world from different angles and interact with the same content in real time, each from different devices.

God mode is essential to maintaining the AR cloud and its applications. In other words, the ability to remotely place 3D content (+ interactions) anywhere in a vast point cloud. Like the "Hand of God" controlled by the player in the video game "Black & White", if you select any place on the map, you can see the space from various angles and see what is happening there in real time. We need a way to observe and interact with the content placed there. This could be done on a variety of devices such as computers and tablets. And it will create a new genre of "real world CMS (content management system)".

In the video game Black & White, players can see and control everything with the Hand of God.

Now, these are the three requirements for the AR cloud. Realization seems to be quite difficult, but is there currently a place where this can be built?

Who can build an AR cloud?

When it comes to a project as big as the AR cloud, there are probably only three companies in the world with deep pockets and big ambitions to tackle it.

Apple, Google and Microsoft may one day have a complete AR cloud on their own. If you can't, you'll be caught off guard by the companies that made it happen first.

But given history, it will probably be some (crazy) startup that lays the groundwork for the AR cloud and proves the concept. Because only startups have the audacity to bet on that possibility.

You're going to need a crazy good startup.

It's too big for one company to work on, so multiple startups may be working on different parts. at least at first.

– 13th Lab (acquired by Facebook)

– Obvious Engineering/Seene (acquired by Snap)

– Cimagine (acquired by Snap)

– Ogmento → now Flyby (acquired by Apple) Flyby also licensed technology to Tango.

– Georg Klein* (Microsoft

*Georg is an individual, not a startup, but he has single-handedly surpassed many companies. In 2009, the iPhone 3G ran an ARKit-like demo for the first time.

Startups currently working on AR cloud or part of it

YOUar (Portland)

Scape (London)

Escher Reality (Boston)

Aireal (Dallas)

Sturfee (Santa Clara)

Paracosm (Florida)

Fantasmo (Los Angeles)

Insider Navigation (Vienna (Austria))

InfinityAR (Israel)

Augmented Pixels (Ukraine/Silicon Valley)

Kudan (Bristol (UK)) (*)

DottyAR (Sydney, Australia)

Meta (Silicon Valley)

Daqri (Los Angeles)

Wikitude (Vienna, Austria)

6D.ai (San Francisco)

Poster.io (San Francisco)

Chubu Note: Kudan has two offices in London and Tokyo.

Many of the companies listed here are planning to attend AWE, so you'll see them in action there.

What are major companies doing now?

Apple CEO Tim Cook said that AR is a "big idea" and Apple is actively working on AR. It can be seen that it is a positive attitude.

ARKit was an easy achievement for the company. Because we were able to run it ourselves (by acquiring Flyby), and we also had specialists who could commercialize it (by acquiring Metaio). It is said that Apple decided to release ARKit only six months before this year's WWDC.

If the future iPhone [rear camera] is equipped with a depth camera, the creation of detailed and accurate point clouds will be accelerated, and the depth camera will become a standard feature for future smartphones. Even if it does happen, it's still a long way off.

Although there are reports of work on 3D scanning of roads for self-driving cars, 2D mapping and positioning for map functions, and "more than 1,000 engineers working on AR", Apple is now However, it seems that they are not ready to start building a full-fledged AR cloud yet. Maybe it's because of privacy and security concerns about data. Or, since we're talking about secretive Apple, it could be cleverly hiding something it's already working on.

With Google's mission to "organize the world's information," it's only natural that it will build an AR cloud next. Google may have the most pieces to complete this epic project, but it doesn't look like they're making any big moves right now.

Tango started in 2013 as a stand-alone R&D project. That vision contains three elements.

This sounds pretty close to the AR Cloud concept, but currently Tango is a mechanism to store environmental data on the device, and there is no mechanism to share it with other Tango devices or users. The Tango API also does not natively support data sharing in the cloud.

At I/O 2017, Google announced a new service, the Visual Positioning Service (VPS). Once completed, the realization of the AR cloud will be one step closer, but as far as I can tell, it seems that the current situation is still in the "very early stage".

However, Google has already built a fairly detailed 3D map (for the outdoors) with the Street View car. In a few years, we may have all the pieces together and complete the AR cloud. However, Google's API tends to be limited in functionality and may not be sufficient to provide an optimal experience.

So while Google is very likely to be the next AR cloud owner, it probably won't be the best, and it probably won't be the first.

The Microsoft CEO's commitment to this area is evident in cloud computing and holographic computing. HoloLens is currently the most complete AR glasses system, albeit bulky, and it does a fantastic job of creating point clouds that persist locally on the device. However, it cannot be shared natively to multiple devices. Lacking a strong trump card to enter the current mobile AR race, Microsoft has built its lead by getting an early start on its next-generation focus, smart glasses. However, the spread of smart glasses is still slow, so Microsoft's need for AR cloud is still in the future.

It may be of active interest, but it is years behind the cutting edge (which means acquisitions are very promising for the aforementioned startups).

I'm a candidate in that I've already started building 3D maps of the real world outdoors and indoors, but I don't necessarily have a particular interest in AR.

It is highly possible that Google-class startups since Magic Leap will emerge from among the companies working on the AR cloud. Speaking of which, Magic Leap should have once aimed to build an AR cloud. I don't know if there is still a desire to do so, or if we should consider the possibility.

However, none of the above companies have the incentive to create a cross-platform solution (with the possible exception of Facebook). In that respect as well, it is meaningful for startups to take the lead in building AR clouds.

The purpose of this post is not to define requirements for the AR Cloud at the requirements document level, but to convey what I envision the AR Cloud to look like. I look forward to seeing approaches that overturn that expectation in the future.

If you are interested in building, using, or investing in an AR cloud, please contact us. Become a partner.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Articles specializing in each component and service of AR cloud about usage after AR cloud is realized

What will the app store look like in the AR cloud?

Possibility of indie AR cloud providers Is there a need for indie AR cloud providers?

If AR becomes another reality shared by all, who should maintain the content within it? Do we need a universal mechanism like DNS?

Reference materials (all links are in English)

Mixed reality service

What Raph Koster learned from 20 years of MMO history (see his website)

Information about point cloud hardware and software solutions

An excellent collection of public resources on point cloud algorithms

Background Information on Spatial Awareness from the Naval Postgraduate School

Mixed Reality in Shared Spaces: On Augmented Reality Interfaces for Collaborative Computing, by Mark Billinghurst (white paper)

On the ROI of building a persistent point cloud

The first real-time SLAM using computer vision: Andrew Davidson's 1998 paper (supervised by David Murray)

Why 3D maps in AR are more valuable than Google Maps

Contributed by: Matt Miesnieks

*This article is based on the article "ARKit and ARCore will not usher massive adoption of mobile AR" posted on the company's blog by Super Ventures co-founder Ori Inbar on September 13, 2017 with permission. translated and reposted in Mogura VR.