Natural communication between humans and machines au VISION STUDIO's virtual human "coh"

In March 2021, KDDI's new creative team "au VISION STUDIO" announced a virtual human "coh" with visuals that are as close to humans as possible.

"Coh" was born as the "ultimate interface".

The operation methods of devices in the world are evolving steadily. Voice input has also become commonplace so that anyone can easily handle it. Still, there are many people who feel uncomfortable about "talking to machines."

The virtual human "coh" was developed to overcome the discomfort. The appearance, conversation, facial expressions, gestures, etc. are closer to humans, and you can talk to "coh" as if you were talking to a person, operate the device, and receive various services. I envision such a future.

This virtual human "coh" appeared in the demonstration experiment "HYPER LANDSCAPE" in collaboration with "National Museum of Emerging Science and Innovation" from March 11th to March 14th, 2021.

Virtual human "coh" appears in the demonstration experiment of "au VISION STUDIO" and "National Museum of Emerging Science and Innovation"

Participants wear smart glasses. "Coh" appeared in AR in this glass and played the role of "attendant".

Let's take a look at the future of Virtual Human and "au VISION STUDIO" while introducing the pattern of this event.

Virtual human "coh" becomes a guide at "National Museum of Emerging Science and Innovation"

The "National Museum of Emerging Science and Innovation" is a facility for introducing the latest science and technology to the general public. There is also a laboratory for researchers, and the results can be exhibited immediately.

The stage of this demonstration experiment was the symbol of "National Museum of Emerging Science and Innovation", the giant spherical display "Geo Cosmos" that imitated the earth, and the spiral passage "Oval Bridge" that can be seen up close. ".

"Geo Cosmos" of "National Museum of Emerging Science and Innovation" which became the event venue

Experienced people stand here and wear smart glasses "Nreal Light". Unlike VR goggles, smart glasses allow you to see the surrounding scenery through the lens. Therefore, you can walk around while wearing it, and you can see the information and images projected on the lens part by superimposing it on the actual scenery.

Wear smart glasses "Nreal Light" and walk "Oval Bridge"

And the scenery that spreads in front of you changes completely. "Geo Cosmos" and its surroundings are filled with blue light sources like tiles, and a virtual human "coh" appears when approaching the white circle displayed on the floor.

After introducing himself, "coh" says, "Please enjoy another National Museum of Emerging Science and Innovation." The gesture usually does not have the discomfort that we have in CG. At first, I tend to look down a little, and the movement of slowly looking at me while talking is very human.

As you walk down the oval bridge following the arrows that appear on the floor, "coh" will reappear and you will be prompted to see "Geo Cosmos". With the naked eye, it looks like a blue earth, but through smart glasses, you can see it covered with innumerable artificial satellites as shown on the right.

In addition, "coh" explained "changes in immigration regulations around the world due to the effects of the new pneumonia".

Click here for a movie that summarizes the pattern of this event.

In this way, when the experiencer wears smart glasses and walks along the route, "coh" appears and guides them. The voice is a synthetic voice, but it is not inferior to the intonation spoken by humans. He gradually adjusted his gaze to this one when he appeared, and at the end of the guide, he showed a shy expression to spare no part.

Background of the birth of virtual human "coh"

We talked to Osamu Mizuta, who promoted 5G / XR projects at KDDI, founded "au VISION STUDIO", and created the virtual human "coh", about the background of the birth of the virtual human "coh" and the future he is aiming for.

KDDI Personal Business Headquarters Service Management Headquarters 5G / xR Service Planning and Development Department Service Product Planning G Group Leader Osamu Mizuta

"As a telecommunications company, KDDI has helped connect people. The equipment used at that time has evolved from feature phones to smartphones, for example. However, even if the communication method has evolved further. I thought it was good.

At that time, "people" would be the easiest for everyone to accept. The idea was to eliminate the feeling of "manipulating" as much as possible. First of all, if you enter the "Menu screen" and tap "Settings" and are instructed to do something like ..., I think it will be "difficult!" For children and the elderly. However, talking to people is a communication that anyone can do, no learning is required, and I don't think it feels like operating a machine.

To be honest, if you only pursue convenience, I think smart speakers are fine. But which one is more fun? Do you have an attachment? And I think it's the virtual human that fits into our lives more naturally than smart speakers. "

Furthermore, in this exhibition, where the virtual human "coh" appeared through smart glasses, we challenged to expand the physical limits of the exhibition space.

"Of course, there is a limit to the size of exhibition spaces such as museums. Not everything you want to see can be placed. If you use smart glasses to display in AR, you can actually use the space of the exhibition space. You can see things that can't be placed in the glass. Moreover, no difficult operation is required, and you can listen to the commentary just by talking to the virtual human "coh" that appears. I think it has a high affinity for

Technology that supports virtual humans

The virtual human "coh" speaks with a human-like expression and tone so that people can communicate naturally. What surprised me was that he always turned to the experiencer and talked to me. Why doesn't "coh" look elsewhere or float in the air?

We spoke with Satoshi Komorita of KDDI R & D Laboratories about the technology to achieve this.

Satoshi Komorita, Group Leader, Media Recognition Group, Media ICT Division, KDDI R & D Laboratories

"In order for'coh'to appear in a certain position and talk to the experiencer properly, a technology called VPS (Visual Positioning Service) and high-speed communication of au 5G are used.

Take a picture of the event venue with a 360 degree camera in advance and create a 3D map for VPS. And at the time of experience, by recognizing the site as an image with the camera mounted on the smart glass and finding the matching point with the 3D map, it is estimated where the experiencer is and which direction, and a virtual human appears accordingly. It is a mechanism to make it

Create a 3D map by shooting the venue with a 3D camera.Based on this, refer to the local image seen with the smart glass camera and specify the position and orientation.

"The extremely realistic and human-like tone and movement of'coh'is displayed on smart glasses with VPS while finely changing the CG image according to the standing position and orientation of the experiencer. For that purpose, a huge amount of data is exchanged. ..

If the CG deteriorates or the movement is delayed, it is no longer natural communication. A high-speed, large-capacity, low-latency communication technology of au 5G is required for these series of operations. "

There is a "coh" in front of me, and he talks to me properly. Only with au 5G can a virtual human who can communicate naturally be established.

The future of virtual humans "coh" and "au VISION STUDIO"

"Au VISION STUDIO" is KDDI's aim to solve diversifying social issues and provide new experiences that have never been seen before by actually incorporating 5G and XR into society with an eye on the future. It's a creative team.

We are presenting what we want to achieve in future society as the following "five initiatives."

"UX that expands the five senses"

For example, the AR experience of superimposing images and videos on the scenery of the city using smart glasses is "expansion of vision". An initiative to use digital technology to expand all senses, not just vision, to provide new experiences. By transmitting AR information without actually creating advertisements or signage, it is possible to preserve the landscape.

"Zero distance world"

Efforts to eliminate the wall of distance. By utilizing VR, not only can you travel without moving and watch live concerts in remote areas, but you can also communicate as if you were there, such as by linking humans and robots to convey tactile sensations. We will promote the experience of "participating" rather than just watching and listening.

"Shopping that is kind to people and the earth"

You can easily get clothes of the size and design that suits you while you are at home by trying on clothes using AR in virtual space and diagnosing with AI. Not mass-produced, but made to order. At the manufacturing stage, it is designed digitally and the pattern is virtual. A new way of shopping that does not consume unnecessary materials and does not require inventory.

"Unlimited viewing experience"

We provide unique technology for video content that can be experienced online, such as live music and theater. With powerful AR and digital stage expansion, multi-channel distribution that allows you to go back and forth between multiple stages like an actual festival, create a viewing experience that is "realized only by distribution" rather than "real alternative" with eventers and artists. go.

"Daily life of virtual humans"

The easiest and least costly communication method is "communication with people". We will continue to create a system in which virtual humans are used on a daily basis as the ultimate and natural interface that reacts like humans in the form of humans.

In addition, "coh" has already been appointed as a make-up model for "KANEBO". Makeup experience content "coh x Shinobu Igari supported by KANEBO" by hair and makeup artist Shinobu Igari at the virtual cosmetics shop "@cosme TOKYO -virtual store-" where you can shop at "au XR Door", a service for smartphones that utilizes XR. Appeared in, receiving make-up by professional hands and showing a completely different look.

He is also active as an attendant explaining his works by watching art using smart glasses at "KDDI ART GALLERY".

With the spread of 5G, communication technology will continue to evolve. KDDI will utilize 5G and XR to create new experiences that make the future a little ahead of "au VISION STUDIO" interesting and convenient, and provide excitement to society.