Synthetic Reality

1. Rapid evolution and diffusion of virtual reality devices

Head-mounted displays (HMDs), which provide immersion experiences in virtual space, finally reached full-scale adoption in 2016. The technology began when different companies sought to replace human visual perception with that of virtual reality (VR) and introduced VR-HMD devices, which achieved widespread popularity in approximately 2010. These efforts coincided with the time when VR-HMD technologies became available at affordable prices. Typical units consist of a computer that generates virtual three-dimensional (3D)-space images in real time, a position-tracking technology that tracks human movements, and a small display that enables high-definition images. With prices one-tenth of those in the past, these products are rapidly becoming popular in many households.

2. Immersion experience generated by VR devices

The experience of replacing human eyesight with a virtual space is a sensation that humans have never before experienced. It is completely different from that of 3D television, where the audience passively watches images that seem to pop out of the screen. With a VR device, the user’s eyesight is completely covered by a virtual space. A movement of the user’s neck triggers the rendering of additional virtual space, which rapidly heightens the sense of immersion. In addition, moving the face closer makes an image look larger.

Successful VR-HMDs have inspired great anticipation for the development of a device that replaces tactile and other human senses with VR. However, a method to recreate the sensation of touching an object, lifting it to feel its weight and feeling its movement or temperature as well as other sensations is still in development. An enormous technological barrier needs to be overcome before minute finger movements can be sensed together with appropriate tactile sensations provided as a response. To accomplish this, the development of an intuitive controller that minimizes the user’s recognition of isolation from the real world, and a combination of a visual effects device and a suit that enhances sensory experiences will be required.

3. Pioneering practical applications of augmented reality

The development of technologies called augmented reality (AR) and mixed reality (MR) also made significant progress during 2016. While VR means a complete replacement of the real-world view with virtual space, in AR the real world is blended together with virtual space. Unlike VR, the goggles used in an AR-HMD are transparent, letting the user see the real world. Virtual space is pasted on real-world space to create an intermingled world. Expected applications for this technology include learning, training, and design. For example, the user can make a gigantic aircraft engine appear virtually in the room, observing its details at will by getting close to and into the object.

Many developers and forward-thinking companies have been focusing their investments on AR to speed its evolution. However, AR-HMDs currently have several technological obstacles, and devices on the market are only for developers. One significant limitation is that the area viewable in VR is still narrow, enabling VR to blend with the real world only in a specific area. In spite of its present limitation, the superimposition of VR to a human’s natural eyesight may someday lead to a revolutionary user interface (UI) to connect humans to information systems.

AR is also expected to generate new applications designed for the flat screen of a smartphone. An example of the many commercial uses of such applications include a situation where the user takes photos of his/her room and superimposes the image with that of a new piece of furniture prior to purchase. Although VR is used for product introductions and advertisements, more widespread adoption has been hindered by psychological and physical resistance to wearing goggles. Smartphone AR can overcome this obstacle. Furthermore, AR may well be effective even under the current circumstances, where the development of a device that recreates human senses, except for eyesight, is still in its infant stage.

For example, delicate assembly work and training in medical practice require the use of actual tools or a full-sized model to repeat a task or operation while feeling a tactile sensation. AR can paste a patient body or a material that changes while being processed into the real space using VR. This enables effective and repeatable training that blends VR with a reality where an object such as a model and a tool can be physically touched.

4. Improvement of devices

An HMD interface that connects the user to a virtual space still has significant room for improvement. For example, an HMD needs to be connected by a thick cable to the base unit of a high-speed computer. For this reason, the user with an HMD cannot move about at will. An HMD should optimally be wireless. The next best thing would be to have an HMD without a base unit, which is being proposed. This HMD copies an idea from a smartphone processor that has an advanced computation ability and also conserves power use. As the built-in lenses become thinner, the current situation, where a large, heavy, black box is worn in front of the face, will gradually improve.

This technology will also be combined with the reproduction of images with a definition of over 8K, which exceeds the level recognizable by the retina, and with a response delay of 1 ms or less. In addition, with the release of more accurate, innovative, and outdoor-ready position-tracking technology, the so-called drunken issue, caused by the gap between the position information detected by humans and the virtual space, the difficulty in outdoor use, and other limitations, will most likely be mitigated. There is also a proposal to develop a system that enables collaboration without HMDs by pasting a virtual space in the physical space, with the ability to interact with it using projection mapping technology.

5. Connection with 3D space

As HMDs advance and improved interfaces are provided, it is expected that people will use VR on a daily basis. An interface called VROS (virtual reality operating system), which provides an integrated UI by combining different virtual spaces created by the computer, will emerge as well. This type of UI will probably blend the human-to-computer interactions with the real world, making us forget that it is a system UI.

However, the essence of the current fast-moving technical evolution may be a new exploration into the unknown world of human senses. Thanks to the diffusion of HMDs and the accompanying increase in the number of developers, as well as huge amounts of user feedback, in-depth knowledge about different human senses is being accumulated. For example, the size of an object identified by human eyesight has been assumed to change depending on the person’s attentiveness. This is a challenge for 3D content producers as they struggle to determine the display size of an object that humans feel is appropriate. Some have reported that the actual-world action of catching a ball flying toward you is done with different movements or speeds if the same action is presented through VR. Knowledge such as this will not only be used in the recreation of human senses as a form of entertainment, but also in the creation of more natural interfaces, contributing to the improvement of the system-human relationship.

	Rena Oi Assistant Manager, Strategy Development Section, Research and Development Headquarters, NTT DATA Corporation. She received a B.A. in sociology from Keio University, Tokyo, in 2008. After joining NTT DATA in 2008, she worked on the development of a customer contact system. She joined the NTT DATA Technology Foresight team in 2017.