Feature Articles: Media Robotics as the Boundary Connecting Real Space and Cyberspace
Presenting Material Properties with Mid-air Pseudo-haptics
With the development of image presentation and behavioral-data-acquisition technologies, interaction between a virtual object and user has become commonplace. This article introduces mid-air pseudo-haptics, a technology to provide haptic perception of material properties to the user when the virtual object is manipulated by the user’s actions. With this technology, the virtual object is deformed by the user’s actions, and the impression of the object’s softness is controlled by its deformation pattern. In the future, we will develop technology to provide richer perceptual material properties to virtual objects.
Keywords: mid-air pseudo-haptics, material perception, illusion
In Hollywood science fiction movies, we often see scenes in which the main character in a futuristic world skillfully manipulates images in the air. What would be the haptic nature of images floating in the air? This simple question is the motivation for this study. Mid-air image representation is no longer science fiction but is now possible with optical devices and head-mounted displays. NTT’s “Kirari!” immersive telepresence technology is another means of displaying images in the air. Research on presenting haptic sensation to aerial images (virtual objects) is being conducted. Various methods have been proposed, but most use devices that provide physical haptic sensations.
In this study, we investigated the possibility of manipulating the material properties of virtual objects by changing their appearance in response to user actions. The user obviously cannot directly touch a virtual object; therefore, we thought that it would be possible to provide haptic properties to a virtual object by using perceptual illusion. Specifically, we investigated whether it would be possible to provide an illusory haptic sensation to a virtual object when we see a causal relationship between the user’s manipulation of the object and change in the object’s appearance.
Before explaining our research in detail, let us consider the pseudo-haptic effect. Suppose a user moves the cursor displayed on the computer screen with the mouse. Now, suppose the user keeps moving the mouse in the same manner, but the cursor suddenly slows down. Studies have shown that this type of slowdown leaves the user with a sense of resistance and increases the perceived weight of the cursor. This effect is called pseudo-haptics . Pseudo-haptics is an illusion that involves multiple sensory modes including vision, touch, perception, and proprioception, but the effect of vision on other modalities is particularly critical. Pseudo-haptics can manipulate impressions of the shape and stiffness of virtual objects, as well as their weight and resistance.
3. Mid-air pseudo-haptics
The pseudo-haptic sensation generated when the user touches a haptic presentation device has been investigated . In our study, we investigated whether it would be possible to generate pseudo-haptic sensation when there is no physical contact. We call such an effect mid-air pseudo-haptics . To achieve this effect, two problems had to be solved. The first was how to track the user’s hand movements, which was easily solved using an optical hand-tracking sensor (Fig. 1(a)). The second problem was what kind of changes in the virtual object associated with the tracked hand motion would create the perception of material properties. This problem was not straightforward because it involves human perceptual characteristics but is a very interesting question.
4. Key points of this study
We addressed mid-air pseudo-haptics by focusing on the following three points.
4.1 Poisson effect and Poisson ratio
We investigated the user’s action of pulling a virtual object. When a user applies tension by pulling an elastic object in the horizontal direction, a compressive force acts on it in the vertical direction. For example, when we pull on a rubber object, it stretches in the direction of the pull but decreases in thickness at the center. This phenomenon is called the Poisson effect. We devised a system in which a virtual object stretches horizontally according to the movement of the hand when the user makes a pulling action in that direction (Fig. 1(b)). At the same time, the object deforms vertically according to the Poisson’s ratio, which is the ratio of the strain in the material in the direction of the applied force to the strain occurring in the direction perpendicular to that direction. In the natural world, the Poisson’s ratio generally does not exceed 0.5. We investigated how the Poisson effect affected the perception of the stiffness of a virtual object by varying the Poisson ratio applied to the object’s deformation.
4.2 Ratio of hand movement to object deformation
Previous research  on pseudo-haptics has shown that the smaller the ratio of the amount of change on the screen to the amount of action by the user, the more resistance and heaviness is perceived. In our study, we varied the ratio of the amount of deformation of the virtual object to the amount of action taken by the user to pull the virtual object and investigated the relationship between this ratio and perceived stiffness of the virtual object. When this ratio becomes larger, the amount of deformation of the virtual object becomes larger for the same amount of movement. Therefore, the perceived softness of the object should increase accordingly.
4.3 Consistency between action and expected deformation
When a user pulls a virtual object horizontally, and the object deforms horizontally in response, the relationship between action and deformation is consistent. However, it is not clear whether the consistency of action and object deformation is necessary to manipulate the perceived material properties of an object. We addressed this issue by investigating experimental conditions for manipulating the consistency of action and object deformation, i.e., consistent and inconsistent. Under the consistent condition, when the user pulled the object horizontally, the object extended horizontally. Under the inconsistent condition, when the user pulled the object horizontally, the object contracted vertically.
5. Experimental results and discussion
Figure 2 shows the results of an experiment in which the stiffness of a virtual object was evaluated on five levels of Poisson’s ratio. When the object had a Poisson’s ratio of 0.5, it was judged to be less stiff than when it had lower Poisson’s ratios. Regarding the ratio of deformation to action, the higher the ratio, the less stiff the object was reported as being. It is interesting to note that regardless of the consistency between action and deformation, the same pattern of evaluation was observed.
From these results, the following conclusions can be drawn.
6. The future targeted with this technology
The Innovative Optical and Wireless Network (IOWN) will achieve high-speed, high-capacity communications in 2030. At that time, technology to present a large amount of information to users in a natural manner will be essential. As described in the IOWN concept document, NTT is aiming for a futuristic user interface “Point of Atmosphere (PoA)” to create a world where people are in harmony with their digital environment without being aware of surrounding devices .
The pseudo-haptic technology proposed in this study enhances the sense of realism in the virtual world by extending the experience of real object manipulation to a virtual object so that the virtual object appears to stretch when the user makes a pulling action. This type of object manipulation and impression of material properties in augmented reality will be necessary for future interface design. We hope to bring PoA closer to reality by promoting basic research for the development of not only mid-air pseudo-haptics but also perception-based presentation technology.