Presenting Material Properties with Mid-air Pseudo-haptics

1. Motivation

In Hollywood science fiction movies, we often see scenes in which the main character in a futuristic world skillfully manipulates images in the air. What would be the haptic nature of images floating in the air? This simple question is the motivation for this study. Mid-air image representation is no longer science fiction but is now possible with optical devices and head-mounted displays. NTT’s “Kirari!” immersive telepresence technology is another means of displaying images in the air. Research on presenting haptic sensation to aerial images (virtual objects) is being conducted. Various methods have been proposed, but most use devices that provide physical haptic sensations.

In this study, we investigated the possibility of manipulating the material properties of virtual objects by changing their appearance in response to user actions. The user obviously cannot directly touch a virtual object; therefore, we thought that it would be possible to provide haptic properties to a virtual object by using perceptual illusion. Specifically, we investigated whether it would be possible to provide an illusory haptic sensation to a virtual object when we see a causal relationship between the user’s manipulation of the object and change in the object’s appearance.

2. Pseudo-haptics

Before explaining our research in detail, let us consider the pseudo-haptic effect. Suppose a user moves the cursor displayed on the computer screen with the mouse. Now, suppose the user keeps moving the mouse in the same manner, but the cursor suddenly slows down. Studies have shown that this type of slowdown leaves the user with a sense of resistance and increases the perceived weight of the cursor. This effect is called pseudo-haptics [1]. Pseudo-haptics is an illusion that involves multiple sensory modes including vision, touch, perception, and proprioception, but the effect of vision on other modalities is particularly critical. Pseudo-haptics can manipulate impressions of the shape and stiffness of virtual objects, as well as their weight and resistance.

3. Mid-air pseudo-haptics

The pseudo-haptic sensation generated when the user touches a haptic presentation device has been investigated [2]. In our study, we investigated whether it would be possible to generate pseudo-haptic sensation when there is no physical contact. We call such an effect mid-air pseudo-haptics [3]. To achieve this effect, two problems had to be solved. The first was how to track the user’s hand movements, which was easily solved using an optical hand-tracking sensor (Fig. 1(a)). The second problem was what kind of changes in the virtual object associated with the tracked hand motion would create the perception of material properties. This problem was not straightforward because it involves human perceptual characteristics but is a very interesting question.

4. Key points of this study

We addressed mid-air pseudo-haptics by focusing on the following three points.

5. Experimental results and discussion

Figure 2 shows the results of an experiment in which the stiffness of a virtual object was evaluated on five levels of Poisson’s ratio. When the object had a Poisson’s ratio of 0.5, it was judged to be less stiff than when it had lower Poisson’s ratios. Regarding the ratio of deformation to action, the higher the ratio, the less stiff the object was reported as being. It is interesting to note that regardless of the consistency between action and deformation, the same pattern of evaluation was observed.

Fig. 2. Experimental results.

From these results, the following conclusions can be drawn.

• By manipulating the relationship between the user’s action and deformation of a virtual object, we can create a mid-air pseudo-haptic sensation for the user.
• Using the Poisson effect, it is possible to manipulate the perceived stiffness of a virtual object by changing the amount of deformation of the object in accordance with the user’s action.
• Even if the user’s motion and deformation of the virtual object do not match, it does not affect the impression of the object’s softness.

6. The future targeted with this technology

The Innovative Optical and Wireless Network (IOWN) will achieve high-speed, high-capacity communications in 2030. At that time, technology to present a large amount of information to users in a natural manner will be essential. As described in the IOWN concept document, NTT is aiming for a futuristic user interface “Point of Atmosphere (PoA)” to create a world where people are in harmony with their digital environment without being aware of surrounding devices [5].

The pseudo-haptic technology proposed in this study enhances the sense of realism in the virtual world by extending the experience of real object manipulation to a virtual object so that the virtual object appears to stretch when the user makes a pulling action. This type of object manipulation and impression of material properties in augmented reality will be necessary for future interface design. We hope to bring PoA closer to reality by promoting basic research for the development of not only mid-air pseudo-haptics but also perception-based presentation technology.

References

[1]	A. Lécuyer, S. Coquillart, A. Kheddar, P. Richard, and P. Coiffet, “Pseudo-haptic Feedback: Can Isometric Input Devices Simulate Force Feedback?”, Proc. of IEEE Virtual Reality 2000 (Cat. No.00CB37048), pp. 83–90, New Brunswick, NJ, USA, Mar. 2000. https://doi.org/10.1109/vr.2000.840369
[2]	A. Lécuyer, “Simulating Haptic Feedback Using Vision: A Survey of Research and Applications of Pseudo-haptic Feedback,” Presence: Teleoperators and Virtual Environments, Vol. 18, No. 1, pp. 39–53, Feb. 2009. https://doi.org/10.1162/pres.18.1.39
[3]	T. Kawabe, “Mid-air Action Contributes to Pseudo-haptic Stiffness Effects,” IEEE Transactions on Haptics, Vol. 13, No. 1, pp. 18–24, Dec. 2019. https://doi.org/10.1109/toh.2019.2961883
[4]	A. Lécuyer, J.-M. Burkhardt, and L. Etienne, “Feeling Bumps and Holes without a Haptic Interface: the Perception of Pseudo-haptic Textures,” Proc. of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’04), pp. 239–246, Vienna, Austria, Apr. 2004. https://doi.org/10.1145/985692.985723
[5]	J. Sawada (supervisor), M. Ii, and K. Kawazoe (authors), “IOWN: Beyond the Internet,” NTT Publishing, 2019 (in Japanese).