Feature Articles: Media Robotics as the Boundary Connecting Real Space and Cyberspace

Presenting Material Properties with Mid-air Pseudo-haptics

Takahiro Kawabe

Abstract

With the development of image presentation and behavioral-data-acquisition technologies, interaction between a virtual object and user has become commonplace. This article introduces mid-air pseudo-haptics, a technology to provide haptic perception of material properties to the user when the virtual object is manipulated by the user’s actions. With this technology, the virtual object is deformed by the user’s actions, and the impression of the object’s softness is controlled by its deformation pattern. In the future, we will develop technology to provide richer perceptual material properties to virtual objects.

Keywords: mid-air pseudo-haptics, material perception, illusion

PDF

1. Motivation

In Hollywood science fiction movies, we often see scenes in which the main character in a futuristic world skillfully manipulates images in the air. What would be the haptic nature of images floating in the air? This simple question is the motivation for this study. Mid-air image representation is no longer science fiction but is now possible with optical devices and head-mounted displays. NTT’s “Kirari!” immersive telepresence technology is another means of displaying images in the air. Research on presenting haptic sensation to aerial images (virtual objects) is being conducted. Various methods have been proposed, but most use devices that provide physical haptic sensations.

In this study, we investigated the possibility of manipulating the material properties of virtual objects by changing their appearance in response to user actions. The user obviously cannot directly touch a virtual object; therefore, we thought that it would be possible to provide haptic properties to a virtual object by using perceptual illusion. Specifically, we investigated whether it would be possible to provide an illusory haptic sensation to a virtual object when we see a causal relationship between the user’s manipulation of the object and change in the object’s appearance.

2. Pseudo-haptics

Before explaining our research in detail, let us consider the pseudo-haptic effect. Suppose a user moves the cursor displayed on the computer screen with the mouse. Now, suppose the user keeps moving the mouse in the same manner, but the cursor suddenly slows down. Studies have shown that this type of slowdown leaves the user with a sense of resistance and increases the perceived weight of the cursor. This effect is called pseudo-haptics [1]. Pseudo-haptics is an illusion that involves multiple sensory modes including vision, touch, perception, and proprioception, but the effect of vision on other modalities is particularly critical. Pseudo-haptics can manipulate impressions of the shape and stiffness of virtual objects, as well as their weight and resistance.

3. Mid-air pseudo-haptics

The pseudo-haptic sensation generated when the user touches a haptic presentation device has been investigated [2]. In our study, we investigated whether it would be possible to generate pseudo-haptic sensation when there is no physical contact. We call such an effect mid-air pseudo-haptics [3]. To achieve this effect, two problems had to be solved. The first was how to track the user’s hand movements, which was easily solved using an optical hand-tracking sensor (Fig. 1(a)). The second problem was what kind of changes in the virtual object associated with the tracked hand motion would create the perception of material properties. This problem was not straightforward because it involves human perceptual characteristics but is a very interesting question.

4. Key points of this study

We addressed mid-air pseudo-haptics by focusing on the following three points.

4.1 Poisson effect and Poisson ratio

We investigated the user’s action of pulling a virtual object. When a user applies tension by pulling an elastic object in the horizontal direction, a compressive force acts on it in the vertical direction. For example, when we pull on a rubber object, it stretches in the direction of the pull but decreases in thickness at the center. This phenomenon is called the Poisson effect. We devised a system in which a virtual object stretches horizontally according to the movement of the hand when the user makes a pulling action in that direction (Fig. 1(b)). At the same time, the object deforms vertically according to the Poisson’s ratio, which is the ratio of the strain in the material in the direction of the applied force to the strain occurring in the direction perpendicular to that direction. In the natural world, the Poisson’s ratio generally does not exceed 0.5. We investigated how the Poisson effect affected the perception of the stiffness of a virtual object by varying the Poisson ratio applied to the object’s deformation.


Fig. 1. (a) Experimental setup. Hand movement is tracked using an optical hand-tracking sensor, and the virtual object is deformed in accordance with the movement. (b) Deformation of the object resulting from hand movement.

4.2 Ratio of hand movement to object deformation

Previous research [4] on pseudo-haptics has shown that the smaller the ratio of the amount of change on the screen to the amount of action by the user, the more resistance and heaviness is perceived. In our study, we varied the ratio of the amount of deformation of the virtual object to the amount of action taken by the user to pull the virtual object and investigated the relationship between this ratio and perceived stiffness of the virtual object. When this ratio becomes larger, the amount of deformation of the virtual object becomes larger for the same amount of movement. Therefore, the perceived softness of the object should increase accordingly.

4.3 Consistency between action and expected deformation

When a user pulls a virtual object horizontally, and the object deforms horizontally in response, the relationship between action and deformation is consistent. However, it is not clear whether the consistency of action and object deformation is necessary to manipulate the perceived material properties of an object. We addressed this issue by investigating experimental conditions for manipulating the consistency of action and object deformation, i.e., consistent and inconsistent. Under the consistent condition, when the user pulled the object horizontally, the object extended horizontally. Under the inconsistent condition, when the user pulled the object horizontally, the object contracted vertically.

5. Experimental results and discussion

Figure 2 shows the results of an experiment in which the stiffness of a virtual object was evaluated on five levels of Poisson’s ratio. When the object had a Poisson’s ratio of 0.5, it was judged to be less stiff than when it had lower Poisson’s ratios. Regarding the ratio of deformation to action, the higher the ratio, the less stiff the object was reported as being. It is interesting to note that regardless of the consistency between action and deformation, the same pattern of evaluation was observed.


Fig. 2. Experimental results.

From these results, the following conclusions can be drawn.

  • By manipulating the relationship between the user’s action and deformation of a virtual object, we can create a mid-air pseudo-haptic sensation for the user.
  • Using the Poisson effect, it is possible to manipulate the perceived stiffness of a virtual object by changing the amount of deformation of the object in accordance with the user’s action.
  • Even if the user’s motion and deformation of the virtual object do not match, it does not affect the impression of the object’s softness.

6. The future targeted with this technology

The Innovative Optical and Wireless Network (IOWN) will achieve high-speed, high-capacity communications in 2030. At that time, technology to present a large amount of information to users in a natural manner will be essential. As described in the IOWN concept document, NTT is aiming for a futuristic user interface “Point of Atmosphere (PoA)” to create a world where people are in harmony with their digital environment without being aware of surrounding devices [5].

The pseudo-haptic technology proposed in this study enhances the sense of realism in the virtual world by extending the experience of real object manipulation to a virtual object so that the virtual object appears to stretch when the user makes a pulling action. This type of object manipulation and impression of material properties in augmented reality will be necessary for future interface design. We hope to bring PoA closer to reality by promoting basic research for the development of not only mid-air pseudo-haptics but also perception-based presentation technology.

References

[1] A. Lécuyer, S. Coquillart, A. Kheddar, P. Richard, and P. Coiffet, “Pseudo-haptic Feedback: Can Isometric Input Devices Simulate Force Feedback?”, Proc. of IEEE Virtual Reality 2000 (Cat. No.00CB37048), pp. 83–90, New Brunswick, NJ, USA, Mar. 2000.
https://doi.org/10.1109/vr.2000.840369
[2] A. Lécuyer, “Simulating Haptic Feedback Using Vision: A Survey of Research and Applications of Pseudo-haptic Feedback,” Presence: Teleoperators and Virtual Environments, Vol. 18, No. 1, pp. 39–53, Feb. 2009.
https://doi.org/10.1162/pres.18.1.39
[3] T. Kawabe, “Mid-air Action Contributes to Pseudo-haptic Stiffness Effects,” IEEE Transactions on Haptics, Vol. 13, No. 1, pp. 18–24, Dec. 2019.
https://doi.org/10.1109/toh.2019.2961883
[4] A. Lécuyer, J.-M. Burkhardt, and L. Etienne, “Feeling Bumps and Holes without a Haptic Interface: the Perception of Pseudo-haptic Textures,” Proc. of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’04), pp. 239–246, Vienna, Austria, Apr. 2004.
https://doi.org/10.1145/985692.985723
[5] J. Sawada (supervisor), M. Ii, and K. Kawazoe (authors), “IOWN: Beyond the Internet,” NTT Publishing, 2019 (in Japanese).
Takahiro Kawabe
Senior Distinguished Researcher, Leader of Sensory Representation Group, Human Information Science Laboratories, NTT Communication Science Laboratories.
He received a Doctor of Psychology from Kyushu University, Fukuoka, in 2005. In 2011, he joined NTT Communication Science Laboratories, where he studies applied aspects of human perception. He received the 2013 JPA Award for International Contributions to Psychology: Award for Distinguished Early and Middle Career Contributions from the Japanese Psychological Association. In 2018, he was also awarded the Young Scientists’ Prize of the Commendation for Science and Technology by the Minister of Education, Culture, Sports, Science and Technology. He is a member of the Vision Sciences Society and the Vision Society of Japan.

↑ TOP