Does Exploratory Hand Movement Provide Information on Tactile Sensations?

1. Introduction

People can feel a variety of tactile sensations (e.g., hardness and roughness) by actively exploring the surface of an object with their hands and fingers. While haptic-display technology and product-surface design technology, which are applied to induce such a predetermined tactile sensation, have been attracting attention, how these technologies actually create an end-user tactile sensation has received little attention. In fact, even if many people touch the same object, they will not all feel the same tactile sensation. For example, how surface roughness is perceived varies from person to person [1], and the perceived roughness can change as the force of the fingers pushing the surface changes [2, 3]. Accordingly, even if people are using the same haptic display or touching the surface of the same object, the tactile sensation felt by each person is likely to change from person to person. One way to determine what type of tactile sensation a person actually felt is to actually ask them; however, from the viewpoint of an end user, having the person report their tactile sensation every time they touch something is inconvenient. It would therefore be useful if it were possible to estimate externally—by using an objective measurement index—how a person felt when touching an object.

This article focuses on exploratory movement of a hand while touching an object as a source of information (i.e., a measurement index) about a tactile sensation. In many previous tactile studies, such hand movements were treated as something that should be controlled, and the tactile ability of a stationary hand and a hand performing simple movements (such as linear ones) were investigated. However, they overlooked the role of hand movements. It is known that how hands move on objects changes according to what type of tactile sensation is to be determined [4], and it is suggested that the movement of the hand plays the role of a window that reflects the tactile sensation. This article introduces my efforts to investigate how exploratory hand movements provide information about tactile sensation.

2. People know the relationship between exploratory hand movements and tactile sensation

Do exploratory hand movements actually contain tactile information? If so, we might be able to use that information. In other words, by looking at the hand movements of someone else touching an object, we might be able to imagine how that object feels when touched. Although this situation may seem strange, we are already familiar with it through television (TV) commercials (Fig. 1). For example, we have seen scenes in which skin is stroked by a hand, a towel is pushed by a hand, or a plate is rubbed with a finger. Watching such scenes, we can imagine the smoothness of the skin after shaving, the softness of a towel washed using softener, and the frictional squeak of greasy plates that have been washed using detergent. This imaginative ability of people was previously investigated through psychophysical experiments [5].

Fig. 1. TV commercials use hand movements to convey a rich sensation of touch to viewers.

First, six participants (each called a toucher) touched an object (such as wood or glass) and evaluated the tactile sensation they felt. The hand movements of each toucher were recorded while he/she was touching his/her target object. Ten other participants (each called an observer) were asked to observe the touchers’ hand movements and guess how the touchers felt when they touched the target object. To investigate whether hand movement is actually used as an information source, a toucher’s hand movement was reproduced with moving light points that exclude information other than hand movement (e.g., skin deformation and nail color) (Fig. 2). The experimental results indicated that all ten observers responded in a similar manner. That is, when the observers watched the light-point hand movements of a toucher touching an object that the toucher felt was sticky, they tended to judge that the toucher was touching a sticky object. When the observers watched the light-point hand movements of a toucher touching an object that the toucher felt was fluffy, they tended to judge that the toucher was touching a fluffy object. The results indicated that people can use a common strategy—based on hand movements—for estimating tactile sensations. It is thought that people possess such a common strategy because the movement of hands touching objects contains information on the tactile sensation felt by the toucher.

Fig. 2. A scene from a video of hand movements of a toucher in a previous experiment [5]. The hand movements were drawn as moving light points and observed by observers.

3. Analysis of exploratory hand movements

What exploratory hand movements actually contain certain tactile information? An experiment was conducted in which participants were asked to touch various objects that they encounter on a daily basis and report their tactile sensations [6]. By analyzing the measured hand movements, what movements can explain these tactile sensations were determined. The experimental results revealed that the speed of a stroking hand and the magnitude of the hand’s pushing force contained information on the hardness, roughness, stickiness, and warmth felt by the participants. Two trends were identified: (i) the evaluations of hardness and temperature (related to pushing and static contact) are related to the magnitude of the pushing force and (ii) the evaluation of roughness and stickiness (related to surface friction) is related to the speed of the stroking hand. Even more interesting is that active eye movements also contain tactile information. The reason for this is still open to question, but exploratory eye movements might be related to the physical properties of the object surface.

To estimate tactile sensations, it may be useful to combine measurements of hand speed and active eye movements. As shown in Fig. 3, active eye movements contain information on hardness, warmth, and roughness, and the speed of the hand contains information on stickiness, roughness, and warmth. From these results, it might be possible to specify which of the four main touch sensations—hardness-softness, roughness-smoothness, stickiness-slipperiness, and warmth-cold—are felt. For example, when a person moves his/her eyes and hands, he/she is feeling a smooth sensation (first quadrant in Fig. 3), and when a person is not moving his/her eyes but moving his/her hands, he/she is feeling a warm sensation (fourth quadrant in Fig. 3).

Fig. 3. Correlation between active movements of the eyes and hands and evaluation of tactile sensation. For example, active eye movements (saccade motion) have a positive correlation with coldness, which indicates that the eyes move more actively when someone is touching an object judged as cold. On the contrary, the stroking motion of the hand has a negative correlation with coldness, which means that the hand moves more actively when someone is touching an object judged as warm.

4. From perceived physical properties to evaluated preference

Hardness, roughness, stickiness, and warmth are tactile sensations that are closely related to the physical properties of an object. For example, perceived hardness is strongly affected by compliance [7, 8], and perceived roughness is strongly affected by friction [9–11]. Perception of these sensations is necessary for properly adjusting movements, such as when a person is gripping an object. However, tactile sensation has not only this function but also a function that determines how favorable the touched object is to the person touching it. Differing from sight and hearing, tactile sensation is a perception that occurs when the body is directly in contact with an object, so this evaluation of tactile preference is important. I also investigated whether movements of a hand touching an object also contain this perception of preference [12]. First, I showed that the evaluation of preference can change if the manner of touching changes. For example, woody materials were evaluated as preferable when being stroked but evaluated as less preferable when being pressed. This means that there is a close relationship between how an object is touched and tactile sensation (preference evaluation). Next, I discussed the analysis of the movement of a hand touching an object and that the features related to stroking motion (such as hand speed) contained information on the evaluation of preference. On the contrary, the features related to pushing did not contain such information. These results suggest that measuring stroking motion will allow us to estimate evaluated preference.

5. Benefits of measuring hand movements and applying the measurement results

According to the results presented thus far, it may be possible to estimate how a person feels by measuring his/her hand movements. Therefore, what are the benefits of measuring and analyzing hand movements and how does this differ from measuring and analyzing physical properties (such as compliance and friction)? One benefit of measuring hand movements is making it possible to consider the effects of individual differences and manner of touching, as described in the introduction. For example, individual differences due to differences in search strategies of end users cannot be determined by measuring the physical properties of touched objects; however, they might be determinable by measuring hand movements. In fact, which information is used preferentially, spatial information or temporal information regarding the perception of roughness, depends on the person [1]. Since it is also known that hand movements differ according to the manner in which such spatiotemporal information is used [13], measuring such movements might allow us to consider individual differences. As stated above, the tactile sensation (evaluation of roughness or evaluation of preference) can change when the manner of touching an object changes even if the same person touches the object. This fact cannot be understood from measuring the physical properties of the object; instead, it could be understood by measuring hand movements. To analyze the physical properties of objects, we must measure the physical properties of all objects end users might touch. To analyze hand movements, however, all we need to do is attach a motion sensor to the hand. This approach has a high affinity with a haptic display worn on the arm; that is, attaching such a device to measure the movement of the arm makes it possible to continuously monitor the tactile sensation felt by the wearer.

6. Conclusion

Studies on whether hand movements contain information on tactile sensation were introduced. The results of these studies are expected to contribute to not only establishing an objective method for estimating tactile sensation but also revealing human-tactile mechanisms that are still unknown.

References

[1]	M. Natsume, Y. Tanaka, and A. M. L. Kappers, “Individual Differences in Cognitive Processing for Roughness Rating of Fine and Coarse Textures,” PLoS One, Vol. 14, No. 1, e0211407, 2019.
[2]	S. J. Lederman and M. M. Taylor, “Fingertip Force, Surface Geometry, and the Perception of Roughness by Active Touch,” Percept. Psychophys., Vol. 12, No. 5, pp. 401–408, 1972.
[3]	S. J. Lederman, “Tactile Roughness of Grooved Surface: The Touching Process and Effects of Macro- and Microsurface Structure,” Percept. Psychophys., Vol. 16, No. 2, pp. 385–395, 1974.
[4]	S. J. Lederman and R. L. Klatzky, “Hand Movements: A Window into Haptic Object Recognition,” Cogn. Psychol., Vol. 19, No. 3, pp. 342–368, 1987.
[5]	T. Yokosaka, S. Kuroki, J. Watanabe, and S. Nishida, “Estimating Tactile Perception by Observing Explorative Hand Motion of Others,” IEEE Trans. Haptics, Vol. 11, No. 2, pp. 192–203, 2018.
[6]	T. Yokosaka, S. Kuroki, J. Watanabe, and S. Nishida, “Linkage between Free Exploratory Movements and Subjective Tactile Ratings,” IEEE Trans. Haptics, Vol. 10, No. 2, pp. 217–225, 2017.
[7]	R. Harper and S. S. Stevens, “Subjective Hardness of Compliant Materials,” Q. J. Exp. Physiol., Vol. 16, No. 3, pp. 204–215, 1964.
[8]	S. Wongsriruksa, P. Howes, M. Conreen, and M. Miodownik, “The Use of Physical Property Data to Predict the Touch Perception of Materials,” Mater. Des., Vol. 42, pp. 238–244, 2012.
[9]	L. A. Stone, “Subjective Roughness and Smoothness for Individual Judges,” Psychon. Sci., Vol. 9, No. 6, pp. 347–348, 1967.
[10]	R. T. Verrillo, S. J. Bolanowski, and F. P. McGlone, “Subjective Magnitude of Tactile Roughness,” Somatosens. Mot. Res., Vol. 16, No. 4, pp. 352–360, 1999.
[11]	G. Ekman, J. Hosman, and B. Lindstroem, “Roughness, Smoothness, and Preference: A Study of Quantitative Relations in Individual Subjects,” J. Exp. Psychol, Vol. 70, pp. 18–26, 1965.
[12]	T. Yokosaka, M. Inubushi, S. Kuroki, and J. Watanabe, “Frequency of Switching Touching Mode Reflects Tactile Preference Judgment,” Sci. Rep., Vol. 10, 3022, 2020.
[13]	E. Gamzu and E. Ahissar, “Importance of Temporal Cues for Tactile Spatial-Frequency Discrimination,” J. Neurosci., Vol. 21, No. 18, pp. 7416–7427, 2001.