Recognizing Liquid from Image Motion and Image Deformation

1. Introduction

Humans can easily recognize an object such as a chair, a desk, or a book; we call this object recognition. In addition, we are also fairly proficient at recognizing materials such as metal, wool, plastic, and glass (material recognition). The visual mechanism of object recognition was clarified in the field of visual science over a decade ago. Meanwhile, the mechanism for material recognition had been virtually ignored [1]. Researchers working in various scientific areas such as psychophysics, computer graphics, and computer vision have recently started investigating how humans (or machines) recognize materials [2].

Still, most of the research that has been done has explored how human observers recognize static materials whose shape does not change over time. Very little research has focused on how human observers are able to recognize nonrigid materials such as rubber that easily deform, or how the visual system can recognize liquids and their viscosity. Here we focused on liquids and sought to clarify the underlying mechanism of liquid recognition. In this article, we first describe our scientific investigations of how humans recognize liquid. Then we describe image presentation techniques that can alter the impression of materials through the manipulation of several characteristics of image motion and image deformation, which strongly affect the recognition of liquid.

2. Recognition of liquid viscosity from image motion speed

We can easily discriminate a low-viscosity liquid such as water from a high-viscosity liquid such as honey. How can the visual system recognize a liquid’s viscosity? In everyday life, it is easy for us to recognize the viscosity of liquids in still images. However, it may also be intuitively known that a low-viscosity liquid flows more quickly and smoothly than a high-viscosity liquid. We investigated how image motion generated by a flowing liquid contributed to the recognition of liquid viscosity [3].

We first created computer-rendered movies that simulated various scenes containing the flows of opaque liquids with different viscosities (Fig. 1). Next, we calculated the optical flow fields of liquid flows to quantify to what extent each pixel in the simulation movies moved between frames. On the basis of the calculated optical flow fields, we moved low-pass white noise within each of the 15 × 15 matrices of noise patches (noise motion movies). The noise motion movies lacked the static form information contained in the original movies. Thus, by using the noise motion movies, we were able to investigate how image motion information affected the recognition of a liquid’s viscosity. We asked observers to view the noise motion movies and rate the viscosity. Surprisingly, despite the fact that the noise motion movies contained no static form information, the observers were able to judge the viscosity with reasonably high accuracy. The results indicate that human observers can recognize a liquid’s viscosity solely from image motion. Moreover, we also found that artificially increasing or decreasing image motion speed, respectively decreased or increased the rated viscosity values. This indicates the possibility that the visual system uses image motion speed to judge liquid viscosity.

Fig. 1. (a) Computer-rendered movies simulate the flow of an opaque liquid. (b) Optical flow fields of the liquid flow were calculated. (c) Band-pass white noise within each patch in noise motion stimuli was moved based on the optical flow fields. Human observers view the noise motion stimuli and can recognize the flow of liquid from image motion only.

3. Recognition of liquidness from smooth image motion flow

However, not all liquids that move slowly are always perceived to be highly viscous. Thus, an additional constraint is clearly necessary in order to recognize viscosity on the basis of image motion speed. In general, a liquid flows smoothly. Thus, we suggested that the additional constraint might be the smoothness of image motion flow. To investigate this idea, we scrambled the positions of the patches in the noise-motion movies to produce a more discrete, or discontinuous, image motion flow (Fig. 2), and we found that the spatial scramble of patches strongly reduced the impression of liquidness. Importantly, it became difficult to rate the viscosity under the scenario with a discontinuous image motion flow. We further investigated the image statistics that could characterize the smoothness of the image motion flow (and liquidness impression) and found that the discrete Laplacians (Laplace operators, i.e., the sum of the second derivatives of image motion vectors) were negatively correlated with the rated impressions of liquidness. These results indicate that the visual system can use image motion speed to recognize liquid viscosity only when the image motion flow is spatially smooth and, resultantly, when observers interpret the image motion flow as liquid.

Fig. 2. Smooth and discrete image motion flows create respectively strong and weak impressions of liquid flow.

4. Recognition of a transparent liquid from image deformation

We have thus far summarized our investigations of how humans see an opaque liquid and its viscosity. However, many liquids such as water, honey, and oil are transparent. When light hits the surface of a transparent liquid, some portions of the light are reflected at the surface of the liquid (Fig. 3) and produce highlights in the resulting image. Other portions of the light penetrate the body of the liquid, are reflected at the underlying surface, and are refracted at the liquid’s surface. The refraction tends to cause image deformation of the scene underneath the liquid, and the pattern of image deformation depends critically on the three-dimensional structure of the liquid’s surface.

Fig. 3. (a) Light reflection and refraction at the surface of a transparent liquid. (b) Left: original image of an underlying scene. Right: deformation of the left image due to refraction at the surface of a transparent liquid.

We reported in Kawabe, Maruya, & Nishida (2013) that the visual system uses image deformation to see a transparent liquid [4]. We simulated the flow of a transparent liquid that generated the refraction-based dynamic image deformation of an underlying scene. In rendering, we removed reflection components at the surface of a transparent liquid to investigate the pure effect of dynamic image deformation on the recognition of a transparent liquid. We asked observers to view simulation movies and rate their impression of a transparent liquid. The observers were able to recognize a transparent liquid from dynamic image deformation even when the liquid lacked the reflection components. In further studies, we found that human observers often reported the presence of a transparent liquid when dynamic image deformation contained high-amplitude spectra in specific bands of the image deformation’s spatiotemporal frequency. These results indicate that human observers recognize a transparent liquid from dynamic image deformation due to refraction.

5. Altering the impression of liquid materials in movies and printed materials

We recently succeeded in extending the obtained scientific knowledge to information presentation techniques that can alter the impression of liquids in movies and prints. For example, our finding that image motion speed can alter the recognition of liquid viscosity led to the idea that manipulating the amplitude spectra of the spatiotemporal frequency can alter the impression of liquid viscosity. Specifically, when we reduced the amplitude spectra of a high temporal frequency in a movie containing a thin liquid, the perceived viscosity of the liquid increased. In another example, we demonstrated that when dynamic image deformation was added to an online movie captured with a camera, the viewers were able to get the impression that a scene captured by the camera was behind the flow of a transparent liquid.

Recently, we developed a new light projection technique that was inspired by our understanding of how people see transparent liquids (Fig. 4). When we see a transparent liquid, we simultaneously perceive both an underlying static scene and a transparent layer. Thus, the brain possibly has two sets of representations, one corresponding to the static scene and the other to the dynamic liquid layer. However, what image features are causing these representations? Here, we tentatively hypothesize that the static scene representation is obtained from the direct current (DC) components of the temporal frequency, while the representation of the dynamic liquid layer comes from residual motion components. When we projected residual motion components onto the printed version of the DC components of the temporal frequency, we found that the printed DC components were perceptually deformed. Moreover, when the DC and residual motion components were obtained from the same movies of a liquid flow, the light projection of the residual motion components onto the printed DC components added the impression of a liquid flow to the printed DC components. With this light projection technique, it is possible to add the impression of a liquid flow to a floor, a wall, or paper.

Fig. 4. Schematic illustration of our novel light projection technique. We first extract DC components of temporal frequency and residual motion components separately from a movie of a liquid flow. Then we print out the DC components of the temporal frequency. Finally, we project the residual motion components onto the printed DC components. When observers see this combination of the projected motion components and printed DC components, they see a transparent liquid layer in front of the printed DC components.

6. Concluding remarks

We investigated how the brain works when a person sees materials. We plan to accumulate further scientific knowledge about what happens in the brain when we see various materials and explore the development of other novel and useful information presentation techniques that can communicate realistic and vivid material impressions.

References

	Takahiro Kawabe Research Scientist, Sensory Representation Group, Human Information Science Laboratories, NTT Communication Science Laboratories. He received the Ph.D. in psychology from Kyushu University, Fukuoka, in 2005. In 2011, he joined NTT Communication Science Laboratories, where he studies human material recognition and cross-modal perception. He received the 2013 JPA (Japanese Psychological Association) Award for International Contributions to Psychology: Award for Distinguished Early and Middle Career Contributions. He is a member of the Vision Sciences Society and the Vision Society of Japan.
	Masataka Sawayama Research Associate, Sensory Representation Group, Human Information Science Laboratories, NTT Communication Science Laboratories. He received the M.A. and Ph.D. in psychology from Chiba University in 2010 and 2013, respectively. In 2013, he joined NTT Communication Science Laboratories, where he studies human material processing. He received the Best Presentation Award in 2009 from the Japanese Psychonomic Society and in 2011 from the Vision Society of Japan. He is a member of the Vision Sciences Society, the Vision Society of Japan, the Japanese Psychonomic Society, and the JPA.
	Kazushi Maruya Senior Research Scientist, Sensory Representation Group, Human Information Science Laboratories, NTT Communication Science Laboratories. He received the M.A. and Ph.D. in psychology from the University of Tokyo in 2001 and 2005, respectively. He joined NTT Communication Science Laboratories in 2008. He was a researcher in the Intelligent Modeling Laboratory, the University of Tokyo, from 2004 to 2005, and a visiting scientist at the Department of Psychology, Vanderbilt University, Nashville, TN, USA, from 2006 to 2008. He received best presentation awards at the 2001 and 2004 meetings of the Vision Society of Japan and at the 2003 and 2004 meetings of the Japanese Psychonomic Society. He is currently engaged in research on visual motion processing. He is also interested in human computer interface design for reading digital texts. He is a member of the Vision Sciences Society and the Vision Society of Japan.
	Shin’ya Nishida Senior Distinguished Scientist and Group Leader, Sensory Representation Group, Human Information Science Laboratories, NTT Communication Science Laboratories. He received the M.A. and Ph.D. in psychology from Kyoto University in 1987 and 1996, respectively. He joined NTT Basic Rsearch Laboratories in 1992 and moved to NTT Communication Science Laboratories in 1999. He is an expert in psychophysical research on human visual processing, particularly motion perception, cross-attribute/modality integration, time perception, and material perception. He is the current president of the Vision Society of Japan, and an editorial board member of the Journal of Vision and Vision Research.