Feature Articles: New Developments in Communication Science
Recognizing Liquid from Image Motion and Image Deformation
Our research confirms that the human visual system recognizes a liquid and its viscosity on the basis of image motion and image deformation. This scientific understanding enables us to alter the impression of materials in movies and prints by manipulating several characteristics of image motion and image deformation.
Keywords: material recognition, liquid, projection mapping
Humans can easily recognize an object such as a chair, a desk, or a book; we call this object recognition. In addition, we are also fairly proficient at recognizing materials such as metal, wool, plastic, and glass (material recognition). The visual mechanism of object recognition was clarified in the field of visual science over a decade ago. Meanwhile, the mechanism for material recognition had been virtually ignored . Researchers working in various scientific areas such as psychophysics, computer graphics, and computer vision have recently started investigating how humans (or machines) recognize materials .
Still, most of the research that has been done has explored how human observers recognize static materials whose shape does not change over time. Very little research has focused on how human observers are able to recognize nonrigid materials such as rubber that easily deform, or how the visual system can recognize liquids and their viscosity. Here we focused on liquids and sought to clarify the underlying mechanism of liquid recognition. In this article, we first describe our scientific investigations of how humans recognize liquid. Then we describe image presentation techniques that can alter the impression of materials through the manipulation of several characteristics of image motion and image deformation, which strongly affect the recognition of liquid.
2. Recognition of liquid viscosity from image motion speed
We can easily discriminate a low-viscosity liquid such as water from a high-viscosity liquid such as honey. How can the visual system recognize a liquid’s viscosity? In everyday life, it is easy for us to recognize the viscosity of liquids in still images. However, it may also be intuitively known that a low-viscosity liquid flows more quickly and smoothly than a high-viscosity liquid. We investigated how image motion generated by a flowing liquid contributed to the recognition of liquid viscosity .
We first created computer-rendered movies that simulated various scenes containing the flows of opaque liquids with different viscosities (Fig. 1). Next, we calculated the optical flow fields of liquid flows to quantify to what extent each pixel in the simulation movies moved between frames. On the basis of the calculated optical flow fields, we moved low-pass white noise within each of the 15 × 15 matrices of noise patches (noise motion movies). The noise motion movies lacked the static form information contained in the original movies. Thus, by using the noise motion movies, we were able to investigate how image motion information affected the recognition of a liquid’s viscosity. We asked observers to view the noise motion movies and rate the viscosity. Surprisingly, despite the fact that the noise motion movies contained no static form information, the observers were able to judge the viscosity with reasonably high accuracy. The results indicate that human observers can recognize a liquid’s viscosity solely from image motion. Moreover, we also found that artificially increasing or decreasing image motion speed, respectively decreased or increased the rated viscosity values. This indicates the possibility that the visual system uses image motion speed to judge liquid viscosity.
3. Recognition of liquidness from smooth image motion flow
However, not all liquids that move slowly are always perceived to be highly viscous. Thus, an additional constraint is clearly necessary in order to recognize viscosity on the basis of image motion speed. In general, a liquid flows smoothly. Thus, we suggested that the additional constraint might be the smoothness of image motion flow. To investigate this idea, we scrambled the positions of the patches in the noise-motion movies to produce a more discrete, or discontinuous, image motion flow (Fig. 2), and we found that the spatial scramble of patches strongly reduced the impression of liquidness. Importantly, it became difficult to rate the viscosity under the scenario with a discontinuous image motion flow. We further investigated the image statistics that could characterize the smoothness of the image motion flow (and liquidness impression) and found that the discrete Laplacians (Laplace operators, i.e., the sum of the second derivatives of image motion vectors) were negatively correlated with the rated impressions of liquidness. These results indicate that the visual system can use image motion speed to recognize liquid viscosity only when the image motion flow is spatially smooth and, resultantly, when observers interpret the image motion flow as liquid.
4. Recognition of a transparent liquid from image deformation
We have thus far summarized our investigations of how humans see an opaque liquid and its viscosity. However, many liquids such as water, honey, and oil are transparent. When light hits the surface of a transparent liquid, some portions of the light are reflected at the surface of the liquid (Fig. 3) and produce highlights in the resulting image. Other portions of the light penetrate the body of the liquid, are reflected at the underlying surface, and are refracted at the liquid’s surface. The refraction tends to cause image deformation of the scene underneath the liquid, and the pattern of image deformation depends critically on the three-dimensional structure of the liquid’s surface.
We reported in Kawabe, Maruya, & Nishida (2013) that the visual system uses image deformation to see a transparent liquid . We simulated the flow of a transparent liquid that generated the refraction-based dynamic image deformation of an underlying scene. In rendering, we removed reflection components at the surface of a transparent liquid to investigate the pure effect of dynamic image deformation on the recognition of a transparent liquid. We asked observers to view simulation movies and rate their impression of a transparent liquid. The observers were able to recognize a transparent liquid from dynamic image deformation even when the liquid lacked the reflection components. In further studies, we found that human observers often reported the presence of a transparent liquid when dynamic image deformation contained high-amplitude spectra in specific bands of the image deformation’s spatiotemporal frequency. These results indicate that human observers recognize a transparent liquid from dynamic image deformation due to refraction.
5. Altering the impression of liquid materials in movies and printed materials
We recently succeeded in extending the obtained scientific knowledge to information presentation techniques that can alter the impression of liquids in movies and prints. For example, our finding that image motion speed can alter the recognition of liquid viscosity led to the idea that manipulating the amplitude spectra of the spatiotemporal frequency can alter the impression of liquid viscosity. Specifically, when we reduced the amplitude spectra of a high temporal frequency in a movie containing a thin liquid, the perceived viscosity of the liquid increased. In another example, we demonstrated that when dynamic image deformation was added to an online movie captured with a camera, the viewers were able to get the impression that a scene captured by the camera was behind the flow of a transparent liquid.
Recently, we developed a new light projection technique that was inspired by our understanding of how people see transparent liquids (Fig. 4). When we see a transparent liquid, we simultaneously perceive both an underlying static scene and a transparent layer. Thus, the brain possibly has two sets of representations, one corresponding to the static scene and the other to the dynamic liquid layer. However, what image features are causing these representations? Here, we tentatively hypothesize that the static scene representation is obtained from the direct current (DC) components of the temporal frequency, while the representation of the dynamic liquid layer comes from residual motion components. When we projected residual motion components onto the printed version of the DC components of the temporal frequency, we found that the printed DC components were perceptually deformed. Moreover, when the DC and residual motion components were obtained from the same movies of a liquid flow, the light projection of the residual motion components onto the printed DC components added the impression of a liquid flow to the printed DC components. With this light projection technique, it is possible to add the impression of a liquid flow to a floor, a wall, or paper.
6. Concluding remarks
We investigated how the brain works when a person sees materials. We plan to accumulate further scientific knowledge about what happens in the brain when we see various materials and explore the development of other novel and useful information presentation techniques that can communicate realistic and vivid material impressions.