Ukuzo—A Projection Mapping Technique to Give Illusory Depth Impressions to Two-dimensional
Real Objects

1. Introduction

Painters use a variety of techniques to elaborately express depth. For example, shadowing has been employed as an important technique to increase the realistic impression of objects in paintings. In medieval Europe, it was prohibited to use shadowing as a drawing technique in pictures for religious reasons. In the Renaissance period, Leonardo De Vinci, an eminent painter as well as scientist, reintroduced shadowing as a drawing technique and used it to enhance the realistic impression of his paintings [1]. Shadowing has also recently been used digitally in the graphical user interface of computers. For example, it is used to give illusory depth impressions to virtual objects shown on the computer display.

A vision science study [2] proposed an interesting illusion in which a shadow produces an illusory three-dimensional (3D) layout of an object. This is illustrated in the two pink squares shown in Fig. 1. The right square is perceived as being more separate from its background than the left one. In fact, the two squares have identical image information. The difference between them is the spatial relationship between the square and its shadow. The distance between the right square and its shadow is larger than the distance between the left one and its shadow. The previous study reported that as the distance between an object and a shadow increases, the object is perceived to apparently float up from its background to a greater extent [2].

Fig. 1. Effect of a shadow on the depth of an object.

Another study [3] on a light projection technique reported that light projection of a shadow motion can give wobbly motion impressions to a miniature car, indicating that the projection of a cast shadow can give illusory motion impressions to a real object.

We recently developed a light projection technique to give illusory depth perceptions that make it look as if an object in a 2D image was floating upward. This is caused by perceptual effects coming from the projected pattern of a shadow. We call this technique Ukuzo.

In this article, the hardware and software systems of Ukuzo are explained, and perceptual experiences that Ukuzo offers to observers are described. The role of light projection techniques in future applications of spatial augmented reality is also explained.

2. Ukuzo system

A photograph of the Ukuzo hardware system is shown in Fig. 2. A shadow pattern is emitted onto a projection target from a video projector. Since it is impossible to project black light, the shadow pattern we employed consists of a darker region serving as a shadow against a brighter background. A digital camera is used to capture the image of the projection target and to conduct geometric calibrations. We create a shadow pattern based on the captured image of the projection target. Finally, the created shadow pattern is projected to the spatial vicinity of the projection target so that it seems as if the projection target is casting the shadow pattern.

Fig. 2. Photograph of Ukuzo hardware components.

3. Perceptual effects of Ukuzo

The main visual effect of Ukuzo is to give illusory depth impressions to 2D real objects. A photograph of the Ukuzo effect is shown in Fig. 3. A dynamic loop—the NTT logo—is printed on a piece of paper, and Ukuzo conveys a cast shadow pattern to the dynamic loop. The viewer gets the impression that the dynamic loop is floating up from its original position. Moreover, the distance between the dynamic loop and the cast shadow pattern can be dynamically changed. In this way, Ukuzo can give the illusory perception that the dynamic loop is dynamically changing its depth positions. As the previous study [2] indicated, it is possible to change the apparent depth position by manipulating the degree of blurring of a shadow pattern.

Fig. 3. A dynamic loop (NTT logo) with a cast shadow pattern projected by Ukuzo.

Ukuzo can give not only depth impressions but also transparency impressions to an opaque paper object. A photograph of printed materials in which two disk-shaped areas with different colors are overlapping in the middle is shown in Fig. 4(a). In Fig. 4(b), Ukuzo projects a shadow pattern to the greenish disk shape on the left. When we view the effect shown in Fig. 4(b), the left disk appears to consist of a transparent greenish material such as glass or plastic. Similarly, in Fig. 4(c), Ukuzo projects a shadow pattern to the reddish-colored disk on the right. This gives the effect that the right disk shape is made of a transparent material. The cast shadow pattern used in Fig. 4(c) is shown by itself in Fig. 4(d). Such patterns can be projected onto 2D printed material such as that shown in Fig. 4(a). When the shape of the projected cast shadow pattern is elaborated, Ukuzo can give the impression that opaque paper materials are made of transparent materials such as glass or plastic.

Fig. 4. (a) Photo of printed materials in which two disk shapes are overlapping. (b) The left disk has a cast shadow pattern projected by Ukuzo. (c) The right disk has a cast shadow pattern projected by Ukuzo. (d) The cast shadow pattern projected on the right disk shape.

Moreover, Ukuzo can give depth impressions to letters and illustrations that users draw. As shown in Fig. 2, the Ukuzo system has a camera to capture a projection target. The system captures the images of handwritten letters and illustrations, then generates the cast shadow pattern based on the captured images. When Ukuzo projects the cast shadow pattern onto the handwritten letters and illustrations, it gives illusory depth impressions to the handwritten materials (Fig. 5).

Fig. 5. (Left) Handwritten words. (Right) Cast shadow patterns are projected onto the handwritten words.

4. Future of Ukuzo

Ukuzo can be categorized as a kind of spatial augmented reality technique that is generally designed for changing the appearance of real objects by projecting digital images through a video projector. Spatial augmented reality techniques are usually intended to change the appearance of real objects in a physically correct manner. Specifically, if engineers want to change the appearance of a piece of paper from opaque to transparent, they need to calculate and project the desired light transport in order to display the appearance of a transparent sheet. However, in general, the light intensity of a typical projector is not sufficiently strong to achieve the physically correct light transport in ambient lighting.

In contrast, Ukuzo is not aimed at changing the appearance of real objects in a physically correct way. Rather, Ukuzo takes advantage of visual illusions wherein observers recognize a projected dark region as the shadow of an object and uses the shadow to estimate the 3D location of the object. Because it is possible for a standard projector to project the darker region in the vicinity of an object, it is feasible to use Ukuzo under normal ambient lighting.

Ukuzo can be used in various scenes. For example, Ukuzo can modify depth impressions of objects in paper posters that are displayed in stores and transport stations. Paper posters cannot usually be edited after printing. By making use of Ukuzo, designers can highlight the portions of posters on which they want customers to focus. Moreover, when a shadow pattern is projected toward an area of text on the poster that might be overlooked, it is possible to get customers to notice it. Furthermore, Ukuzo may be able to motivate children to draw pictures on the basis of interactions between hand-drawn pictures and illusory depth impressions that Ukuzo can give to pictures. That is, Ukuzo may help to support children’s artistic capabilities. Adults can also enjoy the effects of Ukuzo when cast shadow patterns are conveyed to their calligraphy work or woodcut prints. That is, Ukuzo effects can be employed as a technique in future visual arts.

References

[1]	F. Fiorani, “The Colors of Leonardo’s Shadows,” Leonardo, Vol. 41, No. 3, pp. 271–278, 2008.
[2]	D. Kersten, D. C. Knill, P. Mamassian, and I. Bülthoff, “Illusory Motion from Shadows,” Nature, Vol. 379, No. 6560, p. 31, 1996.
[3]	R. Raskar, R. Ziegler, and T. Willwacher, “Cartoon Dioramas in Motion,” Proc. of the 2nd International Symposium on Non-Photorealistic Animation and Rendering (NPAR 2002), pp. 284–300, Annecy, France, June 2002.