Unsupervised Depth and Bokeh Learning from Natural Images Using Aperture Rendering Generative Adversarial Networks

2.1 GAN

GANs [5] can mimic training data without defining their distribution explicitly. This property enables GANs to be applied to various tasks and applications in diverse fields.

As shown in Fig. 1, a GAN is composed of two neural networks: a generator and discriminator . These two networks are optimized through a two-player min-max game using an objective function L_GAN:

where, given a latent variable attempts to generate an image that can deceive a by minimizing L_GAN. By contrast, the attempts to distinguish a generated image from a real image by maximizing L_GAN. Superscripts r and g denote the real and generated data, respectively. Through this adversarial training, a generative distribution reaches close to a real distribution

Fig. 1. Architecture of GAN.

2.2 Light field aperture rendering

Light field aperture rendering [4] is a module that simulates an optical phenomenon (particularly bokeh) on a camera aperture in a differentiable manner. Note that such a differentiable property is necessary for deep neural networks (DNNs), such as a (discussed in Section 2.1), to update the parameters through the backpropagation commonly used for DNNs.

More concretely, as shown in Fig. 2, the rendering provides an aperture renderer that synthesizes a bokeh image from an all-in-focus image and depth map . Here, indicates the spatial coordinates of the light field on the image plane.

Fig. 2. Processing flow of light field aperture rendering.

I explain the details in a step-by-step manner. First, a depth map is expanded into a depth map for each view in the light field, i.e., , using a neural network T:

where indicates the angular coordinates of the light field on the aperture plane. Subsequently, an all-in-focus image is warped into an image for each view of the light field, i.e., , using the predicted :

From this formulation, the left-side images in the light field (5 × 5 images in Fig. 2) represent images when viewing objects from the left side, and the right-side images represent vice versa.

Finally, the is integrated using an aperture (an indicator that represents the disk-shaped camera aperture and takes ones for views within the aperture (indicated with white in Fig. 2) and zeroes otherwise (indicated with black in Fig. 2)) to render a bokeh image

When an object is on the focal plane, the object’s position is consistent regardless of the . Therefore, no bokeh occurs when the is integrated by the above equation. By contrast, when an object is separate from the focal plane, the object’s position varies depending on the . Thus, bokeh occurs in this case. Hereafter, and are omitted for simplicity except in necessary cases.

3.1 Problem statement

The problem statement is clarified before explaining the details of AR-GAN. As described in Section 1, AR-GAN is used to learn depth and bokeh effects only from images without additional supervision. In this setting, it is not easy to construct a conditional generator that directly predicts the depth or bokeh effects from an image due to the absence of pairs of 2D and 3D data or sets of multi-view images. Therefore, as an alternative, the aim is to learn an unconditional generator that can generate a tuple of an all-in-focus image , depth map , and bokeh image , from a latent variable .

AR-GAN uses focus cues as a clue for addressing this challenge. When the training images are highly biased in terms of bokeh effects (e.g., all training images are all-in-focus), it is difficult to gain focus cues from the images. Therefore, it is assumed with AR-GAN that the training dataset includes various bokeh images. Note that this assumption does not mean that the training dataset contains sets of different bokeh images for each instance. Under this assumption, AR-GAN learns the generator in a wisdom of crowds approach.

3.2 Model architecture

The processing flow of the AR-GAN generator is presented in Fig. 3. Given a latent variable , the AR-GAN generator generates an all-in-focus image and depth map using an all-in-focus image generator and a depth generator , respectively. Subsequently, the aperture renderer (explained in Section 2.2) synthesizes a bokeh image . Using this configuration, AR-GAN makes it possible to generate a tuple of an all-in-focus image , depth map , and bokeh image using a camera with an optical constraint on the light field.

Fig. 3. Processing flow of AR-GAN generator.

3.3 Training method

As shown in Fig. 1, a typical GAN applies a to the final output of the (i.e., in the case of the AR-GAN generator). However, in the AR-GAN generator, three modules, i.e., and are trainable. Therefore, they compete for roles if there is no constraint. For example, they can fall into an extreme solution (e.g., learns strong bokeh effects and learns over-blurred images).

To alleviate this problem, AR-GAN is trained using DoF mixture learning. Figure 4 illustrates the comparison between the standard GAN learning and DoF mixture learning. In the standard GAN learning shown in Fig. 4(a), the attempts to cover the overall real image distribution using generated images without any constraint. Consequently, it cannot determine to make a generated image close to a real all-in-focus image or a real bokeh image (indicated with question marks “?” in Fig. 4(a)).

Fig. 4. Comparison between standard GAN learning and DoF mixture learning.

By contrast, as shown in Fig. 4(b), in DoF mixture learning, the AR-GAN generator attempts to represent the real image distribution using generated images, the bokeh degrees of which are adjusted by a scale factor s. More concretely, the GAN objective function presented in Section 2.1 is rewritten as follows:

where s ∈ [0, 1]; when s = 0, an all-in-focus image is generated, whereas when s = 1, a bokeh image is rendered. Intuitively, the aperture renderer , which has an optical constraint on the light field, functions as a bokeh image prior. This prior encourages a generated all-in-focus image to approximate a real all-in-focus image (indicated by the “All-in-focus image” in Fig. 4(b)) and promotes a generated bokeh image to mimic a real bokeh image (indicated by the “Bokeh image” in Fig. 4(b)). Consequently, , which connects and , is also optimized. In this manner, the DoF mixture learning allows optimizing and together under an optical constraint.

A remaining challenge specific to unsupervised depth and bokeh learning is the difficulty in distinguishing whether blur occurs ahead of or behind the focal plane. For this challenge, on the basis of the observation that the focused image tends to be placed at the center of a photo, AR-GAN uses the center focus prior, which encourages the center to be focused while promoting the surroundings to be behind the focal plane. In practice, this prior is only used at the beginning of training to determine the learning direction.

4.1 Image and depth synthesis

The previous AR-GAN study [3] demonstrated the utility of AR-GAN using various natural image datasets, including flower (Oxford Flowers [6]), bird (CUB-200-2011 [7]), and face (FFHQ [8]) datasets. The implementation details are omitted because of space limitations. See that AR-GAN study [3] if interested in the implementation details.

Figure 5 shows examples of generated images and depth maps. AR-GAN succeeds in generating a tuple of an all-in-focus image (upper left), bokeh image (upper right), and depth map (lower left) in every setting. For example, in Fig. 5(a), the background is blurred while the foreground is unchanged in bokeh conversion (the conversion from the upper left to upper right). In depth prediction (the transformation from the upper left to lower left), the depth map (lower left) corresponding to the image (upper left) is successfully predicted. A light color indicates the foreground while a dark color indicates the background. Recall that the training data are only 2D images, and depth and bokeh effects are not provided as supervision. In this manner, learning depth and bokeh effects only from 2D images is the main strength of AR-GAN.

Fig. 5. Examples of generated images and depth maps.

4.2 Application to bokeh rendering and depth prediction

As discussed in Section 3.1, AR-GAN learns an unconditional generator that generates a tuple of an all-in-focus image , depth map , and bokeh image from a latent variable . Therefore, it cannot be directly used to convert a given image to the bokeh image or depth. However, AR-GAN can generate sets of all-in-focus and bokeh images or sets of all-in-focus images and depth maps artificially and abundantly by randomly changing the latent variable. By using these data, we can learn a bokeh renderer (i.e., a converter that converts an all-in-focus image to a bokeh image) and depth predictor (i.e., a predictor that predicts a depth map from an image) in a supervised manner.

Figure 6 shows example results obtained with the bokeh renderer and depth predictor mentioned above. A photo I took was used as an input (Fig. 6(a)). The bokeh renderer synthesizes a bokeh image (Fig. 6(b)), and the depth predictor predicts a depth map from the input image (Fig. 6(c)). Similar to the results in Fig. 5, the background is blurred while the foreground remains unchanged in the bokeh conversion (the conversion from (a) to (b)), and the depth map corresponding to the input image is predicted in the depth prediction (the transformation from (a) to (c)).

Fig. 6. Examples of bokeh rendering and depth prediction.

Note that the data required for training the bokeh renderer and depth predictor are only the data generated by AR-GAN, and no additional data are needed. That is to say, in this setting, we can learn a bokeh renderer and depth predictor in a fully unsupervised manner, similar to AR-GAN. This is a strength of an AR-GAN-based approach.