To view PDF files

You need Adobe Reader 7.0 or later in order to read PDF files on this site.
If Adobe Reader is not installed on your computer, click the button below and go to the download site.

Special Feature: Cutting-edge Technologies for Seeing and Showing

High-resolution Multiband Imaging for Accurate Color Reproduction

Masaru Tsuchida, Takahito Kawanishi, and Junji Yamato

Abstract

Multiband imaging technology for highly accurate color reproduction has been developed by NTT Communication Science Laboratories. In this article, we describe the principle of our image-capturing and color reproduction techniques using multiband images. In a recent project, our systems generated images containing over 100 megapixels (maximum: 2 gigapixels) for cultural-heritage digital archiving. We also present recent research results for a multiband moving-picture system.

PDF
NTT Communication Science Laboratories
Atsugi-shi, 243-0198 Japan

1. Introduction

In digital archiving for preserving our cultural heritage, in the medical field, and in some industrial fields, high-fidelity reproduction of color, gloss, texture, and shape are very important. Multiband or full-spectrum imaging technology is a way to achieve accurate color reproduction. Several types of multispectral image capturing system are available [1]–[3]. However, hardly any of them have been used widely in business activities or cultural-heritage digital archiving, mainly because they are expensive to construct and require special equipment. We have taken two approaches to solving these problems.

One of our solutions is to use a two-shot six-band camera system based on a cost-efficient commercial digital camera and a customized interference filter [4], [5]. To reproduce color accurately, we use a multispectral color reproduction technique. This technique can accurately reproduce an object’s color and its spectral reflectance, which is independent of the image-capturing device and illumination condition. The spectral reflectance of the object’s surface is estimated by using the data for every pixel. The estimated spectral reflectance enables us to accurately simulate the effects of converting the illumination from, for example, a daylight lamp to an incandescent lamp. Moreover, it allows us to identify the material and colorant. Our system has been used for digital archiving for cultural heritage preservation and for insect photography.

Our other solution is for moving pictures. Most multiband camera systems [1]–[5] are of the multishot type, and they cannot take images of moving objects. Ohsawa et al. [6] have developed a six-band high-definition television (HDTV) camera system. However, it requires very expensive customized optical equipment. For multiband technology to become widely used, equipment costs must be reduced and systems must be able to take images of moving objects. To meet these requirements, we developed a novel multiband image capturing system that combines multiband and stereo-imaging techniques [7]. This system can acquire both spectral color information and depth information at the same time. In this article, we focus on the generation of six-band images from a pair of stereo images.

2. Image-capturing system

2.1 Principle of color estimation using multispectral data

As shown in Fig. 1, an object’s surface reflects light from an illumination source. Let the illumination spectrum and spectral reflectance be E(λ) and f(λ), respectively. The observed spectrum I(λ) can then be represented as


Fig. 1. Observation model.

where λ is the wavelength. Let us consider the situation where the reflected light is captured by an N-band sensor. Let the sensor’s spectral sensitivity be S=[S1(λ), S2(λ), กฤ, SN(λ)]t and the matrix whose diagonal elements represent the spectral power distribution of the illumination be W. Equation (1) can then be rewritten in a vector representation as

By using the Wiener estimation method, we can estimate the spectral reflectance from the camera signal, c=SWf=Hf, as

where M is the Wiener estimation matrix obtained from H, and R is a priori knowledge about the spectral reflectance of objects [8].

2.2 Two-shot-type six-band image capturing system

The principle of two-shot-type six-band image capturing [9] is shown in Fig. 2. The system consists of a commercially available 35-mm digital camera and a customized interference filter. The filter is mounted in front of the camera lens and cuts off the left (i.e., short-wavelength) sides of both the blue and red peaks in the camera’s original spectral sensitivity. It also cuts off the green peak’s right (i.e., long-wavelength) side. First, an image is taken with the filter in front of the lens. Then, another image is taken without the filter by manually sliding the filter horizontally. Although the filter can be moved smoothly, there is a displacement error of a few pixels between the two images, which can be corrected by software [10]. After the correction process, the two three-band images are combined into a six-band image.


Fig. 2. Principle of two-shot-type six-band image capturing.

2.3 High-pixel-number image integrated by image stitching

In the conventional system [9], the number of pixels is determined by the image sensor. When an object 2 m × 2 m in size is captured with a 100-megapixel image sensor, the image resolution is 0.2 mm per pixel. To achieve image resolution of 0.01 mm per pixel with this sensor, the object should be divided into several parts, and an image of each part should be captured. The obtained images then need to be integrated into a high-pixel-number image.

There are two ways to integrate divided images. The general approach is to move the camera or object. Although this method is very simple and requires no special equipment, the effects of lens aberration, such as distortion and shading, have to be corrected before image integration. The second approach is to move the image sensor instead of the camera or object. This requires a good-quality lens with a large enough focused image area on the sensor plane. In addition, the camera needs a function for shifting the sensor horizontally and vertically. One advantage of this method is that the effects of lens aberration can be ignored in image integration. A practical way to implement this method is to combine a large-format camera (e.g., a 4 × 5 inch format camera) and a digital camera back. We used both methods in the experiments described below.

3. Experimental results

3.1 Image-capturing system

In estimating spectral reflectance by using multiband images, one should use full-spectrum light that does not have any emission lines. As the illumination source in the experiments, we used SOLAXTM (SERIC Ltd., Japan), which has a spectral power distribution close to that of natural sunlight. To avoid damaging the objects with ultraviolet (UV) and infrared (IR) light, we mounted UV- and IR-cut filters in front of the lamp. The spectral power distribution of this light is shown in Fig. 3.


Fig. 3. Spectral distribution of illumination.

The camera system used in our experiments is shown in Fig. 4. We used a 4 × 5 inch format camera (TOYO-VIEW 45GII) to which we attached a scanning digital back (DF-S2, Pioneer) as an image sensor. The sensor is a line charge-coupled device (CCD) sensor with 10,600 pixels. Scanned images have 10,600 × 17,460 pixels (185 megapixels). It takes about 15 minutes to make one image scan (depending on the brightness of the illumination light), so it takes almost 30 minutes to obtain a six-band image. Note that we replaced the IR-cut filter with a customized one that cuts wavelengths in the 800-nm range. The original IR-cut filter cuts wavelengths in the 680-nm range, which degraded the signal-to-noise (S/N) ratio of the near-IR channel of the six-band image capturing system used in the experiments. The spectral transmittance of the customized interference filter mounted in front of the lens and the spectral sensitivity of the six-band image capturing system are shown in Fig. 5. Regarding the color reproduction accuracy of this camera system, the average color difference between a real object and its image displayed on a liquid crystal display was ΔEab=1.7 when the target was the Macbeth ColorCheckerTM. The color gamuts of the six-band and three-band capturing systems (under sunlight) on the u’-v’ chromaticity diagram are shown in Fig. 6. The color gamut of Adobe RGB is also shown on the diagram as a reference. These results show that the six-band capturing system can record more deeply saturated color, especially red and blue, than an imaging system based on the Adobe RGB color space.


Fig. 4. Two-shot six-band camera system.


Fig. 5. Spectral transmittance of the filter and spectral sensitivity of the six-band camera.


Fig. 6. Comparion of color gamuts.

3.2 Captured images

To confirm the ability of the six- and three-band camera system to record saturated color, we captured a photograph of a butterfly using it. Figure 7 shows the resultant images after color reproduction using six- and three-band images captured by the camera system described in section 3.1. From these images, it is clear that the saturated blue color can be reproduced better when the six-band images are used. Figure 8 shows the definition of the butterfly image in Fig. 7 and that of a wood-block print image. The pixel numbers of the two photographs are 10,000 × 10,000 pixels for the butterfly and 28,000 × 14,000 pixels for the wood-block print. The image resolution is 0.01 mm per pixel.


Fig. 7. Resultant images from three-band image and six-band image.


Fig. 8. Definition of the resultant images.

An image that was captured outdoors under sunlight is shown in Fig. 9. The object is a ceramic wall 2.5 m high and 50 m wide. The pixel number of the synthesized image is 200,000 × 10,000 pixels (i.e., 2 gigapixels). The image resolution is 0.1 mm per pixel.


Fig. 9. Resultant image captured outdoors under sunlight using two-shot six-band camera system.

4. Stereo one-shot six-band camera system

Most current multispectral camera systems cannot take moving pictures or images of moving objects because they take each band image sequentially in time. A six-band HDTV system has been developed [6], but it requires very expensive customized optical equipment. Here, we introduce our novel approach to taking multiband moving pictures or still pictures of moving objects [7].

Our system consists of two commercial digital cameras and a customized interference filter. The filter, which is the same one as introduced in section 2.2, is mounted in front of the lens of one camera (Fig. 10). The camera with the filter captures a specialized three-band image; the other camera captures an ordinary RGB color image.


Fig. 10. Filter is mounted in front of the lens of one camera.

The shape of the image captured with the interference filter is adjusted to that of the other image by projective transformation. For the calculation of the transformation parameters, corresponding points between the two images are detected by the phase-only correlation method [10], which is a scale- and rotation-invariant pattern detection method that uses phase information. The resultant two three-band images are combined into a six-band image.

Some results that we obtained using this system are shown in Figs. 11 and 12. As a target object, we used old paint applied to cloth. Although the cloth looks flat, its surface actually undulates gently and is uneven. First, we took two images of the object using our system at the same time (see Fig. 11). These are raw data images from the image sensor, which is why the color balance of the image captured without the interference filter looks incorrect. Second, the image transformation process was carried out and a six-band image was generated. Next, the six-band image was converted into a spectral reflectance image through Wiener estimation [8]. Finally, the spectral reflectance image was converted into an RGB image by using the measured illumination spectrum and monitor characteristics (Fig. 12). No registration errors remain in the band image generated by our method. The resultant RGB image was compared with the paint and an image generated by the two-shot six-band camera system [9]. The image obtained with our method is the same color as the paint and has the same quality as the image obtained by conventional methods.


Fig. 11. Result of image transformation.


Fig. 12. Color reconstructed image using stereo one-shot six-band camera system.

In our system, a stereo image is used not only for six-band image generation, but also for three-dimensional shape reconstruction. The reconstructed shape of the target object used for color reproduction is shown in Fig. 13.


Fig. 13. Reconstructed shape of the target object used for color reproduction.

5. Conclusion

This article introduced a high-fidelity color reproduction method based on multiband imaging and two multiband image capturing methods. Despite the relatively low construction cost for our image capturing systems, its target object reconstructed colors are almost as accurate as those of the conventional multiband image capturing system. Our systems have recently been used in several cultural-heritage digital archiving projects. In the future, we expect that they will be used in various fields, such as education, medicine, and industrial design.

To enable high-pixel-number images to be viewed smoothly on websites or common computer displays, we have also developed image conversion and viewing software using DeepzoomTM technology provided by the Microsoft SilverlightTM application programming interface. The image converter can handle 20-gigapixel images. Generated images can be viewed with ordinary web browser software. The smooth zooming and viewpoint-changing lets viewers enjoy high-definition interactive contents. Some example images can be seen on our website [11].

References

[1] M. Yamaguchi, Y. Murakami, T. Uchiyama, K. Ohsawa, and N. Ohyama, “Natural Vision: Visual Telecommunication Based on Multispectral Technology,” Proc. of IDW’00, Vol. 7, pp. 1115–1118, Kobe, Japan, 2000.
[2] S. Tominaga and R. Okajima, “Object Recognition by Multi-spectral Imaging with a Liquid Crystal Filter,” Proc. of the 15th Int. Conf. on Pattern Recognition, Vol. 1, pp. 708–711, Barcelona, Spain, 2000.
[3] S. Helling, E. Seidel, and W. Biehig, “Algorithms for Spectral Color Stimulus Reconstruction with a Seven-channel Multispectral Camera,” Proc. of CGIV (2nd European Conference on Color in Graphics, Imaging and Vision), Vol. 2, pp. 254–258, Aachen, Germany, 2004.
[4] M. Hashimoto, “Two-shot Type 6-band Still Image Capturing System Using Commercial Digital Camera and Custom Color Filter,” Proc. of CGIV (4th European Conference on Color in Graphics, Imaging, and MCS/08 Vision 10th International Symposium on Multispectral Color Science), Vol. 4, pp. 538–541, Barcelona, Spain, 2008.
[5] M. Tsuchida, H. Tanaka, and K. Yano, “Development of a High-definition and Multispectral Image Capturing System for Digital Archiving of Early Modern Tapestries of the Kyoto Gion Festival,” Proc. of ICPR (20th International Conference on Pattern Recognition), Istanbul, Turkey, 2010.
[6] K. Ohsawa, T. Ajito, Y. Komiya, H. Haneishi, M. Yamaguchi, and N. Ohyama, “Six-band HDTV Camera System for Spectrum-based Color Reproduction,” J. Imag. Sci. and Tech., Vol. 48, No. 2, pp. 85–92, 2004.
[7] M. Tsuchida, T. Takahashi, K. Ito, T. Kawanishi, J. Yamato, and T. Aoki, “A Stereo One-shot Multi-band Camera System for Accurate Color Reproduction,” Proc. of ACM SIGGRAPH 2010 Posters, Los Angeles, CA, USA.
[8] W. K. Pratt and C. E. Mancill, “Spectral Estimation Techniques for the Spectral Calibration of a Color Image Scanner,” Appl. Opt., Vol. 15, pp. 73–75, 1976.
[9] M. Hashimoto and J. Kishimoto, “Two-shot Type 6-band Still Image Capturing System Using Commercial Digital Camera and Custom Color Filter,” Proc. of CGIV, (4th European Conference on Color in Graphics, Imaging, and MCS/08 Vision 10th International Symposium on Multispectral Color Science), Vol. 4, pp. 538–541, Barcelona, Spain, 2008.
[10] K. Takita, T. Aoki, Y. Sasaki, T. Higuchi, and K. Kobayashi, “High-accuracy Image Registration Based on Phase-only Correlation,” IEICE Trans. Fundamentals, Vol. E86-A, No.8, pp. 1925–1934, 2003.
[11] Example images. http://www.brl.ntt.co.jp/people/tsuchida/index.html
Masaru Tsuchida
Research Scientist, Recognition Research Group, Media Information Laboratory, NTT Communication Science Laboratories.
He received the B.E., M.E., and Ph.D. degrees from Tokyo Institute of Technology in 1997, 1999, and 2002, respectively. In 2002, he joined NTT Laboratories, where his research areas included color science, three-dimensional image processing, and computer vision. His specialty is color measurement and multiband image processing. From 2003 to 2006, he worked at the National Institute of Information and Communication Technology (NICT) as a researcher in the “Natural Vision” project. From 2006 to 2008, he worked at NTT DATA as a project manager and senior researcher in a contract research project initiated by NICT. He also worked on the commercialization of the research achievements. Since 2009, he has been a guest researcher at Ritsumeikan University, Kyoto. He is a member of the Color Science Association of Japan, Optical Society of Japan, and Institute of Electronics, Information and Communication Engineers (IEICE) of Japan.
Takahito Kawanishi
Senior Research Scientist, Recognition Research Group, Media Information Laboratory, NTT Communication Science Laboratories.
He received the B.E. degree in information science from Kyoto University and the M.E. and Ph.D. degrees in information science from Nara Institute of Science and Technology in 1996, 1998, and 2006, respectively. In 1998, he joined NTT Laboratories, where his main research areas were computer vision and media recognition. His specialty is fast media search techniques. From 2004 to 2008, he worked at Plala Networks Inc. (now NTT Plala) as a technical manager and designed set-top boxes and operational systems for IPTV and video-on-demand services. He is currently engaged in R&D of online media content identification, focusing on monitoring and search systems to expand digital media content distribution on the network. He received the FIT2003 Young Researcher Award from the Information Processing Society of Japan (IPSJ), the Plala President’s Award in 2006, and the NTT President’s Award in 2008. He is a member of IPSJ and IEICE.
Junji Yamato
Executive Manager, Media Information Laboratory, NTT Communication Science Laboratories.
He received the B.E., M.E., and Ph.D. degrees from the University of Tokyo in 1988, 1990, and 2000, respectively, and the S.M. degree in electrical engineering and computer science from Massachusetts Institute of Technology in 1998. His areas of expertise are computer vision, pattern recognition, human-robot interaction, and multiparty conversation analysis. He is a visiting professor of Hokkaido University and Tokyo DENKI University and a lecturer of Waseda University. He is a member of IEEE, IEICE, and the Association for Computing Machinery.

↑ TOP