NTT Develops Non-rigid Object Recognition Technology that Achieves Recognition of Database Images―Not Only Rigid 3D Objects but also Non-rigid Objects―at Low Cost

1. Introduction

NTT has developed a new technology called angle-free rigid and non-rigid object information retrieval, which recognizes and searches deformable objects with a small number of database images (Fig. 1). This technology makes it possible to recognize rigid three-dimensional (3D) objects as well as non-rigid objects such as soft packaging or cloth products with high accuracy.

Fig. 1. Features of angle-free rigid and non-rigid object information retrieval.

Conventionally, image recognition technologies including deep learning based ones require the preparation of many database images capturing multiple deformation patterns in order to recognize deformed objects. If an unknown deformation is added to the object, many more database images are needed to recognize it. Our new rigid and non-rigid object image matching technology recognizes an object with an unknown deformation as the same object as the non-deformed object image in the database.

With this technology, the cost of preparing database images is greatly reduced, which makes it possible to develop a product management system for a retail industry in which new products are released frequently. This is also applicable to product recognition, which saves labor in check-out operations, and product information presentation, which enables customers to browse product descriptions in multiple languages.

2. Background

To realize a digital transformation that improves our lives by connecting people with information and communication technology, it is essential to establish recognition and search technologies that can perform these functions using images taken by smartphones and cameras in place of the human eye.

However, non-rigid objects such as soft packaging products and cloth products have different patterns of deformation than rigid 3D objects. Therefore, the appearance of the images changes greatly, resulting in lower recognition accuracy. To maintain recognition accuracy, it is necessary to register the image corresponding to the deformation of the target object as a reference image. However, if the pattern of deformation of the target object is arbitrary, it is necessary to prepare a large number of images, which substantially increases the cost.

3. Research and development achievements

In the angle-free rigid and non-rigid object information retrieval technology developed by NTT, a geometric constraint is applied for each of multiple partial areas rather than for the entire object. The rigid and non-rigid object image matching accurately specifies the correct correspondence from the correspondence relationship of the image features. Thus, the correct correspondence between the input image and the reference image can be specified even if the object is deformed. As a result, it is possible to recognize and search with high accuracy not only rigid 3D objects, but also non-rigid objects in which arbitrary deformations occur.

4. Technical features

Angle-free rigid and non-rigid object information retrieval is an NTT corevo^® artificial intelligence technology that was developed based on angle-free object information retrieval, a technique for recognizing and searching 3D objects only by registering images from a few directions. The features of this technology are as follows.

In the conventional angle-free object information retrieval, the recognition accuracy deteriorates for deformed objects because the constraint condition on the same object is derived from the projective geometric examination based on the assumption that the object has a fixed shape. Furthermore, with existing learning based methods such as those based on deep learning, if a non-rigid object is used as a target, the recognition accuracy also deteriorates, since there are countless deformation patterns, unless a large number of reference images observing various deformations are prepared in advance.

Therefore, rigid and non-rigid object image matching has been developed, which accurately specifies the correct correspondence from the correspondence of the image features by applying the geometric constraint to multiple partial regions instead of to the entire object. This makes it possible to specify the correct correspondence even if the object is deformed, by clustering using geometric features such as distances and rotation angles between multiple correspondence relationships based on the correspondence result of image features between the input image and the reference image. NTT’s angle-free rigid and non-rigid object information retrieval expands this technology, making it possible to achieve recognition and retrieval with high accuracy by only registering a small number of images even if the pattern of deformation is arbitrary.

For Inquiries

Public Relations, NTT Service Innovation Laboratory Group

http://www.ntt.co.jp/news2018/1811e/181126b.html