Automatic Degradation Estimation of Manhole Covers for Efficient Inspection via Vehicle-mounted Cameras

1. Problems in manhole cover inspection

Manholes are positioned to enable maintenance and management of the water supply, sewerage, network transmission lines, and telephone lines. Many manhole covers are exposed on roads all over the world. There are more than 14 million manholes in Japan alone, and a large number of them are driven on by vehicles every day [1]. Manhole covers are typically made of iron. The iron plate is slippery compared to the asphalt road surface, so many iron covers are engraved with a concavo-convex pattern for anti-slip properties. The manhole covers used by NTT are shown as an example in Fig. 1(a). A deep concavo-convex pattern is engraved on the surface, which helps to prevent slipping accidents.

However, the pattern on the cover gradually wears as automobiles pass over the cover, so manhole covers that have been in operation for a long time will potentially lose the effect of slip prevention due to wear, as shown in Fig. 1(b). In addition to the wear of the surface pattern, there may be a step difference between the cover and the frame supporting it due to aging deterioration, as in Fig. 1(c).

Fig. 1. Deterioration of manhole covers.

All companies that manage manholes, including the NTT Group, regularly inspect the degree of deterioration of manhole covers to prevent the occurrence of accidents caused by wear or step differences and then replace deteriorated covers as necessary. However, this requires a lot of manpower and time. Skilled workers must inspect the covers carefully, which sometimes requires part of the road to be blocked off in order to safely inspect them. This often results in traffic delays and inconvenience to road users.

At NTT, we have been developing a way to estimate the degree of deterioration of manhole covers using a vehicle-mounted camera. The objective is to construct a system to automatically inspect manhole covers simply by driving over them with the camera-equipped vehicle. We have combined image recognition technology and three-dimensional (3D) geometric transformation technology to achieve robust degradation degree estimation under various photographic conditions.

2. Manhole cover inspection system using vehicle-mounted cameras

The inspection system we propose is shown in Fig. 2. Cameras are installed in the front and rear of the vehicle used for the inspections, and these cameras continuously take pictures of the road while the vehicle is traveling. When a manhole cover is detected from the camera image, the deterioration degree estimation is applied, and the degree of wear of the target cover and the level difference from the supporting frame or road surface are obtained.

Fig. 2. Inspection system using vehicle-mounted cameras.

Various methods exist to estimate the shape of an object by remote sensing other than image processing, for example, 3D sensing. However, 3D sensing using a laser range finder is very costly as it involves a high unit price for the sensor and special equipment needed to capture 3D data.

In contrast, the proposed inspection system uses relatively inexpensive cameras and an ordinary car. Therefore, this inspection system can be simply and economically constructed using existing vehicles and deployed anywhere the manhole covers are in use all over Japan. In addition, since the deterioration degree can be estimated from just one image, and no special photographic technique is required, it is possible to inspect manhole covers using vehicles intended for non-inspection related work simply by equipping them with cameras.

3. Estimation of manhole cover degradation by image recognition

The process flow of the proposed method of manhole cover deterioration estimation is shown in Fig. 3 [2]. First, we extract the pattern wall surface area for the entire input image. Artificial intelligence is used to recognize the wall surfaces learned beforehand. Then we allocate three plane classes consisting of the top surface, bottom surface, and wall surface of the step to each pixel of the input image. The recognition process results in a map of the plane classes, as shown in Fig. 3(b). Next, the height of the step indicated by each wall from the obtained plane class map is estimated. With the focal length of the camera and the diameter of the manhole cover, it is possible to obtain the step height from the wall height in the image by applying 3D geometry. The height of the step is calculated for each of the recognized wall surfaces. The step height indicates the degree of wear of the anti-slip pattern on the cover and the difference in levels between the edge of the cover and the supporting frame or the road surface.

Fig. 3. Process flow of deterioration degree estimation of manhole covers.

The approach used in the conventional inspection method [3] involves directly learning the relationship between image features and the degree of degradation and then estimating the degree of degradation for each image region. In this approach, however, it is necessary to prepare learning data covering a large variety of images. The appearance of manhole covers changes depending on the illumination conditions, the level of contamination of the covers, and the angle of the camera. The conventional approach thus requires training data obtained in a variety of illumination conditions and shooting angles.

In contrast, with the proposed method, efficient learning is achieved with less data by effectively combining 3D geometry and machine learning, which is used to classify only the three plane classes previously mentioned. By estimating the image capturing angle for each image by 3D geometry and by estimating the step height according to the image, we can not only suppress any effects due to a difference in the image capturing angle but also estimate the step height outside the range of training data. It is not easy to collect a lot of degraded manhole covers for learning, so another advantage of our method is that it is possible to recognize the degradation state accurately even when there are few degraded samples contained in the training data.

4. Wall surface area extraction by deep learning

We utilize a convolutional neural network, the type of deep neural network used most often for image recognition, to extract the wall area. In particular, we use a network that recognizes each pixel of the input image [4] and classifies which kind of plane each pixel belongs to. The network outputs labels for each pixel based on the image features around the target pixel. Its parameters are trained by combinations of the input image and the map of the plane class label.

We used three classification labels: top, bottom, and wall, which make up the step shape, and constructed appropriate training data manually. The map of class labels used for training is shown in Fig. 4. The red parts correspond to the top area of the step, the green correspond to the bottom area, and the blue correspond to the wall area. With a sufficient quantity of this kind of data, the model can be trained to extract walls under various conditions. When a new image is classified, the area of the manhole cover is extracted from the photographed image, and the trained model outputs an appropriate wall area map from it.

Fig. 4. Examples of label maps for training.

5. Size estimation of wall surface

After the wall area map is obtained by extracting the wall area, the step height of manhole covers actually in use is estimated for each wall surface area in the wall map. We select a pixel at the top of the wall area—the starting point of height estimation—and find the pixels that continue in the wall area along the line perpendicular to the plane of the manhole cover from the starting point. The pixels indicate the line segment showing the height of the wall area. To calculate the actual length of this line, it is necessary to find the geometric relationship between the camera and the cover. Utilizing the fact that the focal length of the camera and the diameter of the cover are known, and the parameter of the ellipse formed by projecting the cover onto the image is easily calculated, we can obtain the angle of the camera with respect to the plane including the manhole cover and the height of the camera position. The parameters determined by 3D geometry are shown in Fig. 5. By estimating these parameters for each image, we can robustly estimate the height of the step even if the position of the camera and the installation condition of the cover are changed.

Fig. 5. Computable geometric relationship between the camera and cover.

6. Application to real images of manhole covers

We verified the accuracy of the deterioration degree estimation method using actual NTT manhole covers installed in roads.

The results of wall area extraction for manhole cover images in different illumination conditions are shown in Fig. 6. We can see that the wall area is robustly extracted even when the illumination condition differs or when part of the cover is shaded. The results estimated using the proposed method for deteriorated manhole covers are shown in Fig. 7. It is clear that the worn cover has a smaller extracted wall area, and the estimated depth of the grooves is shallower. Additionally, a large wall area is extracted from the cover where a gap appears relative to the supporting frame, and the estimated step height is also high.

Fig. 6. Examples of wall surface extraction results in various lighting conditions.

Fig. 7. Examples of deterioration degree estimation.

We compared the results of manually measuring the groove depth of the surface pattern of covers using calipers with estimated results using our proposed method for 170 cover images captured with a fixed camera. An average error of 0.55 mm compared to caliper measurement was achieved with our method.

Although it is inevitable that estimation errors will sometimes occur in degradation degree estimation by image recognition, we can still use this method for screening purposes in actual inspections. This involves extensively extracting covers that may have some deterioration from the many manhole covers to be inspected so as not to overlook any and then having skilled workers conduct a detailed visual inspection of them. With this method, an excessive number of deteriorated covers are extracted, but the possibility of missing any deteriorated covers is greatly reduced.

We conducted experiments to carry out screening using the proposed method for 1000 covers installed in roads. The experimental condition (deterioration criteria) was set to extensively extract worn covers so as not to miss any degraded ones. The results indicated that about 40% of the covers that were not actually deteriorated were judged to be deteriorated by the proposed method, while about 60% of the covers were accurately judged to have no deterioration and were thus removed from the inspection work target.

7. Future work

We have established a technology to robustly estimate the degree of manhole cover degradation by recognizing plane classes that make up the concavo-convex shape of manhole covers from images photographed under various conditions. In the future, we aim to implement this method at an actual inspection site, evaluate the accuracy of degradation recognition at the inspection site, and work on solving any problems in operation.

References

[1]	The Association for the Promotion of Next-generation High-quality Ground Manholes (in Japanese), http://www.kouhinigm.jp/gm/ul.html
[2]	K. Murasaki, S. Ando, and T. Kinebuchi, “Manhole Cover Wearing Estimation via Vertical Plane Segmentation,” IEICE Tech. Rep., Vol. 117, No. 210, pp. 205–212, 2017.
[3]	K. Murasaki, K. Sudo, and Y. Taniguchi, “Manhole Cover Wearing Detection by Photo-taking,” Trans. of the Society of Instrument and Control Engineers, Vol. 51, No. 12, pp. 814–821, 2015.
[4]	A. Bansal, B. Russell, and A. Gupta, “Marr Revisited: 2D-3D Alignment via Surface Normal Prediction,” Proc. of 29th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016), pp. 5965–5974, Las Vegas, USA, June 2016.