Feature Articles: Digital Transformation of Urban Areas Focusing on Mobility
Technical Developments and Verification of Connected Cars
Toyota Motor Corporation and NTT have agreed to collaborate on the technical development and verification of connected cars by combining Toyota’s automobile-related technology and NTT’s information and communication technology (ICT)-related technology. They have begun a demonstration experiment to check the current degree of completion of the required technical development. This article presents an overview of the ICT platform for connected cars and describes technical issues, technologies used to address these issues, and the demonstration experiment.
Keywords: connected cars, IoT, big data
1. Development of the ICT platform for connected cars
After the collaboration agreement was concluded , Toyota Motor Corporation and the NTT Group have been developing an information and communication technology (ICT) platform for connected cars by taking future automated driving into consideration. To achieve safe and secure automated driving, various data sent from vehicles must be collected and used in a cloud. The ICT platform is being developed as a means to achieve this.
2. ICT platform for connected cars
A connected car is a car that also functions as an ICT terminal. Since a vehicle holds many items of information, it is possible to understand the vehicle and its surroundings from the data it sends. If we are to solve social problems, such as traffic accidents and traffic jams, and upgrade mobility services, it is essential to use data collected from connected cars via a network.
Many key technologies are required to support connected cars: technology to collect and store a large volume of vehicle data sent by numerous vehicles; technologies to analyze the collected large volume of data in real time and reproduce the surroundings on a cloud; artificial intelligence (AI) algorithms to determine the behavior of vehicles; technology to collect and distribute a large volume of data; 5G (fifth-generation mobile communication system); edge computing; and communication technology to collect and distribute a large volume of data reliably and securely using Internet of Things (IoT) networks.
Specifically, a large volume of data from electronic control units (ECUs) installed in vehicles on roads and sensor data, including images from cameras, need be collected and stored in a cloud reliably and efficiently. From the collected data, the status of vehicles and their surroundings need be reproduced in cyberspace in real time. These processes make it possible to generate maps to be used in next-generation traffic systems and allow various services and automated driving. The ICT platform for connected cars will support the reproduction of the status of vehicles and their surroundings in cyberspace (Fig. 1).
3. Demonstration experiment
The development of the ICT platform for connected cars is being undertaken through short cycles of studying individual key technologies and a demonstration experiment in which these technologies are combined and verified on an end-to-end connection, including terminals, a network, and datacenter. In the demonstration experiment, data from multiple experiment vehicles running on roads are collected via a network and stored in a cloud. The collected data are analyzed to confirm whether technical issues identified for use cases of connected cars and automated driving have been solved. In addition, vehicle data equivalent to millions of vehicles are generated to evaluate the feasibility of communication and datacenter technologies.
Specific uses cases being studied are: (1) generation of a high-precision map, (2) detection and notification of obstacles, and (3) traffic-jam detection in each lane. The extent to which the technical requirements of each use case are satisfied is being examined.
(1) Generation of a high-precision map
Whether video data captured with vehicle-mounted cameras can be sent to the cloud and whether information about vehicle positions and about the surroundings can be reproduced in cyberspace are being examined (Fig. 2). In particular, we are developing a technology for collecting video data and location information from vehicles and estimating the positions of road signs, road marks, and other structures from the collected video data captured with vehicle-mounted monocular cameras with a high degree of precision. Currently, changes in road surfaces are examined by driving a dedicated measurement vehicle over the road to be examined. This method is costly and cannot produce examination data in real time. Instead, our technology can be used to produce a high-precision map from image data captured with an ordinary vehicle-mounted camera.
(2) Detection and notification of obstacles
We are examining whether data sent by a vehicle can be shared with other vehicles in real time (Fig. 3). Specifically, when a vehicle-mounted monocular camera captures image data of an obstacle, the platform receives its data from the vehicle and estimates the position of the obstacle. The information about the obstacle is registered on a dynamic map. The platform quickly searches for vehicles that can be affected by the obstacle. Finally, the information about the obstacle is transmitted to these vehicles. We are developing a technology that accelerates the processing of these steps in less than 10 seconds.
(3) Traffic-jam detection in each lane
We are verifying whether the level of congestion in neighboring lanes can be reproduced from video data captured with a vehicle-mounted monocular camera. In particular, we are developing a technology for identifying the lane in which the vehicle runs and accurately estimating the level of congestion in neighboring lanes from the video data.
Technical targets have been defined for the key technologies being used in the demonstration experiment by assuming several use cases. The targets are the ability to connect tens of millions of vehicles to the system and perform the above steps within just a few seconds. This demonstration experiment began in 2018. It has been gradually expanded and will continue until the end of fiscal year 2020. We have already confirmed that 5 million vehicles can be connected to the system and that all the required steps can be performed in less than 20 seconds.
4. Main technologies that support the ICT platform
The ICT platform for connected cars involves a number of technical challenges. The main technologies being developed to meet these challenges are described below (Fig. 4).
(1) Object-position-estimation technology
This technology estimates the positions of display objects (road signs, signboards, etc.) in the road surroundings from vehicle status data and video data captured with vehicle-mounted cameras. It can even process data from low-end dashcams. It improves positional accuracy by selecting high-quality data from a large volume of data and using statistical methods. Accurate recognition of three-dimensional objects, such as road signs and traffic lights, on roads and high-precision estimation of the positions of these objects make it possible to map the objects on a map. This technology is expected to be used for generating maps for automated driving and automatically updating them.
(2) Algorithm for selective collection of vehicle data
Data collected from connected cars are both voluminous and varied, ranging from vehicle data to video and sensor data. Since these data are collected consecutively from a large number of vehicles, the volume of collected data is enormous. This imposes a huge load on the network, and data processing capacity is stretched to the limit. To solve these problems, we developed a technology for selecting data collected from vehicles based on meta-information, such as location information, time information, sensor type, resolution, observational range, and observation impairment caused by the presence of surrounding vehicles. This technology makes it possible to efficiently collect data with minimum duplication of observational ranges and adjust the volume of collected data in accordance with the load on communication links and the analysis platform.
(3) Real-time spatio-temporal data-management technology
This technology enables a large volume of data to be stored instantaneously and searches a specified range (time and space) of data at high speed. It is a spatio-temporal database that manages time and location information generated by moving objects, such as vehicles and smartphones, as one-dimensional data using a unique method and enables the searching for those moving objects within a specified time and rectangular space range. It also searches within a space with a complex shape, such as a road or parking lot. Therefore, it enables the storing of sensor information generated by a large number of moving objects in real space, and time and space information associated with each item of sensor information, as well as searching for this information within a specified time and space range.
(4) Vertically distributed computing technology
This technology processes applications at high speed by offloading computation tasks done at datacenter servers to edge servers. Since the processing capacity of the system server is limited, it is sometimes not possible for the system to respond to all devices instantly. In reality, the required response time varies depending on the status of each vehicle (vehicle speed, moving direction, time of day, etc.). For this reason, vertically distributed computing technology dynamically distributes applications between edge servers and datacenter servers based on the status of vehicles. This technology can use limited server resources effectively while ensuring that the system responds instantly to those vehicles that require a quick response. This technology enables the system to serve a large number of vehicles and to collect and respond to a large volume of data. Thus, limited distributed computing resources can be used efficiently.
(5) Traffic-jam detection in each lane
This technology detects traffic jams in each lane from vehicle-status data and video data captured with vehicle-mounted cameras. It detects a queue of vehicles from the relative locations of vehicles moving along the surrounding lanes. It can also recognize the cause of a queue by identifying the first and last vehicles and combining this information with the surrounding facility information. Currently, information about traffic jams by road is provided by the Vehicle Information and Communication System (VICS) and smartphone map apps. When the above technology has been developed and in wide use, more valuable traffic-jam information will become available. As with the object-position-estimation technology described above, this technology can detect a traffic jam using video data captured with even low-end dashcams.
It is said that the automobile industry is facing a once-in-a-hundred-year transformation. The environment surrounding automobiles has been changing dramatically over the last few years. One of these changes is the spread of connected cars, resulting in the emergence of various services for drivers, fellow travelers, automakers, insurance companies, and service garages.
When it will become possible to process a large volume of data on vehicles quickly and at low cost, information that cannot be captured using current sensors will become available, and faster and more reliable services than today will be provided. These developments will not only enhance convenience and efficiency but also provide safety and security to the coming age of automated driving.
Young researcher’s view
Research and Development of Digital Twin Computing Technology to Support Future Transport Systems
My research team is developing technology for building a digital twin city using data collected from connected cars. Collection of LiDAR (light detection and ranging) and video data from connected cars allows continuous scanning of an entire city. A digital twin of a city can be built in virtual space by analyzing these data. Such a digital twin can be used for many purposes. In transport, it can be used to create a road-surface image map for autonomous driving cars and enable vehicles to share information about obstacles on the road. It can also be used in other applications such as detecting queues at restaurants located along a road, checking the damage after a disaster, and detecting anomalies in utility poles and cables.
The challenge is the sheer amount of sensing data. A single car equipped with multiple cameras and LiDAR sensors generates 10 MB of data every second. If 10 million cars are running throughout the country, the total amount of data generated per second reaches 100 TB. This is a huge amount, 10 times the amount of data carried by communications networks in Japan. It is extremely difficult to collect all such data due to the capacity limits of communication networks, computing resources, and storage systems. This situation can be mitigated because the urgency of collecting sensing data varies. Some need to be collected immediately for analysis, while other data can be collected after a delay. Some need to be collected only once a day, and some have no utility value whatsoever.
Therefore, we developed a technology that prioritizes data collection based on meta-information, such as location information, time information, sensor type, resolution, observational range, and observation impairment caused by the presence of surrounding vehicles. Although the amount of meta-information is huge, the high-speed spatiotemporal data management technology AxispotTM makes it possible to search meta-information at a practical processing speed. Our technology makes it possible to collect data efficiently with minimum duplication of observational ranges and adjust the volume of data to collect in accordance with the load on communication links and analysis platform.
In a demonstration experiment conducted in FY2019, this technology was implemented on an analysis platform, and vehicles running on public roads were connected to the platform. The function to selectively collect data based on the priority of the data was verified. In a demonstration experiment currently being conducted (FY2020), we are examining a function to dynamically switch the servers executing analysis on the basis of data priorities and processing load.
We are also making an ambitious effort to develop technology capable of predicting impending traffic status, to within just a few minutes to several hours from the present, with a high degree of precision. This technology is based on a multi-agent, microscopic traffic-flow simulator, which can reproduce the movement of each vehicle. We set the initial conditions by analyzing collected sensing data. We are aiming to improve reproduction precision by adjusting important parameters such as the stop-go indications of traffic signals. In the event of an unexpected incident, such as a traffic accident, this technology enables rapid assessment of the accident°«s impact on traffic flow, thereby helping to prevent secondary damage or traffic jams.
I believe that, as connected cars become widespread, there will be an increasing demand for the ability to use spatiotemporal data intuitively in real time. By fully exploiting virtual reality technology, I will develop an intuitive user interface that displays digital twins in an elegant and well-organized manner so that anyone will be able to easily use spatiotemporal data. I will promote data usage in a wide range of applications through this effort and will do my utmost to develop technologies that will close the digital divide.
Masaru Takagi received a B.E. in engineering and an M.E. and Ph.D. in information science and technology from the University of Tokyo in 2013, 2015, and 2018. He joined NTT Network Innovation Laboratories in 2018 and is engaged in a collaboration project with Toyota Motor Corporation. He developed the technology to efficiently collect the sensor data from connected cars using the edge computing platform (ICT platform). He is currently studying multi-agent mobility simulation as part of Digital Twin Computing technology. He received the Best Paper Nominee Award from the Association for Computing Machinery UbiComp 2014. He is a member of the Institute of Electronics, Information and Communication Engineers and Information Processing Society of Japan. He is interested in mobile terminals and owns more than 100 Android devices.