Feature Articles: Platform Technologies for Services that Utilize Lifelogs
Lifelog Remote Control for Collecting Operation Logs Needed for Lifelog-based Services
In this article, we describe our concept of using personal remote controls to collect data for subsequent utilization and the remote control’s user interface design, focusing on the technology that makes it possible, for the first time, to collect truly useful lifelogs.
To provide lifelog-based services, we need to accurately estimate the user’s preferences and behavioral patterns in daily life from various activities. The key to accurate estimations is gathering sufficient data to identify each user’s likes and preferences. In recent years, various services that provide recommendations have been developed. They process data about the user’s Internet access activities such as online shopping and visited websites. Other services process information about the user’s location by defining places through a GPS (global positioning system) function. To use those services, however, users still need to register their likes and preferences in advance, but that is so inconvenient for users that their preferences and behavioral patterns can never be fully captured.
We believe that continuously receiving and storing users’ lifelogs over long periods of time will yield a structure that can collect data more easily without imposing a heavy load on the user.
Our approach is centered on utilizing a personal remote control that each user carries and uses to operate all home devices. The operation data is saved within it and sent to service providers only when the user permits. In this way, each user’s personal information can be protected.
In developing our lifelog collecting approach, we are focusing on two major points: (1) a remote control that allows users to control various devices and saves operation logs within itself and (2) an attractive interface that encourages users to use the remote control all the time. In this article, we describe the technologies underlying the lifelog collecting system and its user interface.
2. Information acquired from operation logs
The operation of home devices is assumed to reflect the user’s free will choices. That is, those actions reflect the user’s likes or preferences. For example, if the user turns on a television (TV) and selects a station that offers music programs, we can assume that the user is interested in music, whereas if it is a movie channel, we can assume that the user likes movies. If the user sets the air conditioner to 17¡ëC everyday, we can assume that the user tends to feel comfortable at that temperature. When the user turns room lights on and off, we can determine what time the user comes home and goes to sleep. Such operation data is, we believe, a strong indicator of the user’s likes, preferences, and behavioral patterns.
Below, we describe our ongoing work.
3. Lifelog collecting system
NTT Cyber Solutions Laboratories has developed a lifelog collecting system that allows users to operate home devices from a single remote control. We targeted devices with infrared sensors because most modern home devices are equipped with such sensors.
The structure of the lifelog collecting system is shown in Fig. 1. As the remote control, we chose to use a smartphone because its operation screen is easy to customize.
Command relay devices, wirelessly driven by the smartphone, are set throughout the house so that the intended device can receive its control signal.
We use wireless local area network (WLAN) connections to link the smartphone to the command relay devices. To stabilize each WLAN connection and ensure that the control signals reach the intended device, we mounted the command relay devices on ceiling lights (Fig. 2). This arrangement allows infrared rays to be sent in eight different directions at the same time, which provides coverage over a wide area.
Remote control applications are written for each appliance and loaded into the smartphone (Fig. 3). Since each appliance uses a different remote control, the program creates a virtual remote control on the smartphone. The user selects the appliance and then presses the virtual keys to input the command desired. The corresponding signal is sent to the pre-set command relay device. The command relay device converts the signal into the appropriate control signal and then transmits the infrared signal. The command relay device nearest the device that the user wants to operate, such as the TV, should be selected as this makes signal reception more reliable. The smartphone automatically makes a list of available command relay devices, and the user can select the desired one from the list.
System implementation proceeds as follows.
(1) Connect all command relay devices to the home’s WLAN.
(2) Connect the smartphone to the WLAN.
(3) Select each device (manufacturer and product) to be operated with the remote control and install the corresponding program in the smartphone.
(4) For each device, select the nearest command relay device and register it in the smartphone program.
(5) Repeat steps 3 and 4 for each device to be operated.
(6) Activate the application for the device and operate the device via the virtual remote control on the smartphone. To operate another device, activate its application.
The user’s remote control application operation data is first saved as logs on the smartphone. Those logs are sent to providers that offer lifelog-based services if the user allows the data to be sent.
4. Device status estimation from operation logs
The operation logs of each user can be collected individually by the lifelog collection system. Although the operation logs contain information about when and which button was pressed, it is difficult to know the status of each device accurately. For example, from an operation log entry that says “the user pressed the TV power button” on the smartphone, you cannot tell if the user turned the TV on or off. If the station 1 button was pressed, we can assume that station 1 was selected, but if the user used the channel up-down button to choose the station, you need to know which station was on before to figure out which station the user chose. Furthermore, new terrestrial digital broadcasting services offer more complicated programs in Japan. Two or three different programs are offered under a single channel button (Fig. 4). In such cases, it is very difficult to determine which channel the user selected from a single operation log.
We have been developing a method of estimating the device status as accurately as possible. It tracks TV status transition and checks the status transition against the operation log containing pressed buttons and the time lag between button presses. For example, if there is no record of any operations immediately after the power button was pressed, we can assume that the user turned the TV off. If the history shows some operations, such as adjusting the volume or changing channels, we can assume that the user turned the TV on. Moreover, from operation data for lights and air conditioners, we can estimate the user’s lifestyle and behavioral patterns by knowing when the user turned lights on or off and what temperature was usually selected.
To obtain users’ preferences and behavioral patterns, we have to estimate the status of each device from information acquired through operation logs. One option would be to determine the status from the device, but if we can determine each device’s status through the remote control log, the usage range will be wider.
5. Attractive user interface that invigorates lifelog-based services
Since a personal remote control must be used for long periods of time to gather sufficient data for lifelog-based services, it must be easy to use.
5.1 “Coool” remote control
For prolonged use, the remote control should be easy to hold and fit comfortably in the user’s hand. Buttons must also be easy to press. We developed the “Coool” remote control to satisfy these requirements (Fig. 5). It is round and each button is placed for easy access by the user’s thumb. The rocker switch in the center can be pressed up or down and left or right. This allows the user to change display contents dynamically and access various functions with fewer button pushes. This remote control is equipped with a GPS function, temperature and acceleration sensors, a WLAN, and infrared and Bluetooth transmitters. It has a stylish design that will attract users while having all the hardware required to provide lifelog-based services. We will continue our research and development activities on remote applications for smartphones and other devices to create various lifelog-based services using the Coool remote control.
5.2 Contents menu for lifelog-based services
Users can display the contents menu on a TV screen. That makes it easier for users to select an appliance and issue detailed commands. As an example, My Menu sent by the Coool remote control is shown in Fig. 6. My Menu lists commonly accessed appliances and programs that the user likes and watched recently as well as contents recommended by lifelog-based services.
5.3 Simultaneous display of multiple My Menus
Each user has a different personal remote control, but multiple users can place their My Menus on the same screen at the same time (Fig. 7). Since My Menu may include information that a user does not want to share with others, the Coool remote control automatically alters the My Menu contents when another user is physically nearby.
6. Future plans
To create convenient and fun lifelog-based services, we will carry out feasibility studies of lifelog collection and device status estimation in actual home environments as well as validity tests of lifelog-based services as soon as possible.