Trends in Lifelog Applications and Progress at NTT R&D

1. Introduction

Amazing advances have been made in recent years in the broadband environment, consumer electronic devices, and mobile phones. For example, users can now view the huge amount of video content on the Internet on their own televisions (TVs) using broadband connections, and weblog (blog) sites are exploding in number as niche information overflows in an unforeseen way. Microblogging, which involves small content, is a service that has appeared recently on social networks such as Twitter. It enables public figures and celebrities, amongst others, to share their musings and activities in nearly real time. Moreover, smartphones let users access the huge amount of information on the Internet in the same way that personal computers do but from wherever they please at anytime. New electronic devices are also appearing in society. Digital photo frames enable digital photographs to be shared over the network and e-book readers with communication functions let users purchase and read books anytime and anywhere. The growth of such services over the last few years has been nothing short of remarkable.

At the same time, there is still some doubt as to whether users will be able to fully enjoy these novel services. Searching for and retrieving desired content and meaningful information from the massive Internet is troublesome. It is also difficult to learn how to use a multifunction electronic device while referring to a bulky manual. Even though an electronic device may be equipped with a variety of convenient functions, the user may not be able to make full use of them unless helpful information is readily at hand. We believe that the means of solving or easing this problem can be found in the use of a lifelog. A lifelog is a digital record of an individual’s activities accumulated over the long term. It may include a history of web pages visited and keywords input to search engines, as well as content created by the individual such as documents prepared using a word processor and photographs taken with a digital camera, electronic records of the individual’s weight and calorie intake, and the history of sent/received emails and incoming/outgoing telephone calls. It has also become possible in recent years to obtain GPS-based location information and use smart cards to record the train stations used for commuting, the products purchased at convenience stores, etc. (GPS: global positioning system). These capabilities suggest the possibility of constructing lifelogs that are closely associated with daily activities. How then, can such an extensive and detailed lifelog be useful?

2. Usefulness of lifelogs

User characteristics have traditionally been obtained in the form of demographic attributes such as age, sex, and home region (Fig. 1), but a lifelog can provide more detailed knowledge about the user. For example, an individual’s web browsing history and e-commerce history can reveal whether that person has an eye for new electronic consumer products, likes Japanese movies more than Western movies, or spends money on clothes. Preferences such as these can be used to make preliminary selections from the huge amount of content and information on the Internet so that the user can easily and enjoyably obtain what he or she needs. Device usage history could be used in a similar manner. If the way in which the user uses a device is known and if functions that the user tends to use can be selected beforehand, it should be much easier for the user to make full use of a multifunction electronic device. In a similar manner, referencing the tendencies of other users who have similar likes and dislikes and making product and service recommendations accordingly should make it possible to provide information that the user might otherwise never notice. Such lifelog usage enables the user to be provided with appropriate information and services.

Fig. 1. Lifelogs.

Building a lifelog over the long term in a continuous fashion is a powerful technique for determining user preferences. For example, let’s assume that a user has been watching soccer matches involving the Japanese national team during the FIFA World Cup. This information in itself cannot be used to determine the extent to which the user is a soccer fan. However, if the user watches soccer matches every week and it is known which teams competed in them, the soccer team that the user is a fan of can be surmised. Similarly, the team that a user follows over a period of several years can be investigated, and if the user’s favorite team is found to have changed, a connection with the transfer of a particular soccer player can be surmised. If, on the other hand, the user almost never watches weekly soccer matches, it can be surmised that the user is only a soccer fan to the extent of watching the World Cup. Analyzing a user’s lifelog obtained over the long term can also be useful in other ways, such as for determining habitual behavior that appears periodically. In short, using user tendencies such as those described above to predict user behavior should enable information that the user truly needs to be provided in a timely manner. In the following section, we take a look at current services that make use of lifelogs.

3. Services using lifelogs

To understand how lifelogs can be used, let us consider two axes (Fig. 2). The first draws a line between individual behavior in the real world and that in cyberspace, while the second separates usage confined to an individual’s lifelog from collective usage of the lifelogs of many people. Traditional health management data such as weight, height, and blood pressure constitutes an example of a real-world log. In this regard, the recent appearance of wearable devices such as pedometers is making it possible to collect data closely associated with the user at all times. And in sports training, GPS-based devices are making it possible to record jogging routes. Logs such as these can be used in the fields of health management and physical training to enable user-specific advice to be given. In short, the use of these logs is confined to the individual. Recently, some users have been uploading their individual records to net-based communities to engage in friendly rivalry with other members of the community in dieting, training, etc. However, log usages such as these are not collective ways.

Fig. 2. Classification of lifelog services.

MyLifeBits [1] is an example of using a lifelog that accumulates a wide array of information over time about a specific individual. It is essentially the first research project to concentrate on lifelogs and is based on the concept that various types of information related to an individual should be managed and used by that person in an integrated manner. MyLifeBits inputs almost all records related to an individual’s activities, from digitally created documents and email and incoming/outgoing telephone calls to photographs and videos taken by that person and scanned images of business cards. The MyLifeBits system is equipped with high-speed search functions that simplify the task of looking back at one’s personal history.

In contrast, Amazon’s product recommendation system [2] is an example of assembling and using the lifelogs of many people. This system makes recommendations in the manner of people who bought this book also bought these books and people who bought this product also bought these accessories. It accumulates the purchasing trends of many users and gives advice to customers who are thinking of purchasing the same products. Since some users may simply forget to purchase accessories or may just not be knowledgeable about necessary items, this service can be very helpful. This sort of recommendation service that also includes search advertising is proving to be popular at present. Such services are successful examples of log usage, and they are helping to attract interest in the use of lifelogs. The data accumulated in the logs of these services relates to user activities in cyberspace, such as web history, search terms, and purchased product history. Logs of these types use the logs of many people and log data is accumulated continuously over the long term. A key point here is that these services are structured in such a way that the data is obtained and stored not through the efforts of the users themselves but by a background process unnoticeable to users.

Location information has recently been attracting attention as the basis for real-world logs. This interest has been driven by the spread of mobile phones equipped with GPS functions. For example, information related to the user’s current location can be given priority for display when the user performs a search from a mobile phone, and photographs taken with a mobile phone can be tagged with location information. These GPS-related services relieve the user from performing extra operations and are proving to be popular as a result. Taking user context into account in the form of location information makes the mobile phone all the more convenient.

As described above, recommendations that take the user’s preferences and context into account are being well received. The benefits for the user include new discoveries that could not be found on one’s own and the receiving of services and information without the burden of having to do extra work. There is no doubt that cyberspace logs are making a big contribution to the realization of these benefits. In the real world, moreover, the appearance of mobile devices raises the possibility of acquiring data closely associated with the user. In a manner similar to the way in which cyberspace logs based on many people are bringing real benefits to users, we envision that real-world logs based on many people or a combination of real-world and cyberspace logs based on many people can be a key to the provision of novel services.

4. Lifelog processing cycle

Service processing making use of lifelogs is shown in Fig. 3. This processing flow must be in the form of a loop so that user satisfaction with services can be continuously fed back to the system, and services can adapt to user needs in a process unnoticeable to users. Using lifelogs requires diverse technologies, and developing all of them on one’s own is not feasible. To provide compelling services, we plan to enhance those technologies that we specialize in while appropriating other suitable technologies. Below, we describe the roles of the technologies covered in the other feature articles in this issue.

Fig. 3. Correspondence between feature articles and lifelog processing cycle.

In lifelog construction, data should be collected over as long a period as possible and in as many varieties as possible. More data enables a greater number and variety of user tendencies to be uncovered.

The second article, “Personal Computer Operation History Data Collection System—Memory Retriever” [3], introduces technology for collecting a history of user actions on a personal computer ((2) in Fig. 3). A user can perform a variety of tasks on a personal computer such as accessing websites, creating documents, and sending and receiving email. The objective here is to acquire as many user tendencies as possible by collecting all sorts of user actions across various types of tasks.

The third article, “Lifelog Remote Control for Collecting Operation Logs Needed for Lifelog-based Services” [4], investigates the use of a remote-control operation log as a real-world lifelog ((3) in Fig. 3). In this regard, a mechanism that can accumulate lifelog data in the background is important; if it takes considerable effort to maintain such records, it will be difficult to collect lifelog data in an ongoing manner over the long term. Remote-control operations have become essential to daily life, which makes them a convenient means of obtaining lifelogs in an incidental manner. Specifically, as remote controls can be used for selecting TV programs, setting the temperature on an air conditioner, turning lights on and off, etc., such operations can be used to understand the lifestyle of an individual in terms of personal likes and dislikes, preferred indoor temperature, bedtime, and other characteristics.

The fourth article, “Data Stream Management Technology for Accumulating and Processing Lifelogs” [5], introduces a data management system for the realtime processing and saving of diverse types of data associated with many people ((4) in Fig. 3). Since lifelogs consist of the records of individuals, the number of users can be expected to increase over time, which calls for a mechanism that can flexibly expand with the system scale. Moreover, services that make use of lifelogs should be current, so realtime processing is necessary. The system introduced here meets both of these requirements.

The fifth article, “Social Science Research Ap-proach for Lifelog Utilization” [6], concerns the issue of privacy, which always arises when lifelogs are discussed. It introduces a means of safeguarding privacy on the twin bases of legal systems and technologies ((5) in Fig. 3).

The remaining two articles introduce two key services. Specifically, the sixth article, “Restaurant Recommendation Service Using Lifelogs” [7], describes a service for recommending restaurants and presents the results of an evaluation experiment ((6) in Fig. 3). This service is one step ahead of existing services of this type since it predicts user behavior and also takes into account not only what the user enjoys eating but also what his or her companions enjoy.

The seventh article, “Prospects for Using Lifelogs in the Medical Field” [8], describes a service that makes use of a person’s vital-signs data and activity-related data in everyday life ((7) in Fig. 3). This service is especially convenient as it connects with network services for viewing medical charts in healthcare institutions.

5. Conclusion

We described the concept of lifelogs and outlined research and development (R&D) efforts in lifelog technologies at NTT Laboratories. While a number of services utilizing lifelogs have already appeared, this field is still in its infancy. It will become increasingly easier to build up lifelogs as an information infrastructure comes into place and devices progress. In future, we plan to continue our research efforts in this field with the aim of making lifelog usage more effective and convenient.

References

[1]	G. Bell and J. Gemmell, “Total Recall: How the E-Memory Revolution Will Change Everything,” Dutton, 2009.
[2]	G. Linden, B. Smith, and J. York, “Amazon.com Recommendations: Item-to-Item Collaborative Filtering,” IEEE Internet Computing, Vol. 7, pp. 76–80, 2003.
[3]	A. Tanaka and T. Uchiyama, “Personal Computer Operation History Data Collection System—Memory Retriever,” NTT Technical Review, Vol. 9, No. 1, 2011. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr201101fa2.html
[4]	T. Watanabe, Y. Takashima, M. Kobayashi, and M. Abe, “Lifelog Remote Control for Collecting Operation Logs Needed for Lifelog-based Services,” NTT Technical Review, Vol. 9, No. 1, 2011. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr201101fa3.html
[5]	T. Hasegawa, I. Naito, T. Menjo, M. Matsuda, H. Akama, and M. Yamamuro, “Data Stream Management Technology for Accumulating and Processing Lifelogs,” NTT Technical Review, Vol. 9, No. 1, 2011. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr201101fa4.html
[6]	K. Takahashi, “Social Science Research Approach for Lifelog Utilization,” NTT Technical Review, Vol. 9, No. 1, 2011. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr201101fa5.html
[7]	H. Tezuka, K. Ito, T. Murayama, S. Seko, M. Nishino, S. Muto, and M. Abe, “Restaurant Recommendation Service Using Lifelogs,” NTT Technical Review, Vol. 9, No. 1, 2011. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr201101fa6.html
[8]	T. Ito, T. Ishihara, Y. Nakamura, S. Muto, M. Abe, and Y. Takagi, “Prospects for Using Lifelogs in the Medical Field,” NTT Technical Review, Vol. 9, No. 1, 2011. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr201101fa7.html

↑ TOP