Personal Information Style

Shunichi Seko, Manabu Motegi, Masayuki Ihara,
Toru Kobayashi, and Ryoji Kataoka

Abstract

NTT Service Evolution Laboratories has developed the personal information style, which is a technique of providing an IT (information technology) usage environment that is customized to each user’s individual style. This article introduces a technique for obtaining information related to a video that a user is watching and casually displaying the information on the terminal in front of the user. This related information can be personalized at the touch of a fingertip, as in the interaction between a television and a tablet terminal.

1. Overview of personal information style

Recent technological advances have resulted in a huge explosion in the amount of available information, and along with this, there has been a rapid increase in content and web services that are attractive to users. At the same time, the widespread use of smartphones and tablet terminals is enabling users to use web services to access information on the web that was virtually inaccessible to ordinary people in the past, and as a result, the user layer is clearly expanding.

There is a range of environments that enable users to utilize web content and services effortlessly. This has been spurred by factors such as improved usability of terminals. However, there is still no environment that will efficiently enable any user, no matter how much experience they have in using the web, to readily track down the content or web services that they want from among the huge volume of information. There is a difference in the information that can be acquired by people who can effectively combine suitable keywords that should be input to the search engine, for example, and those who cannot. Thus, a problem exists in which searching out necessary information from among the huge quantity of content and web servers depends on the skills and information technology (IT) literacy of the user.

This article introduces a personal information style that is aimed at a user-centric information acquisition environment that is closer to the user than content providers and services. This personal information style provides content and services that are matched to factors such as the user’s profile, context, and the device environment. We implement an information acquisition environment that reduces the load on the user by bringing that information closer to the user rather than making the user get closer to the content and services, and that does not depend on skills or IT literacy.

2. Approach

To implement a personal information style, a wedge method is first used to create associations between the user and the content (Fig. 1). Conventionally, the configuration is such that users actively search for their own preferred content and web services, and only the requested content is delivered to the device that made each request, as shown in Fig. 1(a). With the proposed approach, a wedge is inserted between the devices owned by the user and the content, as shown in Fig. 1(b). The wedge acts as a gateway when information on the Web is accessed from devices owned by the user, with the format being such that the wide variety of content on the web arrives in accordance with the user’s idiosyncrasies and the device environment. This gives rise to the following three benefits:

Fig. 1. Approaches for implementing personal information style.

(1) Integrated provision of related content from other media sources: Up to now, content was provided from a single media source specified by the user. However, inserting this wedge makes it possible to integrate and provide related content that is thought to be of interest to the user, from other media sources.

(2) Customization of design and interface for each user: The design and interface that has been customized for each user can be modified by interposing the wedge into a design that has been integrated in media units. Content that has been coordinated between devices owned by the user can also be presented by creating a mashup^*1 of the devices.

(3) No interference to service providers: The provision of content that is integrated with other media has no effect on the media provision interface on the service provider side since it is implemented on the wedge side.

*1	Mashup: Integrating techniques and content from a number of different providers to form a new service.

3. Architecture

The architecture for implementing the wedge method is shown in Fig. 2. The information style engine acts as a wedge to implement the wedge method. It has a content mashup and filtering engine, full-duplex communication functions between devices, and a sensitive user interface.

Fig. 2. Architecture of personal information style.

(1) Content mashup and filtering engine

This engine acquires a mixture of content from a number of media sources, either based on the user’s actions or on the idiosyncrasies or devices owned by the user, and sends the results to the devices owned by the user. This makes it possible to create a mashup of web content and provide it to the user.

(2) Full-duplex communication function between devices

This function provides full-duplex communication to each of the devices owned by the user to enable data transfer between them. Connecting devices by full-duplex communication enables device cooperation (mashup between devices) so that an action taken by the user on one device acts as a trigger to automatically display content on a separate device.

(3) Sensitive user interface

This interface displays information in a way that is sensitive to the user’s direct actions and lifestyle, instead of in the one-sided manner of the past. This enables appropriate review of the content that should be included in the mashup and display of content that matches its status, by responding swiftly with respect to changes that occur on the user side.

As described above, the wedge method performs a cycle of operations that include presenting the result of a content mashup depending on the user’s actions or changes in device states, obtaining feedback with respect to that result, and filtering the content on the web.

4. Use case of content recommendation coordinated with currently watched video

Here, we introduce a system that finds content that is related to a video the user is currently watching and displays it on a smartphone or tablet in front of the user, in coordination with the video, as an example use of personal information style. The user can get more information by utilizing this system. The information might consist of details of a restaurant that has been introduced in the video, or access to sites for purchasing the literary works of the performers or the accessories they are wearing, without actively searching for such information. This system is implemented in HTML5 (hypertext markup language, fifth revision), so we give details of the system and explain the advantages of using HTML5.

4.1 System overview

An overview of the content recommendation system that is coordinated with the video that the user is currently watching is shown in Fig. 3. The user possesses a digital TV or tablet terminal, and full-duplex communication is enabled for this terminal through an information style engine server by the HTML5 WebSocket^*2.

Fig. 3. Overview of content recommendation system coordinated with currently watched video.

Using WebSocket reduces communication traffic and the load on the web server, in comparison with conventional full-duplex communication. The information style engine server can access various types of content on the Web and is configured to temporarily store the acquired content in the server itself. We describe this operation below.

If the user selects a video on digital television (TV), information on that selection is transmitted to the information style engine server. The server acquires metadata containing pertinent video data and keywords related to that video from the web. The video data are transmitted to the digital TV, and replay is started. The acquired metadata are input as parameters to Linked Open Data (LOD)^*3 applications such as DBpedia^*4, and related keywords in the metadata are acquired from the LOD. Since the metadata will not always exist in sufficient quantity, going through the LOD will expand or bulk out the related keywords. The system uses the expanded metadata to search for content on the web and acquires content related to the video being watched. The acquired related content is displayed on the tablet terminal in front of the user, enabling the user to browse content that is related to the video being watched. The user’s actions with respect to the displayed related content are also fed back to the information style engine server. The system acquires new related content based on this feedback, and modifies the displayed related content to match the user’s actions. In addition, if the user changes the video he is watching, the system is designed to restart the above processing; then, content related to the new video footage will be displayed.

4.2 Content rejection/selection user interface

Another feature of the system is an interface that is swiftly responsive to the user’s actions. This interface operates the feedback cycle smoothly and displays or manipulates the related content. A display screen of this user interface is shown in Fig. 4. The image shows the method of manipulation and depicts how the display appears when related content is rejected or selected. Related content is displayed as icons on this user interface, more detailed information will appear when the user taps the icons. Content icons appear automatically as small animations that congregate from outside the screen into a central circle. These automatically disappear from the screen if the user does nothing, and icons for new content appear as similar animations. This display method ensures that the user need only act when wanting to know more details and reduces the pushiness that often exists with recommendation techniques. It is possible not only to tap content icons but also to drag-and-drop them. If an icon enters the central circle, information related to that content automatically gathers around the central circle from outside the screen. This enables the user to collect content of interest by simply moving promising icons into the circle, without having to input keywords. Conversely, the user can deliberately erase content by flicking the icon toward the outside of the screen. Since these actions are also fed back to the information style engine server, it is possible to easily inform the system which information is unwanted. In this manner, the user can readily search for content of interest through the simple actions of pulling in or flicking out the icons that continue to appear.

Fig. 4. Content selection user interface.

This user interface is also implemented in HTML5. Since the icons displayed in Fig. 4 are rectangular images when acquired from the web, they are trimmed into circles by using a Scalable Vector Graphics (SVG)^*5 filtering function for display. The content icon display and animation are all defined as inline SVG^*6, so it is possible to represent any display state as HTML tags. This enables coordination with different systems and interaction with other devices utilizing HTML tags, and will also provide expandability.

*2	WebSocket: A standard for providing full-duplex communication by web servers or web browsers. More precisely, it is a technique that has been isolated by HTML5, but in the broader sense it is handled as a technique by HTML5.
*3	LOD: Abbreviation of a technique for releasing a data collection that has been optimized by computer processing, onto the web. It is configured so that related data are associated, and is expressed as a uniform resource identifier (URI). Accessing that URI enables acquisition of information related to the data.
*4	DBpedia: Community project that extracts Wikipedia information and releases it as LODs. For example, if the data "Shounan" is input, it is possible to acquire data such as the related place names "Enoshima" and "Yuigahama" as well as the names of famous people who have connections to those places.
*5	SVG: An image format in vector graphic form, described by XML (extensible markup language). Broadly, it is handled as one HTML5 technique.
*6	Inline SVG: Method of displaying SVG by describing it directly as a tag within an HTML file without reading and displaying it as an image file. This has enabled display by inline description from HTML5.

5. Future plans

The example of personal information style we described in this article involved interaction with digital TV, but personal information styles can also be applied in the presentation of related content using techniques such as digital signage or position information as the trigger, without using a TV. Note that the NTT Plala video delivery service of Hikari TV has provided a Nagarami Assist (assistance while watching) service that utilizes this technology since November 2012. However, at this point, the service can only cope with a limited number of contexts, and it has not yet reached the ideal form of the wedge method proposed in Fig. 1. We will therefore continue our research and development in the future, with the aim of implementing a service that is sensitive to a large variety of contexts such as the user’s idiosyncrasies, behavior, and preferences, and the environment and state of the devices, to provide the greatest user experience regardless of circumstances.

	Shunichi Seko Researcher, Service Harmonization Project, NTT Service Evolution Laboratories. He received the B.E. and M.E. degrees in media and governance from Keio University, Kanagawa, in 2006 and 2008, respectively. He joined NTT in 2008 and is currently studying information sharing under disaster situations and a user interface for recommendations. He received the FIT 2010 Young Researcher's Awards from the Institute of Information and Communication Engineers (IEICE) and the IEEE ICCE 2011 Special Merit Awards.
	Manabu Motegi Research Engineer, Innovative Service Architecture Project, NTT Service Evolution Laboratories. He received the M.E. and Ph.D. degrees from Tokyo Institute of Technology in mechano-micro engineering in 1995 and 2008, respectively. He has been with NTT since 1995. His research interests include multi-robot collaboration, human-machine collaboration, and data mining with multi-sensors. He received the Encouragement Award from the Robotics Society of Japan in 1998.
	Masayuki Ihara Senior Research Engineer, Supervisor, Service Harmonization Project, NTT Service Evolution Laboratories. He received the M.E. and Ph.D. degrees from Tokyo Institute of Technology in 1994 and 2010, respectively. He worked at the New Media Innovation Centre and at the University of British Columbia in Canada from 2002 to 2003. His research interests include value based design, cognition of human values and preferences, and human-computer interaction. He is a member of the Association for Computing Machinery, the Information Processing Society of Japan (IPSJ), and IEICE.
	Toru Kobayashi Professor of Electrical Engineering and Computer Science Division, Graduate School of Engineering, Nagasaki University. He joined NTT in 1987 just after graduating from the Graduate School of Mechanical Engineering, Tohoku University, Miyagi. He received the Ph.D. degree from the Graduate School of Information Systems, the University of Electro-Communications, Tokyo, in 2011. During his career at NTT, he was engaged in R&D of software engineering, network platforms, groupware, information security, ubiquitous computing, data mining, etc. He also spent almost four years in Düsseldorf, Germany, where he managed joint research projects and standardization activities related to smart card technologies in cooperation with an EU research institution. He is a senior member of IEICE and a member of IEEE and IPSJ. He joined the faculty of Nagasaki University in April 2013. He was Senior Research Engineer, Supervisor, Group leader of Service Harmonization Project, NTT Service Evolution Laboratories at the time of this article was written.
	Ryoji Kataoka Senior Research Engineer, Supervisor, NTT Service Evolution Laboratories. He received the B.E. and M.E. degrees in electronic engineering from Chiba University in 1985 and 1987, respectively. He joined NTT in 1987 and has been engaged in research on transaction processing, multimedia information systems, and information retrieval. He is a member of IPSJ.

↑ TOP