Yu bi Yomu: A New Text Display System Using Tracing Behavior

Abstract

NTT Communication Science Laboratories is researching a text display system called Yu bi Yomu, in which the appearance of text changes dynamically in response to the user’s finger-tracing behavior. Research on digital text display has so far centered on discussions on how to achieve the feeling of using paper. However, digital text display has the potential to surpass attempts at simply imitating the paper medium by exploiting digital features in order to bring about major changes in the way that reading itself is performed. This article provides an overview of the Yu bi Yomu system and introduces the advantages of using this method.

Keywords: text display technology, interactive interface, finger-tracing reading

1. Introduction

Paper has been the main medium for presenting text for a very long time. However, recent progress in digital media technology is rapidly increasing the opportunities for displaying text on computer displays (digital text displays) instead of printing text on paper. Digital text display can do more than just simplify the handling of huge amounts of textual information. It also has the potential to bring about major changes in the way that reading itself is performed, that is, in reading behavior, by exploiting the features of digital devices.

In principal, the content of text presented on paper does not change. A digital device, however, enables the dynamic presentation of information [1–4]. In this regard, the market for devices equipped with a touch-response function such as tablet computers and smartphones has been growing rapidly in recent years. A key feature of these digital devices is the adoption of an interface by which the user can directly touch and manipulate what is being displayed. Compared to keyboard or mouse operations, using one’s fingers or a touch pen is highly intuitive and interactive. Such methods of direct manipulation have also come to be used for perusing text documents, but most of those methods have been focused on reproducing the sense of manipulating media having fixed text, and the paper medium in particular. The benefits of performing direct operations on characters and text as symbols that convey information have hardly been considered to date.

NTT Communication Science Laboratories is researching a text display system called Yu bi Yomu [5, 6] that makes use of the dynamic-display and touch-based direct-manipulation features of tablet devices (Fig. 1). This system displays text very faintly when the reader is not interacting with the display. However, if the reader begins to trace the characters of that text with his or her finger, the system will gradually increase the contrast of those characters, which is known as fade-in. At this time, the high-contrast characters can be left in that state, but it is also possible to configure the settings so the characters gradually fade out after a specific length of time. In either case, the reader proceeds to read the text while the characters of that text become increasingly visible by finger tracing. In addition, dynamic text display enables intuitive manipulation of such features of reading text as emotional response and intonation that are difficult for static text display to explicitly represent.

Fig. 1. Yu bi Yomu system.

2. Basic structure of software using the Yu bi Yomu system

We are currently implementing the Yu bi Yomu system as software running on commercially available tablet computers. We are also developing software for a variety of devices running under iOS^*1, Android^*2 OS (operating system), Mac OS^*3, and Windows^*4 personal computers (touch-panel displays required), although functions will differ somewhat among these devices. This software assumes the preparation of text files that use a special format to manipulate the text that the user would like to display with the Yu bi Yomu system (Fig. 2(a)). This format is a type of Extensible Markup Language (XML)^*5 and describes the text to be displayed and symbols called tags in much the same way that web pages are prepared. Preparing text files in this format (Yu bi Yomu files) enables the user to change the text to be displayed as well as the temporal profile of elements such as the text font and font size without having to modify the software program. For example, specifying the following four parameters controls the temporal change in the appearance of text when finger tracing (Fig. 2(a)).

(1) Time from user touching the screen to start of text fade-in

(2) Time from start of text fade-in to maximum contrast

(3) Time that text is held at maximum contrast

(4) Time from start of text fade-out back to original state

We found that manipulating these parameters describing the temporal profile of text display can change the reader’s overall impression of reading a document, as indicated by the words warm and soft, and cool and stiff in Fig. 2(b) [6].

Fig. 2. Basic structure of software using the Yu bi Yomu system.

The Yu bi Yomu system can also record the user’s tracing behavior. The user can start and stop this recording by pressing a software button and can make use of the recorded tracing behavior without having to do any programming or editing. In particular, the data obtained by recording tracing behavior can be used to prepare animation in which text automatically rises up based on a time series of finger-tracing actions. Additionally, recorded data may be saved in a tab-separated text file and exported to spreadsheet software such as Excel^*6 to enable reading behavior to be analyzed for educational purposes.

*1	iOS is a registered trademark of Cisco Systems Inc. in the United States and other countries and is used by Apple Inc. under license.
*2	Android is a registered trademark of Google Inc. in the United States and other countries.
*3	Mac OS is a trademark of Apple Inc. in the United States and other countries.
*4	Windows is a registered trademark of Microsoft Corporation in the United States and other countries.
*5	XML: The name of a language or its specifications for describing instructions related to the logical structure or form of text together with the body of that text all in a text file.
*6	Excel is a registered trademark of Microsoft Corporation in the United States and other countries.

3. Usage scenarios for the Yu bi Yomu system

The Yu bi Yomu system can be applied to the text-communication and education fields (Fig. 3).

Fig. 3. Application of the Yu bi Yomu system.

3.1 Application to text communication

We consider that incorporating software using the Yu bi Yomu system in text-communication tools such as email and messaging apps can facilitate smooth communication. For example, text-based animation using tracing data can enable a user to add nuance and subtle emotions to short messages. Animated text using tracing data can also be combined with a function for recording audio during finger tracing so that an animated message can be created that combines speech and moving text.

Animation in which text automatically flows on a display has come to be used in many everyday scenarios such as television captions, karaoke systems, and electronic billboards. Experts have traditionally prepared such animation using computers and specialized software, but the Yu bi Yomu system makes it easy for even general users to create animated text for smartphones and other devices.

Furthermore, in most animated text prepared by computer, the speed at which text appears is fixed. In contrast, animated text based on human tracing includes text whose appearance accelerates and decelerates in an uneven manner. Such complex displays of text can express intonation in a message and convey the sender’s individuality and presence [7, 8], which cannot easily be achieved with traditional computer systems.

3.2 Application to education

We consider that the Yu bi Yomu system could also be applied with good effect in the field of education. Compared to silent reading, reading while tracing accompanies a physical action (finger tracing) that can be observed by other people. Moreover, as the position of words being read basically agrees with the position of the finger while tracing, examining a record of finger tracing makes it easy to learn how that person is progressing in reading that text. In this way, the Yu bi Yomu system makes it relatively simple to visualize the way in which someone reads text.

For example, in a classroom setting, using the Yu bi Yomu system to prepare teaching material makes it easy to visualize interactive behavior while reading, which has the potential to improve the quality of instruction. We conducted an experiment in collaboration with NTT-ME in the use of Yu bi Yomu teaching material. In this experiment, we introduced teaching material using the Yu bi Yomu system in classroom-based training targeting ordinary employees, and we compared the results with instruction using the same material but displayed statically in the usual way on a tablet device [9]. We found that the average score in a memory test taken after training was about 10% higher in the group receiving instruction using Yu bi Yomu material than in the group using material presented in the conventional way. There are a number of possibilities as to why test results improved in this way, and further research is needed to arrive at a detailed explanation. At the least, however, the results of this experiment have shown that the quality of instruction can be improved by using the Yu bi Yomu system for certain types of teaching material and classroom settings.

Furthermore, on converting the tracing data obtained from learners who used Yu bi Yomu teaching materials into graphic images and examining the results, it was found that the way in which reading was approached could differ greatly between people. Examples of finger tracing between two learners—one who did well in the recollection test and one who did poorly—are shown in Fig. 4. It is obvious which of the two approached the material in a more serious manner.

Fig. 4. Examples of finger tracing.

Text display using the Yu bi Yomu system can also be combined with a text-to-speech function to prepare teaching material on foreign languages and ancient texts such as sutras that may be difficult to read on one’s own. For example, software to read text out loud while finger tracing could be created by preparing a text-to-speech file beforehand and changing the generated speech in unison with the text being traced. Likewise, educational software with a text-to-speech function could provide a sense of self-improvement as one learns how to read text that one could not otherwise read on one’s own. This approach holds the possibility of keeping the learner motivated and focused with a desire to continue learning.

4. Future outlook

Incorporation of the Yu bi Yomu system introduced in this article in the software used in popular digital devices such as smartphones and tablets is expected to expand the communication functions provided by those devices. Furthermore, combining the software with a text-to-speech function should make it applicable to picture books read out loud by a familiar voice. In addition, collecting a large amount of tracing data over time may make it possible to uncover key relationships between tracing and learning conditions. If so, we can expect the Yu bi Yomu system to be useful in large-scale learning systems (Fig. 5).

Fig. 5. Application to large-scale learning systems.

The Yu bi Yomu system focuses on the temporal and interactive characteristics associated with the digital display of information. It is an attempt at incorporating the temporal expression of emotional response and intonation common to the spoken language in textual expression using tracing behavior. In this research, it is not our aim to reproduce on digital devices textual expression as achieved on the medium of paper. Rather, we seek to exploit the special features and formats of digital devices to open up new possibilities in communication.

References

[1]	J. C. Lee, J. Forlizzi, and S. E. Hudson, “The Kinetic Typography Engine: an Extensible System for Animating Expressive Text,” Proc. of UIST 2002 (15th annual ACM Symposium on User Interface Software and Technology), ACM Press (2002), pp. 81–90, Paris, France, Oct. 2002.
[2]	J. Forlizzi, J. C. Lee, and S. E. Hudson, “The Kinedit System: Affective Messages Using Dynamic Texts,” Proc. of CHI (Conference on Human Factors in Computing Systems) 2003, ACM Press (2003), pp. 377–384, Ft. Lauderdale, Florida, USA, Apr. 2003.
[3]	Y. Y. Wong, “Temporal Typography: a Proposal to Enrich Written Expression,” Proc. of CHI 1996, ACM Press (1996), pp. 408–409, Vancouver, Canada, Apr. 1996.
[4]	G. Möhler, M. Osen, and H. Harrikari, “A User Interface Framework for Kinetic Typography-enabled Messaging Applications,” Proc. of CHI 2004, ACM Press (2004), pp. 1505–1508, Vienna, Austria, Apr. 2004.
[5]	K. Maruya, M. Uetsuki, H. Ando, and J. Watanabe, ““Yu bi Yomu”: Interactive Reading of Dynamic Text,” Proc. of MM 2012 (20th ACM International Conference on Multimedia), ACM Press (2012), pp. 1499–1500, Nara, Japan, Apr. 2012.
[6]	K. Maruya, M. Uetsuki, H. Ando, and J. Watanabe, “Dynamic Text Display Using Finger Trailing,” IPSJ Journal, Vol. 54, No. 4, pp. 1507–1517, 2012 (in Japanese).
[7]	F. Heider and M. Simmel, “An Experimental Study of Apparent Behavior,” The American Journal of Psychology, Vol. 57, No. 2, pp. 243–259, 1944.
[8]	P. D. Tremoulet and J. Feldman, “The Influence of Spatial Context and the Role of Intentionality in the Interpretation of Animacy from Motion,” Perception and Psychophysics, Vol. 68, No. 6, pp. 1047–1058, 2006.
[9]	K. Maruya, J. Watanabe, H. Takahashi, and S. Hashiba, “A Learning System Utilizing Learners’ Active Tracing Behaviors,” Proc. of LAK15 (5th International Conference on Learning Analytics and Knowledge), pp. 418–419, New York, USA, Mar. 2015.

Kazushi Maruya: Senior Research Scientist, Sensory Representation Research Group, Human Information Science Laboratory, NTT Communication Science Laboratories*.
He received an M.A. and a Ph.D. in psychology from the University of Tokyo in 2001 and 2005. He joined NTT Communication Science Laboratories in 2008. He was a researcher in the Intelligent Modeling Laboratory, the University of Tokyo, from 2004 to 2005, and a visiting scientist in the Department of Psychology, Vanderbilt University, Nashville, USA, from 2006 to 2008. His current research interests include visual motion processing for perception, especially for natural scene perception, and human computer interface design for reading digital text.
* He moved to Research Planning Department of NTT Science and Core Technology Laboratory Group on August 1, 2015.

Junji Watanabe: Senior Scientist, Human Information Science Laboratory, NTT Communication Science Laboratories.
He received a Ph.D. in information science and technology from the University of Tokyo in 2005. His academic work has been published in scientific journals in the field of neuroscience and interface technologies. He has also presented his work at technology showcases, science museums, and art festivals such as at SIGGRAPH (2006–2009, 2014) and Ars Electronica (2002, 2004, 2007–2015). His research is focused on cognitive science and communication devices with applied perception.

↑ TOP