Regular Articles

Vol. 18, No. 5, pp. 43–48, May 2020. https://doi.org/10.53829/ntr202005ra2

Operation-visualization Technology to Support Digital Transformation

Fumihiro Yokose, Yuki Urabe, Sayaka Yagi,
Kimio Tsuchikawa, Takeshi Masuda, and Haruo Oishi

Abstract

NTT Access Network Service Systems Laboratories (AS Labs) has developed various technologies to improve business operations using personal computers, for example, robotic process automation, annotation technology, and UI (user interface) augmentation technology. Based on this experience, we are researching and developing technologies useful for business-operation analysis. This article introduces the operation-visualization technology developed by AS Labs that supports objective/quantitative business-operation analysis.

Keywords: digital transformation, operation analysis, data visualization

1. Operation analysis to promote DX

Digital transformation (DX) means improvement through digital technology. In particular, business-operation improvement using digital technology is attracting the attention of many companies. NTT Access Network Service Systems Laboratories (AS Labs) has been working on technologies to improve business operations using personal computers (PCs). It developed the UMS (Unified Management Support System), which is a robotic process automation (RPA) technology [1], and NTT Advanced Technology Corporation is now selling it under the name of WinActor^®^*1 [2]. AS Labs is also working on annotation technology and UI (user interface) augmentation technology for improving operation performance that cannot be supported by RPA technologies (e.g., operations that are difficult to standardize and automate) [3]. NTT TechnoCross Corporation is now selling these technologies under the names BizFront/Annotation and BizFront/SmartUI^*2, respectively [4, 5].

By introducing these technologies into NTT Group companies, AS Labs has clarified several issues when using the technologies in actual operational fields. In RPA, for example, automation personnel have to not only learn an RPA tool but also find automatable process flows or problematic operations from the target business operation to design automation processes for RPA. Ideally, field personnel who have business knowledge should learn RPA skills. Since this requires much time and effort, automation personnel having RPA skills and field personnel having business knowledge work together in many cases. In such a situation, automation personnel directly communicate with field personnel about the information needed to apply RPA. However, this is quite subjective, i.e., automation personnel cannot objectively and quantitatively analyze through only such communications.

This issue is the same not only regarding RPA but also in business-operation-improvement activities such as business process management. These activities require objective/quantitative analysis methods as well as direct communication. Based on this knowledge, AS Labs is researching and developing various technologies for business-operation analysis.

*1	WinActor^® is a registered trademark of NTT Advanced Technology Corporation.
*2	BizFront/Annotation and BizFront/SmartUI are trademarks or registered trademarks of NTT TechnoCross Corporation.

2. Operation-visualization technology

Business-operation improvement involves the following four steps: 1) understand current situation, 2) analyze problems, 3) plan for improvement, and 4) action, as shown in Fig. 1 (to increase efficiency, these steps should be repeated as a cycle). The operation-visualization technology AS Labs is researching and developing supports steps 1 and 2. This technology contains a mechanism for recording manipulations on a PC, which AS Labs has cultivated over many years, and original visualization methods newly developed by AS Labs.

Fig. 1. Improvement steps and operation-visualization technology.

AS Labs is not only researching this technology but also developing software to be readily available for business operation. The software consists of two independent tools (executable files): logging and visualization. The logging tool is used on an operator’s PC and detects operation manipulations on the PC and records them in a log file. The visualization tool is used on an analyst’s PC to visualize the data of the log files to understand current operation and perform analysis. The visualization tool currently uses two visualization methods: timeline visualization and process visualization.

With these tools, an analyst can obtain objective/quantitative data about PC manipulations and analyze the data from various viewpoints. As a result, an analyst can more effectively analyze business operations than through direct communication.

3. Logging tool

The logging tool is currently targeted for Windows 7 and Windows 10. It does not require installation and can be executed by simply unpacking to any folder on a target PC.

During execution, the logging tool monitors the operator’s PC manipulations and records the data to a log file (text format). When detecting a specific manipulation, the logging tool capture a window’s screenshot and saves it in an image file. A folder for saving log/image files can be set freely (a shared folder on the network is also possible). If the size of the log/image files is too large, the operator can turn off the screen-capture function.

A summary of the data the logging tool can record is shown in Table 1. Currently, the logging tool supports only Internet Explorer for manipulation-information recording, but AS Labs is considering support for Microsoft Edge and Google Chrome.

Table 1. Information recorded using the logging tool.

4. Visualization tool

The visualization tool loads log files recorded with the logging tool and displays current operations through timeline visualization and process visualization, respectively. The analyst can obtain the visualization results by loading log files for multiple days, multiple PCs, and multiple users at the same time and display these merged data.

4.1 Timeline visualization

Timeline visualization is a visualization method for analyzing window-level situations. A conventional visualization method visualizes only situations for active windows^*3, but timeline visualization also displays situations for inactive windows [6]. Therefore, it is easy to analyze complicated business operations that use multiple windows, for example, working in a business system’s window while referring to a window displaying an operational manual.

As shown in Fig. 2, timeline visualization displays a two-axis graph with time on the horizontal axis and window attributes on the vertical axis. A window attribute means the information-type of a window (application name, window title, loaded URL, loaded file name, PC name, user name, etc.) recorded using the logging tool. An analyst can freely select the types on the window-attribute axis to change the row order. The row order can be selected hierarchically by selecting multiple window-attribute types. Moreover, an analyst can select multiple window-attribute values and add a new group name to aggregate them into neighboring rows so that the analyst can focus on the selected window-attribute values. For example, in Fig. 2, timeline visualization is set to be displayed in this order: group name, user name, date, application name, and window title.

Fig. 2. Timeline visualization.

Timeline visualization then displays the window’s activity state as rectangles. A rectangle has darker and lighter shades of a color; darker shades indicate that the window was active and lighter shades indicate inactivity. The period with no rectangles indicates that the window did not exist or was not visible (a minimized window is regarded as not visible.). If multiple rectangles exist at the same time in the same value row on the window-attribute axis (e.g., the log data when an operator used multiple windows of the same application are visualized on the window-attribute axis of the application-name type.), the rectangles are shown as overlapping. If a darker rectangle of an active window overlaps a lighter rectangle, the darker rectangle is brought to the front.

Timeline visualization has other features as follows.

By hovering a mouse cursor on a rectangle, the analysist can access more information about the window.
The analysist can freely set a color to the rectangle by the window-attribute value (e.g., application name, window title, file name and URL). This setting is independent of the vertical axis.

4.2 Process visualization

Process visualization is a visualization method for more detailed analysis of the manipulation level than of the window level (e.g., button click, text input, list selection). This method displays a graph^*4 of the manipulation flow recorded with the logging tool. This method is more efficient than the conventional visualization method because it has various filter functions to narrow the number of nodes, manual editing functions (grouping nodes and deleting unnecessary nodes), etc. [7].

As shown in Fig. 3, process visualization displays a directed graph on which nodes represent each manipulation and directed edges represent each transition of the manipulation(s). If log data contain multiple events of the same manipulation, process visualization makes multiple events a single node. To be intuitively understandable for an analyst, process visualization displays a related captured-screen image on the node and manipulation position by adding a red frame around the image (e.g., an external rectangle of a manipulated button). The logging tool can currently record manipulation-level events only on Internet Explorer, so an analyst can only analyze a manipulation-level situation regarding Internet Explorer with process visualization.

Fig. 3. Process visualization.

Process visualization has other features as follows.

By hovering a mouse cursor over the node, the analysist can check more information about this manipulation.
Process visualization organizes the graph as follows: log data are separated by order identifier (ID) for each workflow, and nodes of the same manipulation are united into one.
Process visualization highlights the main route in red, which consists of nodes high in manipulating frequency, in the graph.
The analysist can use filters to reduce nodes and observe only the more essential flow, for example, application name, user name, order ID, and manipulating frequency.
The average transition time is shown on an edge of the graph. This indication represents the average time from manipulation of the previous node to that of the next node.

*3	Active window: An active window is the currently focused window in a window-based system such as Microsoft Windows. The active window accepts user manipulations such as with a mouse and keyboard. There can only be one at a time.
*4	Graph: This term means a structure in graph theory (mathematics). This structure consists of nodes and edges.

5. Use example of operation-visualization technology

Timeline visualization enables the following analyses.

By quantitatively measuring work time for operations and determining operations that require a long time, the analysist can find operations that need to be improved.
By comparing the work times of multiple operators in the same operation, the analysist can find experts for this operation.
By focusing on periodic flows, such as repeated access to the same URL, the analysist can find effective operations to implement RPA.

Process visualization enables the following analyses.

By checking the average transition times between process elements, the analysist can find operation flows that take a long time.
By comparing operation processes between experts and beginners, the analysist can find problems experienced by beginners.
By checking a merged process of multiple workflows of the same operation, the analysist can find a standard workflow (most common workflow) and create an automation flow for RPA based on the standard workflow.

6. Future work

As Labs will use the operation-visualization technology in NTT Group companies with their cooperation. We will use the feedback obtained in the trials to further improve its practicability. We will also research and develop not only the current visualization functions but also an analytical function using various artificial intelligence technologies to support the implementation of RPA. Through these activities, we will further improve the technologies for business-operation analysis.

References

[1]	H. Adachi, K. Hotta, F. Yokose, T. Toyoda, and A. Inoue, “UMS: Software for Automating Operators’ Actions to Rapidly Improve Operational Efficiency at Low Cost,” NTT Technical Review, Vol. 12, No. 2, 2014. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr201402fa1.html
[2]	WinActor^® (in Japanese), https://winactor.biz/
[3]	M. Komiyama, H. Koya, H. Nakajima, A. Kataoka, and T. Masuda, “Business Navigation Technology,” NTT Technical Review, Vol. 17, No. 7, 2019. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr201907fa6.html
[4]	BizFront/Anotation (in Japanese), https://www.ntt-tx.co.jp/products/bizfront-ant/
[5]	BizFront/SmartUI (in Japanese), https://www.ntt-tx.co.jp/products/bizfront/sui/
[6]	S. Yagi, K. Tsuchikawa, F. Yokose, Y. Urabe, and T. Masuda, “An Interactive Grouping Method for Timeline-based Operational Log Visualization,” IEICE Technical Report, Vol. 119, No. 111, ICM2019-13, pp. 41–46, 2019 (in Japanese).
[7]	Y. Urabe, S. Yagi, K. Tsuchikawa, and T. Masuda, “Visualizing User Action Data to Discover Business Process,” The Asia-Pacific Network Operations and Management Symposium (APNOMS) 2019, pp. 1–4, Shimane, Japan, 2019.

Trademark notes

All brand, product, and company/organization names that appear in this article are trademarks or registered trademarks of their respective owners.

	Fumihiro Yokose Research Engineer, Navigation Technology Group, Access Network Operation Project, NTT Access Network Service Systems Laboratories. He received an M.E. in electrical engineering from the University of Electro-Communications, Tokyo, in 2007 and joined NTT Access Network Service Systems Laboratories the same year. He has been engaged in research and development of navigation technology for business operation.
	Yuki Urabe Researcher, Navigation Technology Group, Access Network Operation Project, NTT Access Network Service Systems Laboratories. She received a B.E. and Masters in information science from Hokkaido University in 2013 and 2015. She joined NTT Access Network Service Systems Laboratories in 2015 and has been researching business operation analysis methods and recommendation technologies for enterprise documents. She is a member of IEICE.
	Sayaka Yagi Researcher, Navigation Technology Group, Access Network Operation Project, NTT Access Network Service Systems Laboratories. She received a B.S., M.S., and Ph.D. in information sciences from Ochanomizu University, Tokyo, in 2011, 2013, and 2016. She joined NTT Access Network Service Systems Laboratories in 2016. Her research currently focuses on information visualization and visual analytics for supporting business operation analysis. She is a member of the Visualization Society of Japan.
	Kimio Tsuchikawa Senior Research Engineer, Navigation Technology Group, Access Network Operation Project, NTT Access Network Service Systems Laboratories. He received a B.E. and M.E. in applied physics from Nagoya University, Aichi, in 2000 and 2002. He joined NTT in 2002 and is engaged in research and development of navigation technology for business operation. He is a member of IEICE.
	Takeshi Masuda Senior Research Engineer, Access Network Operation Project, NTT Access Network Service Systems Laboratories. He received a B.E. and M.E. in mechanical engineering from Waseda University, Tokyo, in 1991 and 1993. He joined NTT in 1993 and has been engaged in research of operation support systems. He is a member of IEICE.
	Haruo Oishi Senior Research Engineer, Navigation Technology Group, Access Network Operation Project, NTT Access Network Service Systems Laboratories. He received an M.S. from Tokyo Institute of Technology in 1997 and joined NTT the same year, where he researched and developed the mobile middleware technology and the future network systems. He is currently engaged in research and development of navigation technology for business operation. He is a member of IEICE and is secretary of IEICE Technical Committee on Information and Communication Management.

↑ TOP