Automatic Generation of Recovery-command Sequences

1. Introduction

In current large-scale ICT (information and communication technology) systems, troubleshooting has become more complicated due to the diversification of the causes of network failures. The increase in operational costs has also become a serious problem. We are developing technology for automatic generation of recovery-command sequences that is designed to help system operators recover from failures quickly and achieve automated recovery operations [1].

2. Overview of technology

An overview of our technology is shown in Fig. 1. Sequences of recovery commands are estimated by using a sequence-to-sequence technique (seq2seq) [2], which is a neural-network model that learns the relationship between an input sequence and an output sequence (Fig. 2).

Fig. 1. Outline of automatic generation of recovery-command sequences.

Fig. 2. Estimation by seq2seq.

Seq2seq is widely used in translation systems and dialog tasks. In our technology, we use a sequence that consists of a series of log identifiers (IDs) as an input sequence. The log IDs are generated by associating system logs and alarms related to system failures with unique numbers [3]. We also use a sequence of words that consists of a recovery-command sequence as an output sequence. Learning the relationship between the input sequence and the output sequence makes it possible to estimate a command sequence that will restore the system when a new failure occurs.

When the command sequence estimated in this method is executed, it is necessary to measure the reliability of the estimation and the impact on the system of the command sequence. In our technology, we define the reliability of a command sequence by multiplying the generated probabilities of each word that composes the recovery-command sequence. Thus, the reliability can be regarded as a probability of the system recovery when the obtained command sequence is executed. Moreover, we can define the impact on the system by using the information about the impact on performance of the system when recovery-command sequences were executed in past failures. These indicators (i.e., reliability and impact) can be used to decide whether to execute the obtained command sequence.

3. Future work

We will continue to work on verifying our technology by using data obtained from commercial systems and improving the accuracy of the estimated recovery-command sequences. We will also improve the definitions of the reliability and the impact from the viewpoint of practical system operation to achieve automated recovery operations.

References

[1]	H. Ikeuchi, A. Watanabe, Y. Matsuo, and T. Kawata, “Automatic Generation of Recovery Command Sequences Using Seq2Seq,” IEICE General Conference, B-7-25, Tokyo, Japan, Mar. 2019 (in Japanese).
[2]	I. Sutskever, O. Vinyals, and Q. Le, “Sequence to Sequence Learning with Neural Networks,” Advances in Neural Information Processing Systems 27 (NIPS 2014), 2014.
[3]	T. Kimura, A. Watanabe, T. Toyono, and K. Ishibashi, “Proactive Failure Detection Learning Generation Patterns of Large-scale Network Logs,” Proc. of the 11th International Conference on Network and Service Management (CNSM 2015), Barcelona, Spain, Nov. 2015.