NTT’s LLM “tsuzumi”

1. NTT’s LLM “tsuzumi” expands the application domains of LLMs

A variety of large language models (LLMs), such as OpenAI’s ChatGPT, have been released, attracting significant attention. To accumulate a greater amount of knowledge and provide natural answers, many LLMs have exploded in size. It has been reported that a model of the size of GPT-3 requires about 1300 MWh (equivalent to one hour of electricity generated by a nuclear power plant) for training [1]. Because of the prohibitive initial and operational costs, including the hardware required for large-scale training, many companies find it impractical to create their own language models. They thus turn to commercial LLMs provided as cloud services. However, these companies often handle data with personal and sensitive information, so simply uploading such data to the cloud is problematic. A company’s data are considered a part of its assets. How to make effective use of such data with LLMs has thus become a challenge.

To address these issues, NTT Human Informatics Laboratories has been engaged in the research and development of tsuzumi, a lightweight LLM with superior Japanese-language processing ability, and released it to the public in November 2023. The significant advantages of tsuzumi’s lightweight nature include reduction in hardware resources and power required for using and tuning the LLM. These benefits enable the use of tsuzumi by companies on their premises, allowing them to apply and effectively use an LLM on data that would be problematic when stored on external cloud servers.

A decrease in the size of an LLM generally leads to a reduction in performance. However, NTT laboratories’ over-40 years of experience in natural language processing has allowed tsuzumi to outperform publicly available LLMs with parameters of similar size in Japanese language processing. Furthermore, tsuzumi features adapter tuning to achieve flexible tuning and multimodal support, which enables comprehension of documents that include non-text content such as figures and diagrams. The key features of tsuzumi are explained in detail in the following sections.

2. Lightweight model

The size of an LLM is measured in terms of its parameter size. There are two versions of tsuzumi: a lightweight version with 7 billion (7B) parameters and an ultra-lightweight version with 600 million (0.6B) parameters.

The parameters of an LLM are variables used to store knowledge and skills acquired by the model during training. The greater a model’s parameter size, the greater the model’s capabilities tend to be in accumulating knowledge and responding to queries from humans. However, increasing parameter size also leads to an increase in the computational resources and power consumption required for training and inference. The technological challenge is thus to reduce an LLM’s parameter size while maintaining its performance.

The smaller an LLM’s parameter size, the less knowledge it can generally store, leading to reduced performance. For tsuzumi, we have thus taken the approach of improving the quality of training data. For example, we seek to remove noise—information that should not be learned, such as redundant or erroneous information—and create training data with information from a wide range of fields instead of a few selective fields. For tsuzumi’s ultra-lightweight version, we succeeded in making the model even more lightweight while maintaining its performance by focusing the training on domains and tasks handled by the model.

With this approach, tsuzumi’s parameter size was trimmed to 1/25th to 1/300th the size of publicly available massive LLMs, resulting in significant cost reduction (Fig. 1).

Fig. 1. Relationship between lightweight models and cost reduction.

3. Superior Japanese-language processing ability

When training an LLM, text included in the training data is broken down (tokenized) into “tokens.” Tokens are the fundamental units processed by an LLM.

Tokenizers of recent LLMs determine the token set or vocabulary by learning from a collection of texts. LLMs developed outside Japan have a small amount of tokens for the Japanese language included in their vocabulary, and most break down Japanese text into tokens of a single character or byte. Therefore, a great number of tokens are needed to generate Japanese text, which is highly inefficient in terms of text-generation speed. We have thus developed a tokenizer with vocabulary that can efficiently generate Japanese text by training it based on Japanese text. However, the training algorithms of commonly used tokenizers do not take the structure of the Japanese language into account. Thus, there is the issue of the likelihood of redundant tokens, which often appear in the training corpus but not in other texts, being included in the vocabulary.

To address this issue, tsuzumi incorporates unique processing (lexical constraint) that takes Japanese words into account when tokenizing. The tokenizers of other companies may include redundant tokens, such as those that appear frequently in Wikipedia but not in other texts, in its segmentation results. In contrast, tsuzumi uses lexical constraint to achieve segmentation that strongly reflects the structure of the Japanese language. In deploying lexical constraint, NTT applied the results of its long endeavor in the research and development of morphological analysis tools and dictionaries.

Tokenization affects not only the speed of text generation but also the accuracy of text comprehension. To improve the quality of training data described above, tsuzumi’s unique tokenization also results in high Japanese-language processing ability. Figure 2 shows the results of tsuzumi’s Japanese-language processing ability compared with other publicly available LLMs, as measured using the Rakuda benchmark. This benchmark consists of a series of questions testing knowledge related to Japan and is often used to compare the performances of Japanese-language LLMs. The same questions were given to LLMs for comparison, and GPT-4 was used to determine a better response of each question between two LLMs. The results indicate that tsuzumi has superior Japanese knowledge and Japanese-language processing ability, not only against LLMs of similar size but also against GPT-3.5.

Fig. 2. Comparison of tsuzumi’s Japanese-language processing ability against other commercial LLMs.

4. Adapter tuning

The tsuzumi LLM supports adapter tuning. This tuning adds (adapts) a small domain-specific model to the original model. Figure 3 provides an image of adapter tuning.

Fig. 3. Image of adapter tuning.

Other tuning methods include prompt engineering, which provides relevant information at the same time as the query during inference, and full fine-tuning, which updates the entire model. Figure 4 provides a summary of each method’s features. Each tuning method has its own characteristics in terms of cost and accuracy. Adapter tuning has the advantage of achieving both at a certain level. When considering use cases within different companies, the deployment of different adapters for each company makes it possible to easily create company-specific models.

Fig. 4. Comparison of tuning methods.

We are working on providing support for multiple adapters, as shown in Fig. 5. This will allow the requirements of a variety of use cases to be considered by connecting multiple adapters on top of tsuzumi.

Fig. 5. Multiple adapters: Enable switching and combining adapters in accordance with the user or use case.

5. Multimodal support

As its name implies, an LLM is a model for processing language, so it does not take graphical and audio data as its input or output. However, figures and tables are often included in general documents; such non-text content often has significant meaning. The tsuzumi LLM was designed to be a new type of LLM that comprehends figures and tables [2, 3] as well as speech and user situations (Fig. 6). Such use cases were exhibited at the 2023 NTT R&D Forum.

Fig. 6. Image of modal support.

The feature article “NTT’s LLM ‘tsuzumi’: Capable of Comprehending Graphical Documents” [4] in this issue explains tsuzumi’s techniques for visual comprehension of graphical documents, one of its multimodal capabilities. Please read it in addition to another feature article “Commercialization of NTT’s LLM ‘tsuzumi’” [5], which describes the usage domain and usage applications, solution menu, and other features of tsuzumi commercial services launched in March 2024.

6. Future developments

As we have described, tsuzumi boasts key features, such as being lightweight while possessing superior Japanese-language processing ability and flexibility in tuning, as well as multimodal support. We are continuing research and development of tsuzumi in the following areas:

References

[1]	M. DeGeurin, “Move Aside, Crypto. AI Could Be the Next Climate Disaster,” Gizmodo, Apr. 2023. https://gizmodo.com/chatgpt-ai-openai-carbon-emissions-stanford-report-1850288635
[2]	R. Tanaka, T. Iki, K. Nishida, K. Saito, and J. Suzuki, “InstructDoc: A Dataset for Zero-shot Generalization of Visual Document Understanding with Instructions,” Proc. of the 38th Annual AAAI Conference on Artificial Intelligence (AAAI-24), Vancouver, Canada, Feb. 2024.
[3]	T. Hasegawa, K. Nishida, K. Maeda, and K. Saito, “DueT: Image-Text Contrastive Transfer Learning with Dual-adapter Tuning,” Proc. of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), pp. 13607–13624, Singapore, Dec. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.839
[4]	R. Tanaka, T. Iki, T. Hasegawa, and K. Nishida, “NTT’s LLM ‘tsuzumi’: Capable of Comprehending Graphical Documents,” NTT Technical Review, Vol. 22, No. 8, pp. 26–30, Aug. 2024. https://ntt-review.jp/archive/ntttechnical.php?contents=ntr202408fa2.html
[5]	N. Shibuya, I. Kudo, M. Kaido, and S. Yamamoto, “Commercialization of NTT’s LLM ‘tsuzumi,’” NTT Technical Review, Vol. 22, No. 8, pp. 31–37, Aug. 2024. https://ntt-review.jp/archive/ntttechnical.php?contents=ntr202408fa3.html