Sang-Woo Lee


# Introduction

The natural language processing (NLP) community was surprised to learn that zero/few-shot in-context learning ability also emerges in a large-scale language model such as GPT-3 ([Brown et al., NeurIPS, 2020](https://arxiv.org/abs/2005.14165)) despite no duplication between downstream task data and the pre-training corpus. Last year, we at the HyperCLOVA team introduced [HyperCLOVA](https://naver-ai-now.kr) ([Kim et al., EMNLP, 2021](https://arxiv.org/abs/2109.04650)), a large-scale Korean AI platform based on GPT-3. We have since shared our research results and applied the capabilities of HyperCLOVA to various service products.

GPT-3 has garnered attention through its ability of in-context learning. In-context learning is done by feeding a few training examples and/or task descriptions with a new input to a large-scale language model (LM) for it to produce a target of this input, without requiring any parameter updates. In-context learning can further be categorized by the number of examples used. A few training examples are used in the in-context *few-shot* learning setting, whereas no training examples are used in a *zero-shot* setting.

There still remain many questions regarding in-context learning for language models despite the successful reports on their capabilities. For example, the relationship between the choice of a pre-training corpus and downstream in-context learning task accuracy is unknown. However, pre-training corpora are a significant factor for language model performance, and research on pre-training corpora is important from the perspective of conserving data collection costs. In this article, we will be looking into the results of our ablation study on how the performance of in-context learning changes depending on the corpus used.


# Pre-training corpus

There are seven sub-corpora in the HyperCLOVA corpus: [*Blog*](https://blog.naver.com/), [*Cafe*](https://cafe.naver.com/), [*News*](http://news.naver.com), *Comments*, [*KiN*](http://kin.naver.com), [*Modu*](http://corpus.korean.go.kr), and Ency. Blog, Cafe, and News are taken from blogs, community websites, and the online news articles hosted on NAVER, a Korean web portal service. Comments is the comment threads related to the three sub-corpora mentioned above. KiN comes from a Korean online community Q&A service similar to Quora. Ency is a collection of encyclopedic texts including the Korean section of Wikipedia. Modu consists of five public datasets constructed by the National Institute of Korean Language.

Unlike other corpus used in GPT-3 training, such as Common Crawl, the HyperCLOVA corpus consists of a well-separated sub-corpus with a balanced ratio. So, it is advantageous for analyzing differences by domain type of the sub-corpus.

![proportion of sub-corpus](/public/contents/posts/ko/2022/05/hyperclova-corpus/post5_02.png)  
Figure: Proportion of sub-corpus in the HyperCLOVA corpus. A number in the circle denotes the size of the sub-corpus (Billion tokens).

![illustration of sub-corpus](/public/contents/posts/en/2022/05/hyperclova-corpus/post5_03_en.png)  
Figure: Illustration of sub-corpus in the HyperCLOVA corpus

We evaluate the in-context learning performance of each corpus-specific model on five Korean downstream tasks. The five tasks consist of binary sentiment classification, machine reading comprehension, Korean to English translation, English to Korean translation, and topic classification.


# Discovery 1: In-context learning performance heavily depends on the corpus domain source

It is noticeable that in-context learning ability emerges differently depending on pre-training corpus sources. For example, the Blog model demonstrates in-context few-shot learning performance comparable to the ALL model, the model using the entire HyperCLOVA corpus. However, each of the Cafe and News models hardly show any in-context few-shot learning ability. At first glance, the difference between Blog (blog) and Cafe (online community) corpus does not seem to be large. However, it is surprising how much different the abilities of Blog and Cafe actually are.

In addition, as shown in the below figures, the size of pre-training corpus does not necessarily predict the emergence of in-context learning. Also, perplexity, evaluated with the validation set of seven sub-corpus, and in-context learning are not always correlated: e.g., low perplexity does not always imply high in-context learning performance. For example, Cafe's corpus size and validation perplexity are second best, but the performance of Cafe is lower than Modu.

![few-shot main result](/public/contents/posts/ko/2022/05/hyperclova-corpus/post5_04.png)  
Figure: Averaged in-context few-shot learning performance for five Korean downstream tasks

![zero-shot main result](/public/contents/posts/ko/2022/05/hyperclova-corpus/post5_05.png)  
Figure: Averaged in-context zero-shot learning performance for five Korean downstream tasks

Gopher ([Rae et al., arXiv, 2022](https://arxiv.org/abs/2112.11446)), a GPT-3 model of Deepmind, determines the ratio between sub-corpora based on the perplexity of the validation corpus. They implicitly claim that this ratio results in better downstream task performance, but do not offer any explicit evidence for this. On the other hand, we are in a position to doubt the strong correlation between perplexity and in-context learning, especially in the few-shot setting.


# Discovery 2: In-context learning ability can emerge when multiple corpora are combined

We discovered that in-context few-shot learning ability can emerge by combining two corpora, even if each of both corpora cannot provide in-context few-shot learning ability. For example, the KiN+Ency model succeeds in generating in-context learning ability in most tasks, while each of KiN and Ency fails in most tasks. Likewise, the Cafe+KiN model succeeds in generating in-context few-shot learning ability, while each of Cafe and KiN fails in most tasks.

Unlike these positive cases, a counterexample also exists. Not only Cafe and KiN but also the Cafe+KiN model fails to make in-context few-shot learning in most tasks.

![few-shot combination result](/public/contents/posts/ko/2022/05/hyperclova-corpus/post5_06.png)  
Figure: In-context few-shot learning performance with different corpus combination

# Discovery 3: Pre-training with a corpus related to a downstream task does not always guarantee the competitive in-context few-shot learning performance

KLUE-YNAT is one of five downstream tasks used in our research; it is used for news headline classification by topic. We can infer that the in-context learning performance of KLUE-YNAT would increase when the News corpus is trained.

The experimental results are quite interesting, as shown in the graph below. Training the News corpus consistently increases in-context zero-shot learning in KLUE-YNAT. The models with trained corpora including the News corpus even perform better than the model trained using the entire HyperCLOVA corpus.

However, the News corpus does not enhance the in-context few-shot learning of KLUE-YNAT. The performance of KLUE-YNAT is inferior when only trained with the News corpus. Even when training the News corpus with other corpora, the performance for KLUE-YNAT somehow decreased compared to the case without the News corpus.

![result KLUE-YNAT](/public/contents/posts/ko/2022/05/hyperclova-corpus/post5_07.png)  
Figure: Comparison of in-context learning performance of KLUE-YNAT with and without training the News corpus

Previous studies, like DAPT ([Gururangan et al., ACL, 2020](https://arxiv.org/abs/2004.10964)), argue that pre-training with a corpus similar to the downstream task improves the downstream performance. However, these observations are often limited to cases where a pre-trained language model is fine-tuned for the downstream task. Our study shows that a pre-training corpus related to a downstream task does not always guarantee the competitive in-context few-shot learning performance of the downstream task; the trends shown in in-context learning may be quite different from the cases of fine-tuning.


# Conclusion

In our research, we've investigated the effects of the source and the size of the training corpus on in-context learning ability, using the HyperCLOVA corpus. Our discoveries conclude that corpus sources play a crucial role in whether or not in-context learning ability will emerge in a large-scale language model.
Our study will be presented at the NAACL22 conference as a regular presentation. If you are interested in more discoveries we found, you can read about them in our arXiv paper [[arXiv](https://arxiv.org/abs/2204.13509)].

Our research is by no means the final say on this topic. For example, our understanding of the causes behind the various interesting discoveries described above are admittedly limited. In addition, our research results were only obtained from the Korean corpus, and further studies on other languages are required. However, analytical research on in-context learning has been conducted, and the corresponding understanding has been deepening ([Min et al., arXiv, 2022](https://arxiv.org/abs/2202.12837)). We hope our findings contribute to making better in-context learners along with other research.


Introduction

The natural language processing (NLP) community was surprised to learn that zero/few-shot in-context learning ability also emerges in a large-scale language model such as GPT-3 (Brown et al., NeurIPS, 2020) despite no duplication between downstream task data and the pre-training corpus. Last year, we at the HyperCLOVA team introduced HyperCLOVA (Kim et al., EMNLP, 2021), a large-scale Korean AI platform based on GPT-3. We have since shared our research results and applied the capabilities of HyperCLOVA to various service products.

GPT-3 has garnered attention through its ability of in-context learning. In-context learning is done by feeding a few training examples and/or task descriptions with a new input to a large-scale language model (LM) for it to produce a target of this input, without requiring any parameter updates. In-context learning can further be categorized by the number of examples used. A few training examples are used in the in-context few-shot learning setting, whereas no training examples are used in a zero-shot setting.

There still remain many questions regarding in-context learning for language models despite the successful reports on their capabilities. For example, the relationship between the choice of a pre-training corpus and downstream in-context learning task accuracy is unknown. However, pre-training corpora are a significant factor for language model performance, and research on pre-training corpora is important from the perspective of conserving data collection costs. In this article, we will be looking into the results of our ablation study on how the performance of in-context learning changes depending on the corpus used.

Pre-training corpus

There are seven sub-corpora in the HyperCLOVA corpus: Blog, Cafe, News, Comments, KiN, Modu, and Ency. Blog, Cafe, and News are taken from blogs, community websites, and the online news articles hosted on NAVER, a Korean web portal service. Comments is the comment threads related to the three sub-corpora mentioned above. KiN comes from a Korean online community Q\&A service similar to Quora. Ency is a collection of encyclopedic texts including the Korean section of Wikipedia. Modu consists of five public datasets constructed by the National Institute of Korean Language.

Unlike other corpus used in GPT-3 training, such as Common Crawl, the HyperCLOVA corpus consists of a well-separated sub-corpus with a balanced ratio. So, it is advantageous for analyzing differences by domain type of the sub-corpus.

proportion of sub-corpus
Figure: Proportion of sub-corpus in the HyperCLOVA corpus. A number in the circle denotes the size of the sub-corpus (Billion tokens).

illustration of sub-corpus
Figure: Illustration of sub-corpus in the HyperCLOVA corpus

We evaluate the in-context learning performance of each corpus-specific model on five Korean downstream tasks. The five tasks consist of binary sentiment classification, machine reading comprehension, Korean to English translation, English to Korean translation, and topic classification.

Discovery 1: In-context learning performance heavily depends on the corpus domain source

It is noticeable that in-context learning ability emerges differently depending on pre-training corpus sources. For example, the Blog model demonstrates in-context few-shot learning performance comparable to the ALL model, the model using the entire HyperCLOVA corpus. However, each of the Cafe and News models hardly show any in-context few-shot learning ability. At first glance, the difference between Blog (blog) and Cafe (online community) corpus does not seem to be large. However, it is surprising how much different the abilities of Blog and Cafe actually are.

In addition, as shown in the below figures, the size of pre-training corpus does not necessarily predict the emergence of in-context learning. Also, perplexity, evaluated with the validation set of seven sub-corpus, and in-context learning are not always correlated: e.g., low perplexity does not always imply high in-context learning performance. For example, Cafe's corpus size and validation perplexity are second best, but the performance of Cafe is lower than Modu.

few-shot main result
Figure: Averaged in-context few-shot learning performance for five Korean downstream tasks

zero-shot main result
Figure: Averaged in-context zero-shot learning performance for five Korean downstream tasks

Gopher (Rae et al., arXiv, 2022), a GPT-3 model of Deepmind, determines the ratio between sub-corpora based on the perplexity of the validation corpus. They implicitly claim that this ratio results in better downstream task performance, but do not offer any explicit evidence for this. On the other hand, we are in a position to doubt the strong correlation between perplexity and in-context learning, especially in the few-shot setting.

Discovery 2: In-context learning ability can emerge when multiple corpora are combined

We discovered that in-context few-shot learning ability can emerge by combining two corpora, even if each of both corpora cannot provide in-context few-shot learning ability. For example, the KiN+Ency model succeeds in generating in-context learning ability in most tasks, while each of KiN and Ency fails in most tasks. Likewise, the Cafe+KiN model succeeds in generating in-context few-shot learning ability, while each of Cafe and KiN fails in most tasks.

Unlike these positive cases, a counterexample also exists. Not only Cafe and KiN but also the Cafe+KiN model fails to make in-context few-shot learning in most tasks.

few-shot combination result
Figure: In-context few-shot learning performance with different corpus combination

Discovery 3: Pre-training with a corpus related to a downstream task does not always guarantee the competitive in-context few-shot learning performance

KLUE-YNAT is one of five downstream tasks used in our research; it is used for news headline classification by topic. We can infer that the in-context learning performance of KLUE-YNAT would increase when the News corpus is trained.

The experimental results are quite interesting, as shown in the graph below. Training the News corpus consistently increases in-context zero-shot learning in KLUE-YNAT. The models with trained corpora including the News corpus even perform better than the model trained using the entire HyperCLOVA corpus.

However, the News corpus does not enhance the in-context few-shot learning of KLUE-YNAT. The performance of KLUE-YNAT is inferior when only trained with the News corpus. Even when training the News corpus with other corpora, the performance for KLUE-YNAT somehow decreased compared to the case without the News corpus.

result KLUE-YNAT
Figure: Comparison of in-context learning performance of KLUE-YNAT with and without training the News corpus

Previous studies, like DAPT (Gururangan et al., ACL, 2020), argue that pre-training with a corpus similar to the downstream task improves the downstream performance. However, these observations are often limited to cases where a pre-trained language model is fine-tuned for the downstream task. Our study shows that a pre-training corpus related to a downstream task does not always guarantee the competitive in-context few-shot learning performance of the downstream task; the trends shown in in-context learning may be quite different from the cases of fine-tuning.

Conclusion

In our research, we've investigated the effects of the source and the size of the training corpus on in-context learning ability, using the HyperCLOVA corpus. Our discoveries conclude that corpus sources play a crucial role in whether or not in-context learning ability will emerge in a large-scale language model.
Our study will be presented at the NAACL22 conference as a regular presentation. If you are interested in more discoveries we found, you can read about them in our arXiv paper \[arXiv].

Our research is by no means the final say on this topic. For example, our understanding of the causes behind the various interesting discoveries described above are admittedly limited. In addition, our research results were only obtained from the Korean corpus, and further studies on other languages are required. However, analytical research on in-context learning has been conducted, and the corresponding understanding has been deepening (Min et al., arXiv, 2022). We hope our findings contribute to making better in-context learners along with other research.


On the effect of pre-training corpora on in-context learning by large-scale language model.

Kim Minsub

Lee Sungjae

# Part 3: Optimization points
## Introduction
In this part, I would like to cover two technologies greatly needed for optimizing HyperCLOVA-based services. First, I'll talk about the multi-batch feature that combines multiple requests and processes them at once to efficiently process user requests. Second, I'll show you what we've done to provide a faster response in multi-turn pattern services, such as CLOVA AI Speaker, where conversations are made in turns. Here is the table of contents:

- Multi-batch
- Multi-turn

## Multi-batch
### Why we need multi-batch
Companies dealing with large-scale traffic may consider multi-batch when they need to increase throughput while providing services using AI models.

In general, the inference process, unlike the learning process, focuses more on reducing latency than on increasing throughput. Thus, rather than using a maximum batch size (i.e., increasing the size that can be processed at once to maximum), we reduce latency by processing only a single batch at a time. However, when an AI model has to handle large-scale traffic, even if the batch size is reduced to a minimum, the overall latency for each request may not be reduced. This is because if the amount of requests to be processed increases, the requests that came in later may spend a lot of time waiting in the queue while the AI model is processing the requests that came in first. So even if the computation latency is reduced, the waiting time for entering the computation process increases, and as a result the user experiences a slowdown.

In addition, we should also consider the GPU environment in which the service is provided and the characteristics of GPT-3-based models such as HyperCLOVA. GPUs perform parallel processing using numerous cores and high memory bandwidths. In order to make the most of GPUs with these characteristics, it is recommended to provide data at once as long as the GPU can handle the size. In this case, even if the amount of each computation increases, the latency required for the computation does not increase significantly.

You can see this pattern more clearly in GPT-based models, such as HyperCLOVA. As I mentioned in the previous two blog posts, GPT computation consists of a summarization step for processing data in input sentences at once, and a generation step for generating words sequentially. In the summarization step, multiple token IDs are processed at once, so the numerous cores of the GPU can be used more efficiently. However, in the sentence generation step, tokens are generated one by one, and those many GPU cores may not be fully used. This means that the utilization of cores and memory can be poor.

If we increase batch size to increase the amount of data to be processed in the generation step, computation latency does not increase significantly because the GPU's numerous cores perform the task successfully. Therefore, processing multiple requests at once using multi-batch can be a great benefit in terms of throughput in the generation step.

If you look at the figure below, you can see that the batch size has been increased from 1 to 8 in the process of generating sentences for a specific input, but the computation latency has hardly increased. So, if there are too many requests to be processed, it can be effective in increasing the batch size to increase the instantaneous throughput so that we can shorten the time each request has to wait in the queue and reduce the overall latency.

![img](/public/contents/posts/en/2022/03/hyperclova-part-3/multibatch01.png)

### Application and problems of multi-batch
At first, we tried to apply a technology called dynamic batch for the batch process. Dynamic batching specifies a maximum batch size (e.g., 4, 8) and combines the requests received for a set duration (e.g., 1 second) into one batch before the computation. Through this, we tried to reduce the overall latency by increasing throughput and reducing the waiting time in the queue by processing several requests that came in almost at the same time.

However, when multi-batch was applied, the latency doubled in certain situations while throughput increased without significant changes in latency in most cases. Looking closely, we found this problem occurred when the length of input sentences in the requests combined into one batch was significantly different.

This was due to the structural problem of the transformer. Let's look at the example below. When generating a sentence by setting the batch size to 1 (single batch) for each input sentence, you can simply process each input sentence at once in the summarization step and generate a sentence of the desired length in the generation step. However, if the two input sentences are grouped into one batch, the length of the sentence that can be processed in the summarization step is fixed to the length of the shorter sentence among the two sentences. This is because if it is set to the length of the longer sentence in the summarization step, it is impossible to accurately process the shorter sentence.

![img](/public/contents/posts/en/2022/03/hyperclova-part-3/multibatch02.png)

If the length of the sentence processed in summarization step is decreased for multi-batch processing like this, as all other parts need to be processed one word at a time in the generation step, it can lead to an increase in latency. In the worst case, if the length difference between the sentences you want to combine into batch is several times the length of the sentence you want to generate, multi-batch application can result in an extreme increase in latency and even a timeout.

### First approach: Bucketing strategy
The first method we used to solve the problem that occurred when processing sentences of different lengths in multi-batch is the bucketing strategy. To put it simply, the bucketing strategy classifies requests with similar lengths of sentences using multiple buckets. Even if the requests came in almost at the same time, if the difference in length between the two sentences is greater than the predetermined threshold, they will not be combined and processed as one batch.

![img](/public/contents/posts/en/2022/03/hyperclova-part-3/multibatch03.png)

This method can prevent latency from increasing significantly while applying multi-batch. However, even if this method is used, latency can still increase as the length of sentences to be processed in the generation step increases when sentences with different lengths are grouped into batches. In addition, as we try to minimize the in sentence length difference when grouping them into batches, we often cannot run multi-batch in services, which makes it difficult to benefit from multi-batch.

### Second approach: Using GPT attention mask
The second approach uses GPT attention mask, a method that was inspired by Hugging Face. The bucketing strategy described earlier was not actually optimizing computation but rather preventing a sharp decline in performance in certain situations by considering service patterns. The heart of the problem of inefficient multi-batch operations is that when the input sentence lengths of requests that are to be grouped into batches are different, computations that could have been done in the summarization step are serialized, and the latency increases. So we thought about how we could go beyond the bucketing strategy and optimize computation to solve this problem.

What caught my eye was GPT's masked self-attention. In a GPT model, only decoders, not encoders, are used in the transformer. The main difference between the encoder and the decoder is that the encoder uses self-attention while the decoder uses masked self-attention. A self-attention encoder examines the context both before and after the token being learned. For this reason, it is known to have strengths in extracting the meaning of sentences. On the other hand, a masked self-attention decoder can only look at the context before the token being learned. Therefore, a GPT model composed of decoder layers has an advantage in sentence generation. Here, an attention mask is used to allow the decoder to only look at the previous context, excluding tokens that came out later from the computation.

![img](/public/contents/posts/en/2022/03/hyperclova-part-3/multibatch04.png)

In other words, with the attention mask, we can include only the tokens we want in the computation. The problem we had with the multi-batch feature earlier was that latency increased as calculation that could have been done in the summarization step was passed to the generation step when input sentences of different lengths came in. However, the attention mask and padding can solve this problem.

So here's what you should do. First, among the sentences to be input together as a batch, add padding with a random value in the shorter sentence to match the length of the longest input. By delivering the same length of inputs to HyperCLOVA, you can generate only necessary sentences because the lengths of the inputs processed together as a batch in the HyperCLOVA model are the same.

In this case, a token corresponding to the padding is included after the sentence that came in as a short input, and this can hinder the generation of the correct input sentence. We adjusted attention mask value appropriately so that the value located in the padding would not be included in the computation. We also applied this method to the CUDA kernel inside FasterTransformer in collaboration with NVIDIA.

![img](/public/contents/posts/en/2022/03/hyperclova-part-3/multibatch05.png)

By modifying the attention mask in the multi-batch process, it was possible to prevent the computation time from increasing as computation is serialized when processing requests with different input lengths. The degree of performance improvement varied depending on the distribution of input sentence lengths in the requests, but when the batch size was increased to 8, the processing time that used to be 5 times slower than a single batch was improved to become 20% faster than a single batch. In addition, when the input sentences had similar lengths, the throughput was improved by six times compared to a single batch.

## Multi-turn
There are various patterns in the service using HyperCLOVA. Among them, there is a multi-turn pattern in which HyperCLOVA and the user have a conversation every turn and create a new sentence based on the previous conversation.

In a multi-turn pattern, HyperCLOVA understands the context better and can generate high-quality sentences. This multi-turn pattern becomes an optimization point from an engineering point of view. In general, each time HyperCLOVA and the user have a conversation over several turns, records are accumulated, resulting in a longer request sentence with HyperCLOVA. However, since the previous conversation has already performed the calculation when processing the previous request, if the calculation result is well cached, the overlapping part can be skipped and the response time can be shortened.

The figure below, which was also included in the second post of the three-part series on HyperCLOVA Service, shows the decoder computation process.

![img](/public/contents/posts/en/2022/03/hyperclova-part-3/multiturn01.png)

The decoder uses masked self-attention, generating its own key and value by referring to all the keys and values available as the previous input results. In the figure above, the keys and values of the Token ID [3, 21, 1] are used for the computation for Token ID 7, and for the next token, Token ID 99, the keys and values of the Token ID [3, 21, 1, 7] are used for computation. In other words, if this information is cached elsewhere and everything is the same as the previous token ID, it is possible to bring and use the cached information for the next token ID. The figure below shows how caching works in a multi-turn pattern.

![img](/public/contents/posts/en/2022/03/hyperclova-part-3/multiturn02.png)

For example, if you continue to receive the user's inputs when "nice to meet you" is output after "Hello," you can just use the cached results without performing computation for the keys and values corresponding to "Hello nice to meet you" again. In this way, you can omit the computation for most of the input sentences.

However, you cannot store computation results corresponding to all requests in the GPU. Since the size of the stored keys and values is proportional to the size of the model and the number of tokens, they can take up all GPU memory with just a few requests. Therefore, keys and values should be stored in a space like DRAM. In this case, it takes additional time to exchange data between the CPU and the GPU. In other words, there is a trade-off between the computational time and data transfer time. Therefore, in order to maximize performance, we allocated a part of the DRAM in the form of pinned-memory as a caching space to store the computation results.

Our test showed that performance was improved by 20% compared to the existing method in a case where the caching effect can be experienced, as shown in the figure below. When a cache miss occurred, the latency increased by about 10% due to the time spent to send the keys and values in the GPU to DRAM. However, it was also confirmed that the increase in latency can be minimized by allowing parallel operation of data transmission and computation.

![img](/public/contents/posts/en/2022/03/hyperclova-part-3/multiturn03.png)

So we confirmed that the performance can be improved, but we did not apply this to the service. This is because the benefits were not as great as expected. As I mentioned earlier, in GPT-3, a bottleneck usually occurs in the sentence generation step, so caching did not have much effect. In addition, the size of the keys and values to be cached was large, so it took longer than expected to retrieve data from the GPU.

To provide faster service, we will continue to improve HyperCLOVA by identifying ways to reduce the size of keys and values or more appropriate cases for applying the caching techniques.  



Part 3: Optimization points

Introduction

In this part, I would like to cover two technologies greatly needed for optimizing HyperCLOVA-based services. First, I'll talk about the multi-batch feature that combines multiple requests and processes them at once to efficiently process user requests. Second, I'll show you what we've done to provide a faster response in multi-turn pattern services, such as CLOVA AI Speaker, where conversations are made in turns. Here is the table of contents:

Multi-batch

Multi-turn

Multi-batch

Why we need multi-batch

Companies dealing with large-scale traffic may consider multi-batch when they need to increase throughput while providing services using AI models.

In general, the inference process, unlike the learning process, focuses more on reducing latency than on increasing throughput. Thus, rather than using a maximum batch size (i.e., increasing the size that can be processed at once to maximum), we reduce latency by processing only a single batch at a time. However, when an AI model has to handle large-scale traffic, even if the batch size is reduced to a minimum, the overall latency for each request may not be reduced. This is because if the amount of requests to be processed increases, the requests that came in later may spend a lot of time waiting in the queue while the AI model is processing the requests that came in first. So even if the computation latency is reduced, the waiting time for entering the computation process increases, and as a result the user experiences a slowdown.

In addition, we should also consider the GPU environment in which the service is provided and the characteristics of GPT-3-based models such as HyperCLOVA. GPUs perform parallel processing using numerous cores and high memory bandwidths. In order to make the most of GPUs with these characteristics, it is recommended to provide data at once as long as the GPU can handle the size. In this case, even if the amount of each computation increases, the latency required for the computation does not increase significantly.

You can see this pattern more clearly in GPT-based models, such as HyperCLOVA. As I mentioned in the previous two blog posts, GPT computation consists of a summarization step for processing data in input sentences at once, and a generation step for generating words sequentially. In the summarization step, multiple token IDs are processed at once, so the numerous cores of the GPU can be used more efficiently. However, in the sentence generation step, tokens are generated one by one, and those many GPU cores may not be fully used. This means that the utilization of cores and memory can be poor.

If we increase batch size to increase the amount of data to be processed in the generation step, computation latency does not increase significantly because the GPU's numerous cores perform the task successfully. Therefore, processing multiple requests at once using multi-batch can be a great benefit in terms of throughput in the generation step.

If you look at the figure below, you can see that the batch size has been increased from 1 to 8 in the process of generating sentences for a specific input, but the computation latency has hardly increased. So, if there are too many requests to be processed, it can be effective in increasing the batch size to increase the instantaneous throughput so that we can shorten the time each request has to wait in the queue and reduce the overall latency.

img

Application and problems of multi-batch

At first, we tried to apply a technology called dynamic batch for the batch process. Dynamic batching specifies a maximum batch size (e.g., 4, 8) and combines the requests received for a set duration (e.g., 1 second) into one batch before the computation. Through this, we tried to reduce the overall latency by increasing throughput and reducing the waiting time in the queue by processing several requests that came in almost at the same time.

However, when multi-batch was applied, the latency doubled in certain situations while throughput increased without significant changes in latency in most cases. Looking closely, we found this problem occurred when the length of input sentences in the requests combined into one batch was significantly different.

This was due to the structural problem of the transformer. Let's look at the example below. When generating a sentence by setting the batch size to 1 (single batch) for each input sentence, you can simply process each input sentence at once in the summarization step and generate a sentence of the desired length in the generation step. However, if the two input sentences are grouped into one batch, the length of the sentence that can be processed in the summarization step is fixed to the length of the shorter sentence among the two sentences. This is because if it is set to the length of the longer sentence in the summarization step, it is impossible to accurately process the shorter sentence.

img

If the length of the sentence processed in summarization step is decreased for multi-batch processing like this, as all other parts need to be processed one word at a time in the generation step, it can lead to an increase in latency. In the worst case, if the length difference between the sentences you want to combine into batch is several times the length of the sentence you want to generate, multi-batch application can result in an extreme increase in latency and even a timeout.

First approach: Bucketing strategy

The first method we used to solve the problem that occurred when processing sentences of different lengths in multi-batch is the bucketing strategy. To put it simply, the bucketing strategy classifies requests with similar lengths of sentences using multiple buckets. Even if the requests came in almost at the same time, if the difference in length between the two sentences is greater than the predetermined threshold, they will not be combined and processed as one batch.

img

This method can prevent latency from increasing significantly while applying multi-batch. However, even if this method is used, latency can still increase as the length of sentences to be processed in the generation step increases when sentences with different lengths are grouped into batches. In addition, as we try to minimize the in sentence length difference when grouping them into batches, we often cannot run multi-batch in services, which makes it difficult to benefit from multi-batch.

Second approach: Using GPT attention mask

The second approach uses GPT attention mask, a method that was inspired by Hugging Face. The bucketing strategy described earlier was not actually optimizing computation but rather preventing a sharp decline in performance in certain situations by considering service patterns. The heart of the problem of inefficient multi-batch operations is that when the input sentence lengths of requests that are to be grouped into batches are different, computations that could have been done in the summarization step are serialized, and the latency increases. So we thought about how we could go beyond the bucketing strategy and optimize computation to solve this problem.

What caught my eye was GPT's masked self-attention. In a GPT model, only decoders, not encoders, are used in the transformer. The main difference between the encoder and the decoder is that the encoder uses self-attention while the decoder uses masked self-attention. A self-attention encoder examines the context both before and after the token being learned. For this reason, it is known to have strengths in extracting the meaning of sentences. On the other hand, a masked self-attention decoder can only look at the context before the token being learned. Therefore, a GPT model composed of decoder layers has an advantage in sentence generation. Here, an attention mask is used to allow the decoder to only look at the previous context, excluding tokens that came out later from the computation.

img

In other words, with the attention mask, we can include only the tokens we want in the computation. The problem we had with the multi-batch feature earlier was that latency increased as calculation that could have been done in the summarization step was passed to the generation step when input sentences of different lengths came in. However, the attention mask and padding can solve this problem.

So here's what you should do. First, among the sentences to be input together as a batch, add padding with a random value in the shorter sentence to match the length of the longest input. By delivering the same length of inputs to HyperCLOVA, you can generate only necessary sentences because the lengths of the inputs processed together as a batch in the HyperCLOVA model are the same.

In this case, a token corresponding to the padding is included after the sentence that came in as a short input, and this can hinder the generation of the correct input sentence. We adjusted attention mask value appropriately so that the value located in the padding would not be included in the computation. We also applied this method to the CUDA kernel inside FasterTransformer in collaboration with NVIDIA.

img

By modifying the attention mask in the multi-batch process, it was possible to prevent the computation time from increasing as computation is serialized when processing requests with different input lengths. The degree of performance improvement varied depending on the distribution of input sentence lengths in the requests, but when the batch size was increased to 8, the processing time that used to be 5 times slower than a single batch was improved to become 20% faster than a single batch. In addition, when the input sentences had similar lengths, the throughput was improved by six times compared to a single batch.

Multi-turn

There are various patterns in the service using HyperCLOVA. Among them, there is a multi-turn pattern in which HyperCLOVA and the user have a conversation every turn and create a new sentence based on the previous conversation.

In a multi-turn pattern, HyperCLOVA understands the context better and can generate high-quality sentences. This multi-turn pattern becomes an optimization point from an engineering point of view. In general, each time HyperCLOVA and the user have a conversation over several turns, records are accumulated, resulting in a longer request sentence with HyperCLOVA. However, since the previous conversation has already performed the calculation when processing the previous request, if the calculation result is well cached, the overlapping part can be skipped and the response time can be shortened.

The figure below, which was also included in the second post of the three-part series on HyperCLOVA Service, shows the decoder computation process.

img

The decoder uses masked self-attention, generating its own key and value by referring to all the keys and values available as the previous input results. In the figure above, the keys and values of the Token ID \[3, 21, 1] are used for the computation for Token ID 7, and for the next token, Token ID 99, the keys and values of the Token ID \[3, 21, 1, 7] are used for computation. In other words, if this information is cached elsewhere and everything is the same as the previous token ID, it is possible to bring and use the cached information for the next token ID. The figure below shows how caching works in a multi-turn pattern.

img

For example, if you continue to receive the user's inputs when "nice to meet you" is output after "Hello," you can just use the cached results without performing computation for the keys and values corresponding to "Hello nice to meet you" again. In this way, you can omit the computation for most of the input sentences.

However, you cannot store computation results corresponding to all requests in the GPU. Since the size of the stored keys and values is proportional to the size of the model and the number of tokens, they can take up all GPU memory with just a few requests. Therefore, keys and values should be stored in a space like DRAM. In this case, it takes additional time to exchange data between the CPU and the GPU. In other words, there is a trade-off between the computational time and data transfer time. Therefore, in order to maximize performance, we allocated a part of the DRAM in the form of pinned-memory as a caching space to store the computation results.

Our test showed that performance was improved by 20% compared to the existing method in a case where the caching effect can be experienced, as shown in the figure below. When a cache miss occurred, the latency increased by about 10% due to the time spent to send the keys and values in the GPU to DRAM. However, it was also confirmed that the increase in latency can be minimized by allowing parallel operation of data transmission and computation.

img

So we confirmed that the performance can be improved, but we did not apply this to the service. This is because the benefits were not as great as expected. As I mentioned earlier, in GPT-3, a bottleneck usually occurs in the sentence generation step, so caching did not have much effect. In addition, the size of the keys and values to be cached was large, so it took longer than expected to retrieve data from the GPU.

To provide faster service, we will continue to improve HyperCLOVA by identifying ways to reduce the size of keys and values or more appropriate cases for applying the caching techniques.


Optimization points for HyperCLOVA services

# Part 2: Implementing service features
## Introduction
This is the second part of the three-part series on HyperCLOVA. In this post, I'll talk about how we implemented service features using HyperCLOVA. First, let me introduce Early Stop, a feature we implemented to deliver faster responses to users with NAVER's HyperCLOVA-based service patterns, such as CLOVA AI Speaker. Second, I'll explain how we were able to provide Semantic Search, a feature that analyzes the relevance between documents using a sentence generation model called the HyperCLOVA Model. And finally, I'll talk about P-tuning, a feature we implemented to achieve higher performance with the same model size in the serving environment. Here's the table of contents:

- Early Stop
- Semantic Search
- P-tuning

## Early Stop
The most time-consuming step in a GPT-3 model such as HyperCLOVA is sentence generation. Due to the structural characteristics, transformer computation consists of two steps: a summarization step for processing input sentences and a generation step for generating sentences. In the summarization step, token IDs corresponding to input sentences are processed in batches simultaneously, while in the generation step, tokens are sequentially generated one by one. Therefore, the sentence generation step is responsible for the largest portion of the total computation time and reducing sentence length will directly reduce the response time. Then, how can you reduce sentence length in this step?

![img](/public/contents/posts/en/2022/02/hyperclova-part-2/earlystop01.png)

The figure above is an example where CLOVA AI Speaker requests HyperCLOVA to generate sentences. Based on the previous question of the conversation between CLOVA and the user in the upper-left corner, CLOVA AI Speaker asks HyperCLOVA to generate a sentence. At this time, the length of the sentence to be generated is also indicated as the number of tokens. HyperCLOVA generates tokens based on the requested length, converts them into sentences and then sends a response to CLOVA AI Speaker.

At this point, the requesting party has no idea how much HyperCLOVA will generate until it actually receives an answer. Therefore, a request allows a sufficient number of tokens for sentence generation, and only the part that user needs can be used as a response. In the example, the user only needs the sentence that starts with `CLOVA:` Thus, the sentence that starts with `User:` will be cut off, and only the first sentence will be used for the answer.

If HyperCLOVA generates and returns only the sentence that the user needs when generating sentences as shown in the figure below, it can reduce the sentence generation time, which accounts for the largest portion of the total latency, and also greatly reduce the response time. This feature is called Early Stop.

![img](/public/contents/posts/en/2022/02/hyperclova-part-2/earlystop02.png)

However, for HyperCLOVA with FasterTransformer as a transformer framework, we could not directly implement the Early Stop feature. In order to implement Early Stop, we needed a process to convert the generated token ID into a string every time to check if it matched the string for termination. This is because FasterTransformer, which is provided in the form of a shared library after compilation, can only be used with a set interface. It can only receive the results after the token IDs are generated based on the requested length.

To address this issue, we added a tokenizer for Early Stop in FasterTransformer and an interface to handle it to allow the generated token ID to be changed to a string every time.

![img](/public/contents/posts/en/2022/02/hyperclova-part-2/earlystop03.png)

What we needed was a function to convert token IDs into strings, so we only implemented the decoder part of the tokenizer. In this case, the decoder part can be implemented using a simple hash map, because the information mapped with the Unicode corresponding to each token ID can be moved to the hash map as-is. The figure above shows the process of converting the token ID [3, 2, 10000] to the "User" string using the decoder of the tokenizer. By indexing the vocab mapped to each token ID and combining the Unicode found, the token ID can be converted to a string.

![img](/public/contents/posts/en/2022/02/hyperclova-part-2/earlystop04.png)

The figure above shows a graph where a log was kept and analyzed for the length of the requested sentence and the length of the sentence actually generated after the Early Stop feature was implemented. The blue line represents the requested length, and the green line represents the length of the generated sentence. Here you can see that the requested length was much longer than the length of the generated sentence in the service. In extreme cases, the latter was more than 10 times shorter than the former. In this case, it was also confirmed that the actual response time was reduced by nearly 10 times.


## Semantic Search
GPT-3 models, such as HyperCLOVA, are essentially very suitable for processing sentence generation tasks. However, if you look closely at the principles of sentence generation, you can see that it can also be used for other purposes. In this section, we will summarize how HyperCLOVA can handle the task of semantic search and how we can set up the computational process.

What is Semantic Search? Semantic search is the process of inferring the context and relevance of input sentences or words and scoring them. Semantic search can be used to build a recommendation system using the relevance between sentences and words, and can also be used to evaluate if the trained model selects words of appropriate quality.

As shown in the example below, the semantic search feature calculates the relevance between the basic sentences and the candidate sentences as a score when they are added as input sentences. The higher the relevance between the basic sentences and the candidate sentences, the lower the score. In the example below, it was inferred that among the sentences in the candidate group, "doenjang-jjigae" has the highest relevance with the basic sentence.

![img](/public/contents/posts/en/2022/02/hyperclova-part-2/semantic01.png)

Now let's compare the process of generating sentences in HyperCLOVA with the process of a semantic search.

First, let's look at how these two processes are performed. GPT models, such as HyperCLOVA, basically compute using multiple transformer layers. These transformer layers use the concepts of query, key, and value. Query is related to the token currently given, and key and value are related to the entire sentence flow. In each transformer layer, computations using query, key, and value are sequentially performed one by one from the first layer to the last layer. Vectors are produced as a result of computations with query, key, and value, and using these values called "Hidden state" in the last layer, the probability of each word suitable for the next location can be extracted. The extraction process is as shown in the figure below. For the Hidden state in the last layer, LayerNorm (Layer Normalization), matrix multiplication with embedding table values, and Softmax are performed.

![img](/public/contents/posts/en/2022/02/hyperclova-part-2/semantic02.png)

Once the probability of each word is extracted, the sentence generation and semantic search processes are performed slightly differently. The sentence generation step selects a word through a process called sampling using these probabilities. It can simply choose the word with the highest probability among many words, or randomly select from a few words with high probability by penalizing or giving random characteristics so that existing words are not generated. On the other hand, a semantic search must gather the probabilities of candidate words at each location among the extracted probabilities. Using these probabilities, it can infer which candidate word or sentence is most relevant to the basic sentence.

What was essential to provide a semantic search was adding this probability extraction part in the FasterTransformer framework. We modified the framework so that FasterTransformer can deliver the results of the probability calculation to service logic and implemented service logic so that it can proceed with the score calculation required by semantic search using these probabilities.

As we covered in the Early Stop section, when calculating probabilities, GPT computations are largely divided into a summarization stage, in which input sentences and words are processed in batches simultaneously, and a generation stage, in which the next words are generated one by one. The generation step is the process of extracting probabilities using keys and values in the previous word location and generating words one by one.

At first, we wrote the code so that probabilities were extracted through the computations in the generation step and then sent to service logic for a search score calculation. However, this method had a problem. In the generation step, serialized computations must be performed to extract the probability of each word, and this inevitably leads to greater latency.

![img](/public/contents/posts/en/2022/02/hyperclova-part-2/semantic03.png)

So we thought about how we could perform probability extractions faster using the characteristics of GPT. Earlier, I explained that probabilities of input words can be computed at once in the summarization step of GPT. When running a semantic search for basic and candidate sentences, both are input sentences and the probability at the location in the candidate sentence can be calculated simultaneously, as shown in the figure below.

![img](/public/contents/posts/en/2022/02/hyperclova-part-2/semantic04.png)

So by extracting probabilities all at once in the summarization step, we were able to reduce the latency more than 10 times by using specific input sentences and candidate sentences for semantic searches. If we understood the GPT computations from the start, we could have designed the process like this in the first place. Unfortunately, we didn't think of this method at first, and we were able to come up with this optimization method only after we understood the structure of GPT more deeply.

## P-tuning
As mentioned in Part 1, GPT-3 models, such as HyperCLOVA, are large in size and have a large amount of computations, so they have greater inference latency than other AI models. Since low latency is important for a better service experience, CLOVA is taking various steps to reduce the latency. There are ways to reduce latency. We can optimize computations to shorten the inference time or raise model performance to use a smaller model. We chose the latter and decided to support P-tuning. P-tuning is a process of training prompts tailored to specific tasks based on pre-trained models. Here, a prompt is a selected text inserted into the input example, and it induces sentence generation that is more suitable for the specific task.

![img](/public/contents/posts/en/2022/02/hyperclova-part-2/ptuning01.png)

Compared to the fine tuning that we are familiar with, P-tuning is easier to understand. Suppose that we are planning various AI services for different tasks. In general, rather than training models suitable for each task from scratch, we use pre-trained models by tuning them to suit the situation. In this case, fine tuning is also called model tuning because model weight changes by training the pre-trained model for a relatively short time using a dataset suitable for each task. On the other hand, in P-tuning, the weight of the pre-trained model is not changed, and only the weight corresponding to the input prompt is trained. In general, prompt weight is much smaller than model weight, so P-tuning has a lower training burden and higher scalability than fine tuning. You can see these characteristics of P-tuning more clearly in large models such as GPT-3. The figure above shows the difference between model tuning and P-tuning.

To make the most of these advantages, the CLOVA Model Team planned to provide services with P-tuning and had to make preparations to support the feature in the serving environment. What things did we have to consider to support P-tuning in the serving environment instead of a model training environment? Since we had new inputs called "prompts," we needed to have them processed by the model. In order for the model to process these new inputs, we need to make sure the prompts are recognized by the model, and the weights corresponding to the recognized prompts are successfully sent to the model. First, let's take a closer look at the process of transferring input sentences to the transformer.

![img](/public/contents/posts/en/2022/02/hyperclova-part-2/ptuning02.png)

As shown in the figure above, the user sends strings when sending requests to HyperCLOVA, but since the transformer can only process vectors, we need a process to convert strings to vectors. As a result, input sentences are first changed to token IDs through the tokenizer, and token IDs are changed to vectors through the embedding table. In this process, token IDs act as a table offset. So by passing through the tokenizer and embedding table, inputs are converted to vectors that the transformer can recognize.

Then, how can we deliver prompt weight to the transformer for P-tuning? First, like with other string inputs, you should specify the string corresponding to the prompt and register it with the tokenizer so that it can be converted to a token ID. The Special Token function of the tokenizer allows you to register a string corresponding to the prompt at the end of the tokenizer. One thing of note is that the newly registered special token should be carefully selected so that it does not affect the existing tokens.

![img](/public/contents/posts/en/2022/02/hyperclova-part-2/ptuning03.png)

The figure above shows the tokenizer tokens and token IDs corresponding to the tokens. Let's assume that we are registering a prompt in the tokenizer. What happens if we set the token for this prompt as "hello"?

![img](/public/contents/posts/en/2022/02/hyperclova-part-2/ptuning04.png)

As shown in the figure, the token corresponding to "hello" is already registered as ID 2 in the tokenizer. However, if we register a new "hello" token for the prompt as a special token, "hello" will be given the last ID of tokenizer, 10001, and this makes it impossible to distinguish between the existing "hello" token and the prompt "hello" token. To prevent this issue, we set a special token(<|PROMPT|>) that would not overlap existing tokens and used it as a token for the prompt.

![img](/public/contents/posts/en/2022/02/hyperclova-part-2/ptuning05.png)

Then, how can we transfer the weight corresponding to the prompt to the transformer? We used two methods to do this. First, we registered the vector corresponding to the token ID in the embedding table. Second, we transferred the vector directly to the transformer at inference time.

#### 1) First approach: Modifying the embedding table
![img](/public/contents/posts/en/2022/02/hyperclova-part-2/ptuning06.png)

The figure above shows how to modify the embedding table and deliver the prompt weight to the transformer. Earlier, I mentioned that the token ID that comes out as a result of tokenizer computation is used as the embedding table's offset. In other words, it means that if we add weight to the embedding table at this location when we have a token ID, it will work properly as expected. One thing to note here is that the prompt token ID added in this way should not be output as a result of transformer computation. However, due to the transformer's computational structure, there is a possibility that the token ID corresponding to the prompt will be output because computation with the embedding table is performed when the next token ID is selected. To prevent this, we set a variable called sampling size so that token can only be selected within that range.

There were two advantages to this method. First is the convenience in implementation. FasterTransformer is based on CUDA, so it takes a long time to modify and verify the code, but we can quickly implement the embedding table in Python because it only needs to modify the weight that exists as a file. Second is the cache effect. The weight belonging to the embedding table resides in the GPU memory, and the CUDA core has the fastest access to it. Since we registered the weight corresponding to the prompt at the end of the embedding table, we allowed a quick indexing without having to deliver the same prompt value again.

However, there was an important downside to this method: limited scalability. The method may be effective when there are few services using P-tuning, but what if we have more services and many prompts to register? You will have to continue registering tokens and weights corresponding to the prompts in the tokenizer and the embedding table. In general, it takes only a short time in the tokenizer, but the more special tokens there are, the more rapidly the performance decreases. In addition, the embedding table cannot be expanded indefinitely because it is inside the relatively small GPU memory. For this reason, we chose to deliver the prompt weight directly to the transformer without going through the embedding table.

#### 2) Second approach: Direct delivery at inference time
![img](/public/contents/posts/en/2022/02/hyperclova-part-2/ptuning07.png)

The figure above shows how to deliver prompt information directly at inference time. Unlike the first approach, we allowed the tokenizer to recognize all prompts with one token ID. This certainly helped us prevent performance drop in the tokenizer, but we needed to find a way to distinguish prompts. So we chose to send prompt ID along with the existing token ID to FasterTransformer, and delivered the weight of the prompt that can be indexed with the prompt ID to FasterTransformer together. In this way, FasterTransformer can perform indexing in the embedding table if an ID (1 - 10000) is within the token range trained from the token ID information, and in prompt weight using the prompt ID (10001) when the token ID is out of range (10001).

As a result, using the second approach, we were able to service P-tuning requests within a response time similar to that of normal requests.



Part 2: Implementing service features

Introduction

This is the second part of the three-part series on HyperCLOVA. In this post, I'll talk about how we implemented service features using HyperCLOVA. First, let me introduce Early Stop, a feature we implemented to deliver faster responses to users with NAVER's HyperCLOVA-based service patterns, such as CLOVA AI Speaker. Second, I'll explain how we were able to provide Semantic Search, a feature that analyzes the relevance between documents using a sentence generation model called the HyperCLOVA Model. And finally, I'll talk about P-tuning, a feature we implemented to achieve higher performance with the same model size in the serving environment. Here's the table of contents:

Early Stop

Semantic Search

P-tuning

Early Stop

The most time-consuming step in a GPT-3 model such as HyperCLOVA is sentence generation. Due to the structural characteristics, transformer computation consists of two steps: a summarization step for processing input sentences and a generation step for generating sentences. In the summarization step, token IDs corresponding to input sentences are processed in batches simultaneously, while in the generation step, tokens are sequentially generated one by one. Therefore, the sentence generation step is responsible for the largest portion of the total computation time and reducing sentence length will directly reduce the response time. Then, how can you reduce sentence length in this step?

img

The figure above is an example where CLOVA AI Speaker requests HyperCLOVA to generate sentences. Based on the previous question of the conversation between CLOVA and the user in the upper-left corner, CLOVA AI Speaker asks HyperCLOVA to generate a sentence. At this time, the length of the sentence to be generated is also indicated as the number of tokens. HyperCLOVA generates tokens based on the requested length, converts them into sentences and then sends a response to CLOVA AI Speaker.

At this point, the requesting party has no idea how much HyperCLOVA will generate until it actually receives an answer. Therefore, a request allows a sufficient number of tokens for sentence generation, and only the part that user needs can be used as a response. In the example, the user only needs the sentence that starts with CLOVA: Thus, the sentence that starts with User: will be cut off, and only the first sentence will be used for the answer.

If HyperCLOVA generates and returns only the sentence that the user needs when generating sentences as shown in the figure below, it can reduce the sentence generation time, which accounts for the largest portion of the total latency, and also greatly reduce the response time. This feature is called Early Stop.

img

However, for HyperCLOVA with FasterTransformer as a transformer framework, we could not directly implement the Early Stop feature. In order to implement Early Stop, we needed a process to convert the generated token ID into a string every time to check if it matched the string for termination. This is because FasterTransformer, which is provided in the form of a shared library after compilation, can only be used with a set interface. It can only receive the results after the token IDs are generated based on the requested length.

To address this issue, we added a tokenizer for Early Stop in FasterTransformer and an interface to handle it to allow the generated token ID to be changed to a string every time.

img

What we needed was a function to convert token IDs into strings, so we only implemented the decoder part of the tokenizer. In this case, the decoder part can be implemented using a simple hash map, because the information mapped with the Unicode corresponding to each token ID can be moved to the hash map as-is. The figure above shows the process of converting the token ID \[3, 2, 10000] to the "User" string using the decoder of the tokenizer. By indexing the vocab mapped to each token ID and combining the Unicode found, the token ID can be converted to a string.

img

The figure above shows a graph where a log was kept and analyzed for the length of the requested sentence and the length of the sentence actually generated after the Early Stop feature was implemented. The blue line represents the requested length, and the green line represents the length of the generated sentence. Here you can see that the requested length was much longer than the length of the generated sentence in the service. In extreme cases, the latter was more than 10 times shorter than the former. In this case, it was also confirmed that the actual response time was reduced by nearly 10 times.

Semantic Search

GPT-3 models, such as HyperCLOVA, are essentially very suitable for processing sentence generation tasks. However, if you look closely at the principles of sentence generation, you can see that it can also be used for other purposes. In this section, we will summarize how HyperCLOVA can handle the task of semantic search and how we can set up the computational process.

What is Semantic Search? Semantic search is the process of inferring the context and relevance of input sentences or words and scoring them. Semantic search can be used to build a recommendation system using the relevance between sentences and words, and can also be used to evaluate if the trained model selects words of appropriate quality.

As shown in the example below, the semantic search feature calculates the relevance between the basic sentences and the candidate sentences as a score when they are added as input sentences. The higher the relevance between the basic sentences and the candidate sentences, the lower the score. In the example below, it was inferred that among the sentences in the candidate group, "doenjang-jjigae" has the highest relevance with the basic sentence.

img

Now let's compare the process of generating sentences in HyperCLOVA with the process of a semantic search.

First, let's look at how these two processes are performed. GPT models, such as HyperCLOVA, basically compute using multiple transformer layers. These transformer layers use the concepts of query, key, and value. Query is related to the token currently given, and key and value are related to the entire sentence flow. In each transformer layer, computations using query, key, and value are sequentially performed one by one from the first layer to the last layer. Vectors are produced as a result of computations with query, key, and value, and using these values called "Hidden state" in the last layer, the probability of each word suitable for the next location can be extracted. The extraction process is as shown in the figure below. For the Hidden state in the last layer, LayerNorm (Layer Normalization), matrix multiplication with embedding table values, and Softmax are performed.

img

Once the probability of each word is extracted, the sentence generation and semantic search processes are performed slightly differently. The sentence generation step selects a word through a process called sampling using these probabilities. It can simply choose the word with the highest probability among many words, or randomly select from a few words with high probability by penalizing or giving random characteristics so that existing words are not generated. On the other hand, a semantic search must gather the probabilities of candidate words at each location among the extracted probabilities. Using these probabilities, it can infer which candidate word or sentence is most relevant to the basic sentence.

What was essential to provide a semantic search was adding this probability extraction part in the FasterTransformer framework. We modified the framework so that FasterTransformer can deliver the results of the probability calculation to service logic and implemented service logic so that it can proceed with the score calculation required by semantic search using these probabilities.

As we covered in the Early Stop section, when calculating probabilities, GPT computations are largely divided into a summarization stage, in which input sentences and words are processed in batches simultaneously, and a generation stage, in which the next words are generated one by one. The generation step is the process of extracting probabilities using keys and values in the previous word location and generating words one by one.

At first, we wrote the code so that probabilities were extracted through the computations in the generation step and then sent to service logic for a search score calculation. However, this method had a problem. In the generation step, serialized computations must be performed to extract the probability of each word, and this inevitably leads to greater latency.

img

So we thought about how we could perform probability extractions faster using the characteristics of GPT. Earlier, I explained that probabilities of input words can be computed at once in the summarization step of GPT. When running a semantic search for basic and candidate sentences, both are input sentences and the probability at the location in the candidate sentence can be calculated simultaneously, as shown in the figure below.

img

So by extracting probabilities all at once in the summarization step, we were able to reduce the latency more than 10 times by using specific input sentences and candidate sentences for semantic searches. If we understood the GPT computations from the start, we could have designed the process like this in the first place. Unfortunately, we didn't think of this method at first, and we were able to come up with this optimization method only after we understood the structure of GPT more deeply.

P-tuning

As mentioned in Part 1, GPT-3 models, such as HyperCLOVA, are large in size and have a large amount of computations, so they have greater inference latency than other AI models. Since low latency is important for a better service experience, CLOVA is taking various steps to reduce the latency. There are ways to reduce latency. We can optimize computations to shorten the inference time or raise model performance to use a smaller model. We chose the latter and decided to support P-tuning. P-tuning is a process of training prompts tailored to specific tasks based on pre-trained models. Here, a prompt is a selected text inserted into the input example, and it induces sentence generation that is more suitable for the specific task.

img

Compared to the fine tuning that we are familiar with, P-tuning is easier to understand. Suppose that we are planning various AI services for different tasks. In general, rather than training models suitable for each task from scratch, we use pre-trained models by tuning them to suit the situation. In this case, fine tuning is also called model tuning because model weight changes by training the pre-trained model for a relatively short time using a dataset suitable for each task. On the other hand, in P-tuning, the weight of the pre-trained model is not changed, and only the weight corresponding to the input prompt is trained. In general, prompt weight is much smaller than model weight, so P-tuning has a lower training burden and higher scalability than fine tuning. You can see these characteristics of P-tuning more clearly in large models such as GPT-3. The figure above shows the difference between model tuning and P-tuning.

To make the most of these advantages, the CLOVA Model Team planned to provide services with P-tuning and had to make preparations to support the feature in the serving environment. What things did we have to consider to support P-tuning in the serving environment instead of a model training environment? Since we had new inputs called "prompts," we needed to have them processed by the model. In order for the model to process these new inputs, we need to make sure the prompts are recognized by the model, and the weights corresponding to the recognized prompts are successfully sent to the model. First, let's take a closer look at the process of transferring input sentences to the transformer.

img

As shown in the figure above, the user sends strings when sending requests to HyperCLOVA, but since the transformer can only process vectors, we need a process to convert strings to vectors. As a result, input sentences are first changed to token IDs through the tokenizer, and token IDs are changed to vectors through the embedding table. In this process, token IDs act as a table offset. So by passing through the tokenizer and embedding table, inputs are converted to vectors that the transformer can recognize.

Then, how can we deliver prompt weight to the transformer for P-tuning? First, like with other string inputs, you should specify the string corresponding to the prompt and register it with the tokenizer so that it can be converted to a token ID. The Special Token function of the tokenizer allows you to register a string corresponding to the prompt at the end of the tokenizer. One thing of note is that the newly registered special token should be carefully selected so that it does not affect the existing tokens.

img

The figure above shows the tokenizer tokens and token IDs corresponding to the tokens. Let's assume that we are registering a prompt in the tokenizer. What happens if we set the token for this prompt as "hello"?

img

As shown in the figure, the token corresponding to "hello" is already registered as ID 2 in the tokenizer. However, if we register a new "hello" token for the prompt as a special token, "hello" will be given the last ID of tokenizer, 10001, and this makes it impossible to distinguish between the existing "hello" token and the prompt "hello" token. To prevent this issue, we set a special token(<|PROMPT|>) that would not overlap existing tokens and used it as a token for the prompt.

img

Then, how can we transfer the weight corresponding to the prompt to the transformer? We used two methods to do this. First, we registered the vector corresponding to the token ID in the embedding table. Second, we transferred the vector directly to the transformer at inference time.

1\) First approach: Modifying the embedding table

img

The figure above shows how to modify the embedding table and deliver the prompt weight to the transformer. Earlier, I mentioned that the token ID that comes out as a result of tokenizer computation is used as the embedding table's offset. In other words, it means that if we add weight to the embedding table at this location when we have a token ID, it will work properly as expected. One thing to note here is that the prompt token ID added in this way should not be output as a result of transformer computation. However, due to the transformer's computational structure, there is a possibility that the token ID corresponding to the prompt will be output because computation with the embedding table is performed when the next token ID is selected. To prevent this, we set a variable called sampling size so that token can only be selected within that range.

There were two advantages to this method. First is the convenience in implementation. FasterTransformer is based on CUDA, so it takes a long time to modify and verify the code, but we can quickly implement the embedding table in Python because it only needs to modify the weight that exists as a file. Second is the cache effect. The weight belonging to the embedding table resides in the GPU memory, and the CUDA core has the fastest access to it. Since we registered the weight corresponding to the prompt at the end of the embedding table, we allowed a quick indexing without having to deliver the same prompt value again.

However, there was an important downside to this method: limited scalability. The method may be effective when there are few services using P-tuning, but what if we have more services and many prompts to register? You will have to continue registering tokens and weights corresponding to the prompts in the tokenizer and the embedding table. In general, it takes only a short time in the tokenizer, but the more special tokens there are, the more rapidly the performance decreases. In addition, the embedding table cannot be expanded indefinitely because it is inside the relatively small GPU memory. For this reason, we chose to deliver the prompt weight directly to the transformer without going through the embedding table.

2\) Second approach: Direct delivery at inference time

img

The figure above shows how to deliver prompt information directly at inference time. Unlike the first approach, we allowed the tokenizer to recognize all prompts with one token ID. This certainly helped us prevent performance drop in the tokenizer, but we needed to find a way to distinguish prompts. So we chose to send prompt ID along with the existing token ID to FasterTransformer, and delivered the weight of the prompt that can be indexed with the prompt ID to FasterTransformer together. In this way, FasterTransformer can perform indexing in the embedding table if an ID (1 - 10000) is within the token range trained from the token ID information, and in prompt weight using the prompt ID (10001) when the token ID is out of range (10001).

As a result, using the second approach, we were able to service P-tuning requests within a response time similar to that of normal requests.


Extending the features of HyperCLOVA API

Natural Language Processing (NLP) was one of the AI technology fields that has attracted global attention this year, and Generative Pre-trained Transformer 3 (GPT-3) was at the center of it. Technologies such as Microsoft's Copilot, formerly considered to be only possible in our wildest imaginations are becoming a reality through GPT-3. HyperCLOVA is the world's first massive AI model based on GPT-3 that is trained in the Korean language through CLOVA, and is currently being used in many NAVER AI services, such as CLOVA Speaker and Chatbot. We've made various attempts to apply the HyperCLOVA model to actual services, while also making those services more reliable and responsive. These are the five steps that we've taken to provide the services:

1. Selecting a framework
2. Creating a serving environment
3. Implementing service features
4. Optimization points
5. Automating operations

From among these points, I will be covering steps 1, 3, and 4 in more detail in a series of blog posts, including this one.

# Part 1: Selecting a framework
## Selecting a framework to speed up the model serving process
One of the important parts of serving the HyperCLOVA model is the speed of services. The HyperCLOVA model has been growing in size for better quality and performance, thus increasing both the memory consumption and computation volume. The increased data transactions and computation volume between GPUs that resulted from the process of serving large models led to quite long latency. In fact, applying super-large models into services required a two to three times increase in speed depending on the GPU type.

We thought that optimizing computation at the low level should come first in order to increase the serving speed of the HyperCLOVA model. The serving process is largely divided into two parts as shown below: an internal transformer framework and an external service logic. GPT models, such as HyperCLOVA, carry out computation based on the transformer, and a framework is needed to create text through this process. After the text is created in the transformer framework, the result is delivered to users through the service logic. Since most of the time to respond to users are consumed in the transformer framework, reducing computation time in the framework was fundamentally important for increasing the speed.

![img](/public/contents/posts/en/2022/01/hyperclova-part-1/01.png)

The traditional way of serving HyperCLOVA has been based on NVIDIA's Megatron-LM used for training a model through the transformer framework. Although it had the advantage of implementing desirable services relatively easily by leveraging frameworks the way it was used in training, the framework was not the optimized implementation for inference in which latency matters. In fact, inference computation profiling results via Nsight Systems says utilization is substantially low due to the poor use of GPU resources. In addition, we found that there had been plenty of overhead during the data collection process between multiple GPUs, which appeared to be ncclKernel in the following profiling results.

![img](/public/contents/posts/en/2022/01/hyperclova-part-1/02.png)

## Adopting FasterTransformer
In practice, serving does not require all the steps that are required while training; all you need is an inference. Therefore, we wanted to apply the framework optimized for inference services, instead of the framework used in training. We decided to replace the traditional framework with NVIDIA's GPT inference framework: FasterTransformer. FasterTransformer is written with C++ and CUDA native kernels instead of PyTorch, and is applied with CUDA kernel-level optimization techniques, such as kernel fusion. With such an implementation method, we hoped to see more efficiency while using the GPU and less latency in the serving process.

As a result of replacing the existing framework with FasterTransformer, we've seen impressive success in terms of speed. Although there is a difference depending on the size of the model, as you can see in a graph comparing latency and the model sizes below, it was possible to increase the speed about two to three times compared to the framework used in training. With such an improvement in speed, we were able to reach the target speed applicable to the service, especially in large models. Given that the HyperCLOVA model is growing in size to improve the quality of sentence generation, it was an impressive achievement. The results of profiling with Nsight Systems shows which part has been improved in the FasterTransformer compared to the existing framework. As the profiling results shows, GPU resources were used more efficiently with much higher utilization compared to the traditional framework. Moreover, there were relatively fewer overheads in the data collection process (NCCL kernel) between GPUs, which was a problem in the existing Megatron-LM framework.

![img](/public/contents/posts/en/2022/01/hyperclova-part-1/03.png)

## Problems and solutions occurred after introducing FasterTransformer
There were problems to be resolved to immediately apply FasterTransformer to services.

First, we had to modify interfaces between the framework and the service logic. Most of the features in the traditional framework were implemented via PyTorch and API modification in the service logic was much easier. However, since most of the features in FasterTransformer are made in the form of the shared library and provided through APIs only accessible with the computation results in the service logic, we had to modify interfaces between the service logic and the framework.

Next, we found consistency issues in v3.0 and v4.0 of FasterTransformer that we used, making it hard to apply them directly to services. Here, consistency issues refer to when incorrect computation leads to incorrect words or compromised words in quality. As shown in the images below, an issue occurred where incorrect words were outputted instead of correct sentences.

![img](/public/contents/posts/en/2022/01/hyperclova-part-1/04.png)

There were several factors that brought about consistency issues. A few examples include the checkpoint modification issue, the FP16 computation issue, and the bugs in the GPU kernel.

### Resolving the checkpoint modification issue
The checkpoint modification issue also relates to the framework modification process described above. FasterTransformer only offers inference computation, not a model training process. We had to use checkpoints (weight and bias included) in the Megatron-LM used to train the HyperCLOVA model, and also had to modify the Megatron-LM checkpoint for FasterTransformer. We modified the checkpoints by using code provided with FasterTransformer and created sentences with these results, but still words with poor quality were created.

At first, we didn't know exactly where the issue came from, but we finally found out how to resolve this as we saw the existing NVIDIA GPT-2 model ran without any issues, but the HyperCLOVA model trained through NAVER had issues. In the case of the HyperCLOVA model, the indexing of the stored checkpoint was different from the conventional method since the checkpoints were used in a different way from the NVIDIA GPT-2 model. We had to consider this during the checkpoint modification process for FasterTransformer, but we didn't. This issue was resolved by considering the computation process in Megatron-LM in reverse and revising the modification processes.

### Deep dive into GPT-3 with FP16
Next, let's look at issues caused by the Floating-Point (FP) 16 type computation. The HyperCLOVA model was conducted with the FP16 type, instead of FP32, to speed up the serving process. In the case of the HyperCLOVA model, computation with the FP16 type was three to four times faster than with the FP32 type. Computation with FP16 was very important for reducing the time for training and inference, but FP16 caused some issues because the range of expression and the degree of precision were reduced compared to FP32.

First of all, when the HyperCLOVA model was large, computing with FP32 was okay, but computing with FP16 caused quality loss. This is because FP16 caused overflow during computation, resulting in the data values being inf or Not a Number (NaN). In the case of FP32, the maximum value is 3.40*e38 with eight exponent bits. However, the maximum value of FP16 is 65504 with five exponent bits, which is less than FP32, so FP16 is relatively vulnerable to overflow issues.

The figure below shows the simple description of GPT computation sequentially processed in multiple transformer layers and then General Matrix-to-matrix Multiply (GEMM), or a matrix multiplication computation, continues in each layer. In this process, the GEMM computation value in the first layer is relatively small, but this value gradually increases as it goes to the rear layers. Larger models mean more layers and larger data values in each layer, thus increasing the probability of overflow as larger values are made in rear layers. If overflow occurs and the computation values come with inf values, only incorrect words are generated continuously after a certain point.

![img](/public/contents/posts/en/2022/01/hyperclova-part-1/05.png)

There aren't many solutions for this situation. If the inf value can be detected, you can change the value to 65504, the maximum value of FP16, instead of the inf value. Another way is to decrease the overflow probability by changing the GEMM kernel data type and computation type, which are likely to cause overflows, into FP32. Since the former method used NVIDIA's cuBLAS GEMM kernel, it is impossible to change the value. So we used the latter method to ensure the GEMM kernel computation value that causes overflows was stored as a FP32 data type. This configuration can prevent overflows from occurring since the values in between are stored in FP32.

Compared to FP32, FP16 has a very small number of exponent bits that were related to previous overflow issues, as well as mantissa bits called fraction bits. A mantissa bit is a value corresponding to the mantissa of a floating-point data, so the smaller the number, the smaller the number of decimal digits that can be expressed.

![img](/public/contents/posts/en/2022/01/hyperclova-part-1/06.png)

Such characteristics of the FP16 data type can lead to "swamping" during GPT computation. Swamping is a situation in which a small value is lost when adding a large value along with a small value, as if the small value is drowned in the swamp of the larger value. These are the reasons for the loss of value.

Addition of floating-point types starts by matching the exponent bits of two values. At this time, the small value's exponent bit must be set to the exponent bit of a large value. For the small value, you need to move the mantissa bit appropriately to keep the value the same before and after changing the exponent bits. Then, the values representing some of the mantissa located to the right of the mantissa bit are lost. The figure below is an example of such a situation. You can see that 0.00098 is now being added to 3.75. The result is 3.75, not 3.75098. In the process of matching exponent bits of small and large values, the value of 0.00098 was not reflected to the addition because of the small number of mantissa bits in FP16. Since the number of mantissa bits of the FP32 data type is more than double that of the mantissa bits of the FP16 data type, the smaller proportion of data values were lost in this process, but significant values are likely to be lost in the FP16 type. The data loss issue caused by swamping is more likely to occur when the gap between the values for addition increases. For this reason, if the weight distribution is uneven in the training process, the swamping-induced issue can be intensified. Moreover, considering the entire computation process of the GPT model with multiple layers, the computation result of the previous layer is used in the rear layer, thus errors caused by swamping accumulates as it goes to the rear layers.

![img](/public/contents/posts/en/2022/01/hyperclova-part-1/07.png)

Because this swamping issue occurred in the FP16 computation process, two problems can occur in the GPT-3 model, such as HyperCLOVA. First of all, due to the data loss described above, there is a possibility that different results from the existing FP32 computation or damaged text can be produced. Next, in a situation where you expect to generate the same words from the same inputs through a specific option, different results may come out depending on whether the transformer computation is carried out in a single batch or a multi-batch. Here, the specific option is the greedy option. When using the greedy option, it is expected that the same sentences will lead to the same sentences or words regardless of the random values, but different values can be created even with the greedy option because of the swamping effect that comes from the FP16 type computation.

To resolve this issue, we used FP16 for the cuBLAS GEMM kernel data type but set the compute type to FP32. Even if the input data type is FP16, the FP32 compute type setting ensures that the intermediate addition processes are stored in the FP32 data type, and it secures mantissa bits sufficiently, thus mitigating the degree of data lost. As a result, we've also seen that the frequency of different results in a single batch and a multi-batch due to the swamping effect was reduced with the greedy option.

## Results
FasterTransformer was reliable for using with our services after resolving the problems mentioned above, along with additional bugs such as incorrect indexes in the GPU kernel, race conditions, and memory leaks. As a result, using FasterTransformer for services instead of Megatron-LM ensures that we could speed up our services about two to three times depending on the model size or GPU environment.

While resolving the consistency issues of FasterTransformer, we fixed the problem of incorrect results in the earlier version of FasterTransformer, and the issue that caused a 75% drop in relative accuracy compared to Megatron-LM when using certain models. This allows you to expect Megatron-LM level text quality.

![img](/public/contents/posts/en/2022/01/hyperclova-part-1/08.png)

The results of resolving the consistency issue were applied to the FasterTransformer v5.0 beta via open source contribution. Please refer to the DEVIEW 2021 presentation([https://deview.kr/2021/sessions/416](https://deview.kr/2021/sessions/416)) for more details.


Natural Language Processing (NLP) was one of the AI technology fields that has attracted global attention this year, and Generative Pre-trained Transformer 3 (GPT-3) was at the center of it. Technologies such as Microsoft's Copilot, formerly considered to be only possible in our wildest imaginations are becoming a reality through GPT-3. HyperCLOVA is the world's first massive AI model based on GPT-3 that is trained in the Korean language through CLOVA, and is currently being used in many NAVER AI services, such as CLOVA Speaker and Chatbot. We've made various attempts to apply the HyperCLOVA model to actual services, while also making those services more reliable and responsive. These are the five steps that we've taken to provide the services:

Selecting a framework

Creating a serving environment

Implementing service features

Optimization points

Automating operations

From among these points, I will be covering steps 1, 3, and 4 in more detail in a series of blog posts, including this one.

Part 1: Selecting a framework

Selecting a framework to speed up the model serving process

One of the important parts of serving the HyperCLOVA model is the speed of services. The HyperCLOVA model has been growing in size for better quality and performance, thus increasing both the memory consumption and computation volume. The increased data transactions and computation volume between GPUs that resulted from the process of serving large models led to quite long latency. In fact, applying super-large models into services required a two to three times increase in speed depending on the GPU type.

We thought that optimizing computation at the low level should come first in order to increase the serving speed of the HyperCLOVA model. The serving process is largely divided into two parts as shown below: an internal transformer framework and an external service logic. GPT models, such as HyperCLOVA, carry out computation based on the transformer, and a framework is needed to create text through this process. After the text is created in the transformer framework, the result is delivered to users through the service logic. Since most of the time to respond to users are consumed in the transformer framework, reducing computation time in the framework was fundamentally important for increasing the speed.

img

The traditional way of serving HyperCLOVA has been based on NVIDIA's Megatron-LM used for training a model through the transformer framework. Although it had the advantage of implementing desirable services relatively easily by leveraging frameworks the way it was used in training, the framework was not the optimized implementation for inference in which latency matters. In fact, inference computation profiling results via Nsight Systems says utilization is substantially low due to the poor use of GPU resources. In addition, we found that there had been plenty of overhead during the data collection process between multiple GPUs, which appeared to be ncclKernel in the following profiling results.

img

Adopting FasterTransformer

In practice, serving does not require all the steps that are required while training; all you need is an inference. Therefore, we wanted to apply the framework optimized for inference services, instead of the framework used in training. We decided to replace the traditional framework with NVIDIA's GPT inference framework: FasterTransformer. FasterTransformer is written with C++ and CUDA native kernels instead of PyTorch, and is applied with CUDA kernel-level optimization techniques, such as kernel fusion. With such an implementation method, we hoped to see more efficiency while using the GPU and less latency in the serving process.

As a result of replacing the existing framework with FasterTransformer, we've seen impressive success in terms of speed. Although there is a difference depending on the size of the model, as you can see in a graph comparing latency and the model sizes below, it was possible to increase the speed about two to three times compared to the framework used in training. With such an improvement in speed, we were able to reach the target speed applicable to the service, especially in large models. Given that the HyperCLOVA model is growing in size to improve the quality of sentence generation, it was an impressive achievement. The results of profiling with Nsight Systems shows which part has been improved in the FasterTransformer compared to the existing framework. As the profiling results shows, GPU resources were used more efficiently with much higher utilization compared to the traditional framework. Moreover, there were relatively fewer overheads in the data collection process (NCCL kernel) between GPUs, which was a problem in the existing Megatron-LM framework.

img

Problems and solutions occurred after introducing FasterTransformer

There were problems to be resolved to immediately apply FasterTransformer to services.

First, we had to modify interfaces between the framework and the service logic. Most of the features in the traditional framework were implemented via PyTorch and API modification in the service logic was much easier. However, since most of the features in FasterTransformer are made in the form of the shared library and provided through APIs only accessible with the computation results in the service logic, we had to modify interfaces between the service logic and the framework.

Next, we found consistency issues in v3.0 and v4.0 of FasterTransformer that we used, making it hard to apply them directly to services. Here, consistency issues refer to when incorrect computation leads to incorrect words or compromised words in quality. As shown in the images below, an issue occurred where incorrect words were outputted instead of correct sentences.

img

There were several factors that brought about consistency issues. A few examples include the checkpoint modification issue, the FP16 computation issue, and the bugs in the GPU kernel.

Resolving the checkpoint modification issue

The checkpoint modification issue also relates to the framework modification process described above. FasterTransformer only offers inference computation, not a model training process. We had to use checkpoints (weight and bias included) in the Megatron-LM used to train the HyperCLOVA model, and also had to modify the Megatron-LM checkpoint for FasterTransformer. We modified the checkpoints by using code provided with FasterTransformer and created sentences with these results, but still words with poor quality were created.

At first, we didn't know exactly where the issue came from, but we finally found out how to resolve this as we saw the existing NVIDIA GPT-2 model ran without any issues, but the HyperCLOVA model trained through NAVER had issues. In the case of the HyperCLOVA model, the indexing of the stored checkpoint was different from the conventional method since the checkpoints were used in a different way from the NVIDIA GPT-2 model. We had to consider this during the checkpoint modification process for FasterTransformer, but we didn't. This issue was resolved by considering the computation process in Megatron-LM in reverse and revising the modification processes.

Deep dive into GPT-3 with FP16

Next, let's look at issues caused by the Floating-Point (FP) 16 type computation. The HyperCLOVA model was conducted with the FP16 type, instead of FP32, to speed up the serving process. In the case of the HyperCLOVA model, computation with the FP16 type was three to four times faster than with the FP32 type. Computation with FP16 was very important for reducing the time for training and inference, but FP16 caused some issues because the range of expression and the degree of precision were reduced compared to FP32.

First of all, when the HyperCLOVA model was large, computing with FP32 was okay, but computing with FP16 caused quality loss. This is because FP16 caused overflow during computation, resulting in the data values being inf or Not a Number (NaN). In the case of FP32, the maximum value is 3.40\*e38 with eight exponent bits. However, the maximum value of FP16 is 65504 with five exponent bits, which is less than FP32, so FP16 is relatively vulnerable to overflow issues.

The figure below shows the simple description of GPT computation sequentially processed in multiple transformer layers and then General Matrix-to-matrix Multiply (GEMM), or a matrix multiplication computation, continues in each layer. In this process, the GEMM computation value in the first layer is relatively small, but this value gradually increases as it goes to the rear layers. Larger models mean more layers and larger data values in each layer, thus increasing the probability of overflow as larger values are made in rear layers. If overflow occurs and the computation values come with inf values, only incorrect words are generated continuously after a certain point.

img

There aren't many solutions for this situation. If the inf value can be detected, you can change the value to 65504, the maximum value of FP16, instead of the inf value. Another way is to decrease the overflow probability by changing the GEMM kernel data type and computation type, which are likely to cause overflows, into FP32. Since the former method used NVIDIA's cuBLAS GEMM kernel, it is impossible to change the value. So we used the latter method to ensure the GEMM kernel computation value that causes overflows was stored as a FP32 data type. This configuration can prevent overflows from occurring since the values in between are stored in FP32.

Compared to FP32, FP16 has a very small number of exponent bits that were related to previous overflow issues, as well as mantissa bits called fraction bits. A mantissa bit is a value corresponding to the mantissa of a floating-point data, so the smaller the number, the smaller the number of decimal digits that can be expressed.

img

Such characteristics of the FP16 data type can lead to "swamping" during GPT computation. Swamping is a situation in which a small value is lost when adding a large value along with a small value, as if the small value is drowned in the swamp of the larger value. These are the reasons for the loss of value.

Addition of floating-point types starts by matching the exponent bits of two values. At this time, the small value's exponent bit must be set to the exponent bit of a large value. For the small value, you need to move the mantissa bit appropriately to keep the value the same before and after changing the exponent bits. Then, the values representing some of the mantissa located to the right of the mantissa bit are lost. The figure below is an example of such a situation. You can see that 0.00098 is now being added to 3.75. The result is 3.75, not 3.75098. In the process of matching exponent bits of small and large values, the value of 0.00098 was not reflected to the addition because of the small number of mantissa bits in FP16. Since the number of mantissa bits of the FP32 data type is more than double that of the mantissa bits of the FP16 data type, the smaller proportion of data values were lost in this process, but significant values are likely to be lost in the FP16 type. The data loss issue caused by swamping is more likely to occur when the gap between the values for addition increases. For this reason, if the weight distribution is uneven in the training process, the swamping-induced issue can be intensified. Moreover, considering the entire computation process of the GPT model with multiple layers, the computation result of the previous layer is used in the rear layer, thus errors caused by swamping accumulates as it goes to the rear layers.

img

Because this swamping issue occurred in the FP16 computation process, two problems can occur in the GPT-3 model, such as HyperCLOVA. First of all, due to the data loss described above, there is a possibility that different results from the existing FP32 computation or damaged text can be produced. Next, in a situation where you expect to generate the same words from the same inputs through a specific option, different results may come out depending on whether the transformer computation is carried out in a single batch or a multi-batch. Here, the specific option is the greedy option. When using the greedy option, it is expected that the same sentences will lead to the same sentences or words regardless of the random values, but different values can be created even with the greedy option because of the swamping effect that comes from the FP16 type computation.

To resolve this issue, we used FP16 for the cuBLAS GEMM kernel data type but set the compute type to FP32. Even if the input data type is FP16, the FP32 compute type setting ensures that the intermediate addition processes are stored in the FP32 data type, and it secures mantissa bits sufficiently, thus mitigating the degree of data lost. As a result, we've also seen that the frequency of different results in a single batch and a multi-batch due to the swamping effect was reduced with the greedy option.

Results

FasterTransformer was reliable for using with our services after resolving the problems mentioned above, along with additional bugs such as incorrect indexes in the GPU kernel, race conditions, and memory leaks. As a result, using FasterTransformer for services instead of Megatron-LM ensures that we could speed up our services about two to three times depending on the model size or GPU environment.

While resolving the consistency issues of FasterTransformer, we fixed the problem of incorrect results in the earlier version of FasterTransformer, and the issue that caused a 75% drop in relative accuracy compared to Megatron-LM when using certain models. This allows you to expect Megatron-LM level text quality.

img

The results of resolving the consistency issue were applied to the FasterTransformer v5.0 beta via open source contribution. Please refer to the DEVIEW 2021 presentation(https\://deview\.kr/2021/sessions/416) for more details.


CLOVA Engineering blog

Featured Posts

COME GROW WITH US

Recent Posts

Top Tags

Meet CLOVA