What is inferencing a model?

Inferencing a model refers to the process of using a trained machine learning model to make predictions or generate outputs based on new input data. In the context of large language models (LLMs), inferencing typically involves taking a prompt or input text and generating a response or continuation based on the patterns and knowledge learned during training. The inferencing process can be computationally intensive, especially for large models, and often requires optimization techniques to reduce latency and improve throughput while maintaining the quality of the generated output.

Tokenization and how it works?

Tokens are the basic units of text that a language model processes. Tokenization is the process of breaking down input text into these smaller units, which can be words, subwords, or even individual characters. For example, the sentence “Hello world!” might be tokenized into [“Hello”, “world”, “!”].

The user and system prompts are tokenized using different techniques like Byte pair encoding (BPE), which breaks down words into subwords based on frequency, or WordPiece, which is similar but uses a different approach to determine subword boundaries.

The choice of tokenization method can affect the model’s performance and the quality of the generated output. For instance, BPE can help the model handle rare words by breaking them into more common subwords, while WordPiece can provide a more consistent tokenization for certain languages. The tokenized input is then converted into token vectors, which are numerical representations that the model can process during inferencing.

What is the playing field for the optimization paramters?

Following parameters are the key factors to consider when optimizing inference in LLMs:

Techniques for optimizing inference in LLMs

I would like to divide the techniques for optimizing inference in LLMs into three categories based on the stage of the model lifecycle:

Post training optimization techniques

• Pruning
• Distillation

While porting the model to production

• Model compression and Quantization
• Knowledge distillation

During inferencing the model using Systems Engineering

• Disaggregated decoding - prefill and decode (TTFT reduction)
  • As this two steps can be decoupled, we can start generating the output tokens while the prefill step is still processing the input prompt, which can significantly reduce the time taken to generate the first token and improve overall latency.
  • The prefill step involves processing the input prompt and generating the initial token representations, while the decode step involves generating the output tokens based on the prefilled representations. By decoupling these steps, we can reduce the time taken to generate the first token and improve overall latency.
• Continuous batching (TPOT decrease)
  • As soon as any request is processed, it is added to the batch and processed together with other requests, which can significantly reduce the time taken to process each request and increase throughput.
  • Goal is to utilize the GPU clock cycle as much as possible
• Page attention - OS paging, memory management, etc. (TPOT decrease)
  • Involves efficient memory management techniques to overcome memory fragmentation and reduce latency during inference. This can include techniques such as OS paging, which allows the model to efficiently manage memory by swapping data in and out of memory as needed, and optimizing memory access patterns to minimize latency.
• KV cache optimization - (TPOT decrease)
  • Involves optimizing the way the model stores and retrieves key-value pairs during inference, which can significantly reduce latency and improve throughput. This can be achieved through techniques such as caching frequently accessed key-value pairs, using more efficient data structures for storage, and optimizing the retrieval process to minimize overhead.
• Speculative decoding (TTFT reduction)
  • We use a smaller, faster model to generate multiple candidate tokens in parallel, and then use the larger, more accurate model to select the best candidate token. This can significantly reduce the time taken to generate the first token and improve overall latency.