@v0xium: LLM Inference Engineering: Embedding Models Explained 1. An Embedding Model (EM) converts a chunk of text, or any other…

LLM Inference Engineering: Embedding Models Explained 1. An Embedding Model (EM) converts a chunk of text, or any other modality, into a fixed-length vector that captures the semantic meaning/context of the input. 2. Once we have the representation vectors, we can compare the distances between them using simple vector math. That is why these models can be used to build agent memory, RAG applications, search, and recommendation systems. 3. There are two types of traffic profiles for EMs: - High-throughput backfills: Big bulk operations like indexing millions of documents, preparing data for LLM training, etc. - Low-latency lookups: User queries for search, retrieval, recommendation, etc. We need to be fast because each extra millisecond affects the UX. 4. Inference Engineering for EMs should start by asking a question: Which profile are we serving? For example, if you need to support both and have enough traffic, you can build separate systems for both types of usage. 5. Most EMs use one of these two types of architectures. Both of them are transformer-based: - BERT-style: Encoder-only NNs. - LLM-based: Modern language models, but repurposed to generate embeddings. 6. BERT-style models are still used for simple, latency-sensitive tasks like classification. But the bulk of usage is of LLM-based models. 7. They have their own tradeoffs between speed and quality. The size of the output vectors, or embedding dimensionality, can affect this. 8. Dimensionality does not mostly affect inference time, but it affects the storage, retrieval, and distance computation time of the EM system. 9. In most cases, vectors from one EM cannot be compared to a vector from another EM. 10. Inference gains can also come from quantization. FP8 for a larger embedding model offers improved performance with minimal quality loss. 11. To check the quality of an EM post-quantization, you can run the input through the original and quantized versions and check the cosine similarity of the two output vectors. A 99% similarity will indicate good quantization. (end) This is the collection of notes made while reading Inference Engineering Book by Philip Kiely. More posts will be there soon. Image Source: Choosing an Embedding Model, Pinecone.