Skip to main content

Types of FMs in Gen AI

We will try to understand the capabilities of the Generative AI by looking at the commonly used foundation models.

Large Language Models (LLM)​

LLMs can be based on variety of architectures. But today's LLMs are based on tranformer architecture. They are trained on vast amounts of text data from internet, books etc. They learn patterns and generate relationships between the words. Transformer based LLMs are powerful and can understand and generate human-like text.

To know how an LLM works, we need to start with tokens.

Tokens​

Tokens are the basic units of text that a model processes.

  • The given input text is passed to a component called Tokenizer
  • Tokenizer may then break the input text into words or characters or any other way based on the algorithm used
  • Then the Tokenizer returns the tokens to the LLM

For example, an input "A cat is reading" can be broken up into "A|cat|is|reading" or "A|c|a|t|i|s|r|e|a|d|i|n|g" or some other form based on the algorithm used in the transformer.

Embeddings and Vectors​

Embeddings are numerical representations of the tokens. We know that computers do not understand letters or languages. All they understand is numbers. So, each token is represented in a numerical form as a vector (a list of numbers) that captures its meaning and relationship with other words in the text.

For example, we know that dog and dogs are the same except that one is singular and the other is plural. The vectors generated for these words will be close to each other so that the model can understand that they are similar/related.

LLMs use these token, embeddings, and vectors to understand and generate text.

Diffusion models​

These are commonly used in image generation capabilities. Diffusion is a deep learning architecture that starts with a random data and gradually adds more and more meaningful data until they end up with a clear and coherent data.

Diffusion models are trained by

  • forward diffusion: A clear data (text/image) is given to the model. It adds noise gradually until data is fully lost and only noise is left.
  • reverse diffusion: The model is trained to reverse the process. It takes the noisy data and gradually removes the noise until it ends up with a clear data.

Multimodal Models​

Instead of just text or image, multimodal models can accept any type of data and generate different types of output data simultaneously. A multimodal model can take an image and some text as input and ouput a new image with a caption.

Modal means mode. Different modals or modes of data are text, image, video, and audio etc.

Multimodal models can be used for automating video captioning, creating graphics from text instructions, answering questions more intelligently by combining text and visual info, and even translating content while keeping relevant visuals.

The type of FMs we saw above are the most commonly used ones. There are several other generative AI models like Generative Adversial Networks (GANs), `Variational Autoencoders (VAEs) etc.

Optimizing model outputs​

It is not guaranteed that the output from the models is always correct. A model may return an output that's not related to the question asked, or the model may hallucinate. So, we need to optimize the foundation model to get a better answer. There are several ways to do it that vay in cost and complexity.

Prompt engineering​

Prompt engineering is a technique to improve the output of the model by carefully designing the input prompt. Prompts, in a way are instructions to the FMs and prompt engineering focuses on designing and developing prompts to enhance the outputs of FMs to our needs. It is the fastest and lowest cost option.

A prompt may contain some or all the elements below:

  • Instructions: Here we mention what the model should do and how to do it
  • Context: External information to guide the model
  • Input data: Input for which we want a response
  • Output indication: The output type or format like JSON etc

Following is an example prompt: You're a port. Write a 4‑line humorous poem about a cat trying to use a computer.

Fine tuning​

Although the FMs are pre-trained, fine tuning is a technique to improve the output of the model by training it on a smaller, task-specific dataset. A pre-trained model is trained on internet data. You can take that model and train it with your organizational data

There are two ways to fine-tune a model:

  • Instruction fine-tuning: It uses examples of how the model should respond to a specific instruction
  • Reinforcement learning from human feedback (RLHF): We provide feedback to the model's answers so that it can improve

Retrieval augmented generation​

Retrieval augmented generation (RAG) is a technique that combines the power of FMs with external knowledge sources. It may seem similar to fine-tuning but instead of training the FM on smaller, domain-specific data we supply domain-relevant documents or data as context to the model. The model produces response based on the supplied data.

For example, we supply the model with a 1000 page PDF and ask a question, the model gives response from the PDF.

RAG will not change the weights of the foundation model, whereas fine-tuning will change model weights.