# Foundational Models and Large Language Models (LLMs)

> Chapter 2 of the Courseiva GOOGLE-GEN-AI-LEADER curriculum — https://courseiva.com//learn/google-gen-ai-leader/foundational-models-and-large-language-models

**Official objective:** 1.2 — Describe foundational models and LLMs

## Introduction

How do you get a machine to not just understand words, but to create original paragraphs, poems, or even computer code? This is the core problem that foundational models and large language models (LLMs) solve. For someone studying for the Generative AI Leader exam, understanding the difference between these models is critical because it explains how the most powerful AI systems today are built, deployed, and adapted for specific business tasks.

## The Master Recipe Analogy

Ever wondered how a single master recipe can create hundreds of different dishes, from a simple cheese sandwich to a complex lasagne? That master recipe is like a foundational model in the world of AI. Foundational models are massive, general-purpose AI systems trained on a huge variety of data—text, code, images, audio—just like a master chef learning to handle every ingredient and technique in the kitchen. They don't start with a specific job in mind; they learn the underlying patterns of language, reasoning, and creativity, much like a chef learns the basics of taste, temperature, and texture.

Now, take a specific large language model (LLM), like the one powering a chatbot. This is like the chef using that master recipe to make a specific dish for a particular customer. The LLM is a foundational model that has been fine-tuned, or specially adjusted, to excel at understanding and generating human-like text. When you ask it to write an email or answer a question, it applies its vast knowledge of language patterns to produce a tailored response. It doesn't need to learn how to form a sentence from scratch; it already has that foundational skill, just as a chef doesn't need to learn how to boil water every time they start a new recipe. The foundational model is the brain, and the LLM is that brain applied to a focused task—conversation.

## Core explanation

To understand foundational models and large language models (LLMs), you first need to grasp a basic idea: what is a model? In the context of AI, a model is essentially a mathematical program that has learned patterns from data. Think of it as a very advanced pattern-matching machine. If you show it thousands of pictures of cats, it learns the pattern of what a cat looks like (fur, whiskers, pointy ears) and can then identify cats in new pictures.

A foundational model takes this concept and super-sizes it. Instead of learning just one pattern (like cat pictures), it is trained on an enormous, diverse collection of data from across the internet. This data includes text from books and articles, lines of code from software projects, images, and sometimes even audio. The goal is not to master one specific task but to learn the general rules of language, logic, and visual structure. The name 'foundational' means it serves as a base, or foundation, that can be used for numerous different applications.

A large language model (LLM) is a specific type of foundational model that has been trained almost exclusively on text data. The 'large' part refers to the sheer size of the model—it can have billions of parameters. A parameter is a tiny number inside the model that gets adjusted during training to help it learn. More parameters usually mean the model can capture more complex patterns. LLMs are particularly good at anything involving human language: writing, summarising, translating, answering questions, and even having conversations.

How does an LLM actually work? It uses a technology called the Transformer architecture. This architecture allows the model to look at all the words in a sentence at once, rather than reading them one by one. This gives it a much better understanding of context. For example, in the sentence, 'The bank was steep,' the model uses the word 'steep' to know that 'bank' means a riverbank, not a financial institution. It processes the entire sentence to assign the correct meaning to each word. The core operation is predicting the next word. When you give an LLM the prompt 'The capital of France is...', it processes the words it has and calculates the most probable next word, which is 'Paris'. This happens again and again, with each newly generated word becoming part of the prompt for the next word, allowing it to produce entire paragraphs.

Why do these models exist? Before foundational models, building an AI application was slow and expensive. If a company wanted a chatbot for customer service, they had to train a small, specific model from scratch using only their own customer service data. This required huge amounts of labelled data (data that has been manually tagged and sorted) and powerful computers. The process was unique for every single task, so every application was built in isolation. Foundational models changed this. They represent a paradigm shift known as transfer learning. A single massive, pre-trained model (the foundational model) can be downloaded or accessed via an API (a way for programs to talk to each other). Then, it can be fine-tuned with a much smaller amount of specific data to perform a custom task. This is like taking a fully trained chef and giving them a one-hour class on making pizzas, rather than teaching someone to cook from scratch.

This approach has led to the rise of generative AI, which is AI that creates new content instead of just analysing existing content. LLMs are the engine behind most generative AI text tools you see today, such as chatbots, code generators, and creative writing assistants. They replace the old method of rule-based systems, where a programmer had to write explicit instructions for every possible situation. Instead, LLMs learn the rules intuitively from the data. They are powerful because they are generalists—they can perform tasks they were never explicitly trained on, a quality called emergent abilities.

## Real-world context

Let's say an IT professional works for a large insurance company. Their manager asks them to deploy a system that can automatically summarise complex insurance claims, which are often hundreds of pages long. The professional knows that building a custom AI model from scratch for this task would take months and a fortune in computing costs. Instead, they decide to use a foundational model.

Here is the step-by-step process the IT professional follows:

First, they research available foundational models. They look at options like Google's Gemini or other large language models available through cloud services, such as Vertex AI on Google Cloud. They choose one that has already been trained on a vast amount of text, including legal and technical documents.

Second, they set up access to the model. They do this through an API key, which is like a digital keycard that gives the model permission to talk to the company's applications. The model itself is not installed on the company's own computers; it runs on Google's powerful cloud servers, which is why the approach is called 'model as a service'.

Third, they prepare the data for fine-tuning. They take a sample of 500 historical claims and their human-written summaries. This is the 'supervised' part—the data has correct answers (the summaries) that the model can learn from. They format this data into a structure the model understands, typically a JSON file with pairs of prompts (the claim) and responses (the summary).

Fourth, they run the fine-tuning process. This involves sending their prepared data to the foundational model via the API. The model adjusts its internal parameters slightly to become specialised in summarising insurance claims. The IT professional monitors the training job to ensure it completes without errors and checks the accuracy of the resulting summaries against a test set.

Finally, they deploy the fine-tuned model. The model is now ready to be used. The professional integrates it into the company's claims processing software. When an adjuster uploads a new claim document, the software sends the text to the model, which returns a concise one-paragraph summary in seconds. The IT professional sets up logging and monitoring to track the model's performance and cost, ensuring it runs efficiently and within the company's budget. This entire process, which used to take months, can now be completed in a few days.

## Exam focus

The Generative AI Leader exam tests your understanding of the strategic and conceptual differences between foundational models and large language models (LLMs), not how to code them. You will not be asked to write a Python script. Instead, expect scenario-based questions where you must choose the right type of model for a given business problem.

Here are the exact concepts they love to test:

- The definition of a foundational model: It is a large AI model trained on broad data that can be adapted to a wide range of downstream tasks. The key word is 'adaptable'.
- The difference between a foundational model and an LLM: An LLM is a type of foundational model focused on language. Not all foundational models are LLMs. Some are trained on images (like the foundational models behind image generators) or code.
- The concept of fine-tuning: This is the process of taking a pre-trained foundational model and training it further on a smaller, specific dataset to improve its performance on a particular task. The exam will ask when you should fine-tune versus using a pre-trained model as-is (zero-shot prompting).
- Parameters: Understand that models with more parameters are generally more capable but also more expensive to run. The exam might present a trade-off scenario (e.g., speed vs. accuracy) and ask which model size to choose.
- Transfer learning: This is the core benefit. The exam expects you to know that foundational models enable transfer learning, which means knowledge gained from one task (general training) can be transferred to another (specific use case).

Common traps the exam sets:

- Confusing 'foundational model' with 'foundation model initiative' (a separate Google-specific term). Stick to the general definition.
- Thinking that an LLM is always better than a smaller model. The exam will test that smaller, fine-tuned models can be more efficient and cost-effective for narrow tasks than a giant general-purpose LLM.
- Assuming foundational models are only for text. Remember that some foundational models handle multiple modalities (text, image, audio).
- Forgetting the 'large' in LLM. The exam might ask how the size (number of parameters) impacts latency (speed) and cost.

Key definitions to memorise:
- Foundation Model: An AI model trained on a broad set of data at scale, designed to be adapted to a wide range of tasks.
- LLM: A type of foundation model specifically designed for language tasks, often using a Transformer architecture.
- Fine-tuning: The process of further training a pre-trained model on a specific dataset to improve task performance.
- Parameter: A variable within the model that is adjusted during training to learn from data.

## Step by step

1. **Data Collection and Pre-training** — The foundational model is trained from scratch on a massive, diverse dataset (like the entire public internet). This is the most computationally expensive step, done once by organisations like Google. The model learns general patterns of language, syntax, and knowledge.
2. **Model Architecture Selection** — A specific architecture, most commonly the Transformer architecture, is chosen. This architecture allows the model to process all words in a sequence simultaneously, giving it a strong understanding of context. The architecture defines how the model's parameters are structured.
3. **Pre-training and Creating the Base Model** — The model is exposed to the massive dataset and iteratively adjusts its billions of parameters to better predict the next word in a sequence. This results in a 'pre-trained' foundational model that can perform basic language tasks but is not yet specialised for any single business use.
4. **Fine-tuning (Optional)** — An organisation takes the pre-trained foundational model and trains it further, but on a much smaller, curated dataset that is specific to their task (e.g., customer support transcripts). This adjusts the model's parameters slightly to excel at that particular task without losing its general knowledge.
5. **Deployment and Inference** — The final model (whether pre-trained or fine-tuned) is deployed. Users send prompts or inputs to the model via an API or an application interface. The model runs an 'inference' calculation, using its trained parameters to generate a response, which is then returned to the user.

## Comparisons

### Foundational Model vs Large Language Model (LLM)

**Foundational Model:**
- General-purpose AI model trained on broad data (text, images, code).
- Can be adapted for many different types of tasks, not just language.
- Examples: Google's Gemini, which handles text, images, and audio.

**Large Language Model (LLM):**
- A specific type of foundational model for language tasks only.
- Trained predominantly on text data from books, articles, and websites.
- Examples: GPT-3, PaLM 2 (focus on text generation and understanding).

### Fine-tuning vs Zero-shot Prompting

**Fine-tuning:**
- Requires preparing a custom dataset of task-specific examples.
- Changes the model's internal parameters, leading to better performance on that task.
- More costly and time-consuming upfront but yields higher accuracy for specific uses.

**Zero-shot Prompting:**
- Uses the pre-trained model as-is without any additional training.
- Relies on the model's existing knowledge and a well-crafted prompt.
- Cheaper and faster to implement, but may be less accurate for niche tasks.

### Parameter vs Training Data

**Parameter:**
- A numerical value inside the model that is adjusted during learning.
- The number of parameters (e.g., 175 billion) determines model capacity.
- Parameters are the 'memory' of the model; more means more pattern storage.

**Training Data:**
- The raw text, images, or code used to train the model.
- The quality and diversity of training data determines what the model learns.
- Data is the raw material; parameters are the learned rules from that material.

### Transformer Architecture vs Recurrent Neural Network (RNN)

**Transformer Architecture:**
- Processes entire sequence of words in parallel, giving better context understanding.
- Scales very well to large models (billions of parameters).
- Is the standard architecture for modern LLMs like GPT and Gemini.

**Recurrent Neural Network (RNN):**
- Processes words sequentially one at a time, making it slower for long sequences.
- Suffers from memory issues with very long text ('vanishing gradient problem').
- Was the previous standard architecture before Transformers were invented.

## Common misconceptions

- **Misconception:** A large language model is the same thing as a foundational model. **Reality:** A large language model (LLM) is a specific subtype of foundational model. Foundational models can also be trained on images, code, or multiple types of data at once. (The exam uses the term 'foundational model' broadly, while 'LLM' is a common buzzword. Beginners often use the terms interchangeably without realising one is a subset of the other.)
- **Misconception:** Foundational models are fully intelligent and can reason like a human. **Reality:** Foundational models are advanced pattern matchers. They do not understand or reason; they predict the most probable next word based on their training data. Their 'reasoning' is an emergent property of statistical patterns. (The human-like fluency of model outputs leads people to anthropomorphise them (give them human traits). The exam tests that you understand their limitations, specifically around 'hallucinations' (confidently giving wrong answers).)
- **Misconception:** To use a foundational model, an organisation must train it from scratch using their own data. **Reality:** Most organisations do not train models from scratch. They use pre-trained models (foundational models) and either use them as-is or fine-tune them with a much smaller amount of their own data. (This misconception comes from older AI development practices. The shift to transfer learning is a key exam topic, and many beginners think the old expensive method is still the only way.)
- **Misconception:** A larger model (more parameters) is always the best choice for any task. **Reality:** Larger models are more capable but also slower, more expensive, and more difficult to deploy. For a simple, narrow task, a smaller fine-tuned model often performs better and is more cost-effective. (The tech industry often promotes 'bigger is better' for LLMs. Beginners miss the important trade-offs that the exam explicitly tests, such as cost, latency, and operational complexity.)
- **Misconception:** The training data for a foundational model is just the company's internal data. **Reality:** Foundational models are trained on massive public datasets from the internet, including books, articles, code repositories, and images. Company-specific fine-tuning uses only the company's data on top of this broad base. (People confuse the general public training (which is huge and generic) with the final fine-tuning stage (which is small and specific). The exam tests that you understand the stages of building a solution with these models.)

## Key takeaways

- A foundational model is a large, general-purpose AI model trained on a wide variety of data that can be adapted for many different tasks.
- A large language model (LLM) is a specific type of foundational model that specialises in understanding and generating human language.
- Fine-tuning is the process of taking a pre-trained foundational model and training it further on a smaller, task-specific dataset to improve its performance.
- The primary benefit of foundational models is transfer learning, which allows knowledge from broad training to be reused for specific problems, saving time and cost.
- Models with more parameters are generally more capable but also more expensive and slower to run, creating a trade-off decision for businesses.
- Foundational models do not 'understand' text; they predict the most probable next word based on statistical patterns learned from their training data.

## FAQ

**What is the difference between a foundational model and a large language model?**

A foundational model is the broad category for any large AI model trained on a general dataset (like text, images, or code). A large language model (LLM) is a specific type of foundational model that is trained almost exclusively on text data and designed for language tasks.

**Does a foundational model understand what it is saying?**

No. A foundational model does not 'understand' in a human sense. It is a statistical machine that has learned to predict the most likely sequence of words based on patterns in its training data. It can produce coherent text without any real awareness or comprehension.

**Why do these models hallucinate and make things up?**

Hallucinations occur because the model's primary job is to predict the next plausible word, not to verify facts. If the pattern in the data for a question is a confident-sounding but incorrect statement, the model will output that without knowing it is wrong.

**How much data do I need to fine-tune a foundational model?**

Much less than training from scratch. For many tasks, you can achieve good results with a few hundred or a few thousand carefully labelled examples. The exact number depends on the complexity of the task and the similarity to the model's original training data.

**Is a larger foundational model always better?**

Not necessarily. Larger models (more parameters) are more capable and can handle more complex tasks, but they are also significantly slower and more expensive to run. For many simple or narrow business tasks, a smaller, fine-tuned model is a better choice.

**Can I use a foundational model to generate images?**

Yes, but you would not use an LLM for that. Image generation requires a different type of foundational model, often based on diffusion or other architectures, that is specifically trained on image data. Google's Imagen is an example of an image-focused foundational model.

---

Interactive version with quiz and diagrams: https://courseiva.com//learn/google-gen-ai-leader/foundational-models-and-large-language-models