# Model Tuning and Deployment on Vertex AI

> Chapter 14 of the Courseiva GOOGLE-GEN-AI-LEADER curriculum — https://courseiva.com//learn/google-gen-ai-leader/model-tuning-and-deployment-on-vertex-ai

**Official objective:** 4.5 — Explain model tuning (fine-tuning, adapter tuning) and deployment options

## Introduction

Model tuning takes a powerful but generic pre-trained AI model and specialises it for a particular job, like turning a general-purpose assistant into a customer support bot for your specific store. Deployment makes that tuned model available for real users to interact with, which is the ultimate goal of any Generative AI project.

## The Tailor and the Ready-Made Suit Analogy

Because you want a suit that fits perfectly for a specific event, you first go to a shop to buy one. You choose a well-made, off-the-rack suit that is a good starting point. However, it is too loose in the shoulders and the sleeves are a bit long. To make it perfect for your body, you take it to a tailor. The tailor makes small, precise adjustments: taking in the waist, shortening the sleeves, and adjusting the collar. This is like **fine-tuning** a pre-trained model on a small, specific dataset to make it perform a niche task exceptionally well.

Now, because you want to wear this suit to a wedding and also to a business conference, you do not want to buy a new suit each time. Instead, you use a different trick: you add a few removable accessories, like a special pocket square or a temporary lapel pin, that change the suit's style without altering the suit itself. This is **adapter tuning** – you add a small, lightweight module to the model that teaches it a new skill without changing all the original weights.

Finally, after the suit is finished, you need to get it to the event. You cannot just leave it hanging at the tailor's. You must transport it to the wedding venue. This is **deployment** – taking your tuned model from the training environment and placing it into a live system (like Vertex AI's Prediction Endpoints) where users can actually use it to get predictions, answers, or generate content in real time.

## Core explanation

Think of a pre-trained model as a highly educated, multi-talented assistant. This assistant has read millions of books, seen countless images, and can hold a conversation on almost any topic. This is the 'foundation model' – it is incredibly broad but not an expert in your specific business, like a general physician who knows a little about everything.

To make this assistant an expert on your company's specific return policy, you cannot just ask it a question and hope it guesses correctly. You need to **tune** it. There are two main ways to do this on Vertex AI.

**1. Fine-Tuning (Full Tuning):** This is the most thorough method. You take the foundation model and continue its training process, but this time using a dataset of examples specifically for your task. For instance, you provide thousands of examples of your customer support conversations. During fine-tuning, the model's internal parameters (its 'weights') are updated very slightly across the entire network. This is like taking our general physician and giving them years of specialised training in cardiology. They still know general medicine, but their most important circuits are now wired for the heart.

The pros are that it offers the highest performance for a very specific task. The cons are that it is expensive (needs more computing power) and slower to complete, and you lose the ability to easily switch the model to a different task without tuning it again.

**2. Adapter Tuning (Parameter-Efficient Fine-Tuning or PEFT):** This is a smarter, lighter approach. Instead of retraining the entire model, you add a small, new 'adapter' module to the existing model and only train that module. The original model stays frozen. Imagine our physician getting a 'cardiology module' plug-in to their brain. They use their existing general knowledge, but the new module gives them precise cardiology answers. On Vertex AI, this is often done with techniques like LoRA (Low-Rank Adaptation).

The advantages are that it is much faster and cheaper than full fine-tuning, and you can have many different adapters for different tasks saved on top of the same base model. The trade-off is that it might not achieve the same peak performance for a very complex task compared to full fine-tuning.

**Deployment on Vertex AI:** Once your model is tuned (or even if it is not and you just want a ready-made one), you need to **deploy** it. Deployment is the process of making the model available as a service that applications can call.

Vertex AI offers several deployment options:

- **Vertex AI Endpoints:** This is the most common method. You create an endpoint, which is like a dedicated mailbox. You then 'deploy' your model to that endpoint. When a user or application sends a prompt (the input), it is sent to this mailbox. The model processes it and sends back the response. This is a 'real-time' prediction environment.
- **Batch Predictions:** If you need to process a huge number of inputs at once (e.g., summarising 10,000 product descriptions from a spreadsheet), you do not need a live endpoint. You send the whole batch to Vertex AI, it processes them all in the background, and saves the results to a file (like a Cloud Storage bucket). This is cheaper and more efficient for large, non-real-time jobs.

**Why Tune and Deploy?** Taking a raw pre-trained model and deploying it as-is is like handing a brand-new employee a set of instructions. Fine-tuning or adapter-tuning is like giving that employee months of on-the-job experience. The result is a model that is faster, more accurate, and safer for your specific business needs, such as answering legal questions or generating code in your company's own internal programming style.

**Putting it all together:**<br>- You start with a foundation model (like Gemini).<br>- You choose a tuning method (Fine-tuning or Adapter).<br>- You use a dataset of your own examples (e.g., your company FAQs).<br>- You run the tuning job on Vertex AI (this uses a lot of compute power).<br>- You evaluate the newly tuned model.<br>- You deploy the model to an Endpoint for real-time prompts.<br>- Your application sends a prompt, and the endpoint returns a tuned response.

## Real-world context

An IT professional, let us call her Priya, is working for a large online clothing retailer. The company wants to build an AI chatbot on their website to help customers find the perfect outfit and answer questions about sizing, returns, and style.

**Step 1: Selecting the Foundation Model**<br>Priya does not start from scratch. She goes to the Vertex AI Model Garden and selects a pre-trained foundation model, specifically the Gemini Pro model. This model is already good at understanding human language and generating coherent responses.

**Step 2: Preparing the Training Data**<br>Priya collects 5,000 past customer service conversations. She cleans this data, making sure it contains no personal information like names or addresses. She then formats it into a specific structure: a 'prompt' (the customer's question) and a 'response' (the ideal agent's answer). This becomes the **tuning dataset**.

**Step 3: Choosing the Tuning Method**<br>Because the company has a limited budget, Priya decides to use **adapter tuning** (specifically, a technique called LoRA). She does not want to retrain the entire massive Gemini Pro model. She opens the Vertex AI console, selects her base model, uploads her tuning dataset, and chooses the 'Adapter Tuning' option. She sets a few parameters like 'epochs' (how many times the model sees her data).

**Step 4: Running the Tuning Job**<br>Priya clicks 'Start'. Vertex AI spins up a cluster of GPU machines (powerful computers designed for AI) and begins the training. The job runs for about 2 hours, which is much faster than the 20+ hours full fine-tuning would have taken. She monitors the job in the Vertex AI dashboard, watching the loss metric (how well the model is learning) decrease.

**Step 5: Evaluating and Testing**<br>Once the job is done, Priya evaluates the model. She sends it sample questions like 'What is your return policy on dresses?' and compares its answer to the correct answer from her dataset. She is satisfied with the accuracy.

**Step 6: Deployment**<br>Now the real work begins. She wants users on the website to get responses instantly. She goes to the 'Models' tab in Vertex AI and selects her tuned model. She clicks 'Deploy to Endpoint'. She gives the endpoint a name: 'customer-support-v1'. She chooses the 'n1-standard-2' machine type (a specific size of compute resource) and sets it to have a minimum of 2 replicas for reliability. It also enables automatic scaling to handle spikes in traffic.

**Step 7: Integration**<br>Priya gets a URL for her new endpoint. She provides this URL to the web development team. They write code in the company's website that sends the customer's typed question to this endpoint URL and displays the response in the chat window. The entire system is live.

**Ongoing Tasks:**<br>Priya now monitors the endpoint. She checks for latency (how fast it responds), error rates, and throughput (how many requests per second it handles). She also collects new conversations to create a new, even better tuning dataset in a few months, repeating the cycle.

 **Actions and tools Priya uses in her workflow:**<br>- **Vertex AI Model Garden** to find her base model.<br>- **Vertex AI Tuning Pipeline** (an automated workflow) to run the adapter tuning job.<br>- **Vertex AI Endpoints** to deploy the model.<br>- **Cloud Logging and Cloud Monitoring** to observe the live endpoint's health.<br>- **Google Cloud Storage** to store her training datasets and the results of batch predictions.

## Exam focus

The Generative AI Leader exam does not ask you to write code or configure servers. It tests your ability to make **strategic decisions** about when and why to use different tuning and deployment methods. They want to see if you understand the trade-offs, costs, and business implications. Here is exactly what they test, and the traps they set.

**Exam Topics you must know:**

- **Fine-Tuning vs. Adapter Tuning (PEFT):** This is the single most tested distinction in this objective. You must know the precise differences:<br>- **Fine-tuning**: Updates all model weights. Highest quality for a specific task. Requires significant compute (GPU/TPU) time. Can lose generalisability (catastrophic forgetting). Creates a large, separate model version. **Choose this when performance is paramount and cost is a secondary concern, or when the task requires a fundamental domain shift (e.g., turning a general model into a medical specialist).**<br>- **Adapter Tuning**: Only trains new adapter layers. Much cheaper and faster. Preserves the base model's capabilities. Allows for many 'specialised' adapters on one base model (multi-task adaptability). Performance is good but may be slightly worse than full fine-tuning for very complex tasks. **Choose this when you need to be cost-efficient, fast, or when you have multiple, related tasks to support.**

- **Deployment Options:**<br>- **Vertex AI Endpoint (Online Prediction):** For real-time, low-latency applications like a chatbot or a real-time text generator. Cost is per hour for the serving infrastructure, plus per-request. **The exam tests your ability to identify a use case that needs 'real-time inference'.**<br>- **Batch Prediction:** For processing large datasets where latency is not important. Cost is per node hour for the processing job. **The exam tests if you recognise a scenario like 'I have 100,000 PDFs to summarise overnight' as a batch prediction task.**<br>- **Model Garden vs. Vertex AI Studio:** Model Garden is the library of foundation models you can use (Gemini, Imagen, etc.). Vertex AI Studio is the web interface for testing, prompting, and tuning them without coding. **The exam tests if you know where to find models (Model Garden) and where to prompt/tune them (Studio).**

**Common Trap Patterns:**

- **The 'Always Fine-Tune' Trap:** The question will describe a scenario (e.g., a company wants a model to answer HR policy questions) and will offer both fine-tuning and adapter tuning as answers. The trap is assuming fine-tuning is always best because it is 'more powerful'. The correct answer is almost always adapter tuning when cost, speed, or model versioning is mentioned, or when the base model already has decent domain knowledge.
- **The 'Endpoint is the Only Way to Deploy' Trap:** The exam will describe processing a daily report of 5,000 sales calls. A tempting answer is 'Deploy to an endpoint'. The correct answer is 'Use batch prediction', because there is no need for real-time processing.
- **The 'Data Privacy' Trap:** If a scenario mentions sensitive data (e.g., patient health records), the question might ask which tuning method is best. The correct reasoning is that fine-tuning or adapter tuning means the model learns from your data, but you must use a private, isolated environment (like Vertex AI's Dedicated Endpoints or a Virtual Private Cloud) to prevent data leakage. Do not pick an option that sends data to a public API.

**Key Definitions to Memorise:**
- **PEFT**: Parameter-Efficient Fine-Tuning (another name for adapter tuning).
- **LoRA**: Low-Rank Adaptation, the most common adapter method.
- **Epoch**: One complete pass through the entire training dataset during tuning.
- **Inference**: The act of generating a prediction or output from a deployed model.
- **Latency**: The time delay between sending a request and receiving a response.
- **Catastrophic Forgetting**: When a model loses its original knowledge because it was fine-tuned too aggressively on a new task.

## Step by step

1. **Prepare the Tuning Dataset** — You gather a collection of prompt-response pairs or other structured examples that represent the task you want the model to learn (e.g., customer support questions and answers). This dataset must be representative, cleansed of sensitive data, and formatted correctly (often as JSONL). The quality of this dataset directly determines the quality of the tuned model.
2. **Select Foundation Model and Tuning Method** — In Vertex AI Model Garden, you choose a base model (like Gemini 1.5 Pro). Then you decide if you will use full fine-tuning (expensive, high quality) or adapter tuning/ PEFT (cheaper, faster, preserves general knowledge). You configure the training parameters such as learning rate, batch size, and number of epochs.
3. **Run the Tuning Job on Vertex AI** — You submit the tuning job. Vertex AI automatically provisions the necessary compute resources (GPUs or TPUs), runs the training loop, and saves the tuned model's weights (or adapter weights). You can monitor the progress and loss curves in the Vertex AI console.
4. **Evaluate the Tuned Model** — Before deploying, you must evaluate the model's performance. You send it test prompts (not in the training data) and assess the accuracy, relevance, and safety of its outputs. Vertex AI provides evaluation tools for this. If the performance is unsatisfactory, you may adjust the dataset or parameters and re-run the tuning job.
5. **Deploy the Model to an Endpoint or for Batch Prediction** — Once satisfied, you deploy the model. For real-time use, you create a Vertex AI Endpoint, deploy the model version to it, and configure machine type, scaling, and logging. For batch tasks, you configure a Batch Prediction job that processes a file of inputs and saves outputs to Cloud Storage. The model is now live.
6. **Monitor and Iterate** — After deployment, you continuously monitor the endpoint's latency, error rates, and throughput using Cloud Monitoring and Logging. You also collect new user data to improve the tuning dataset. Periodically, you may re-tune the model or deploy a new version to the same endpoint, performing a ‘traffic split’ to test the new version before full rollout.

## Comparisons

### Fine-Tuning vs Adapter Tuning (PEFT)

**Fine-Tuning:**
- Updates all model weights.
- Requires significant compute resources (hours of GPU/TPU).
- Higher risk of catastrophic forgetting.

**Adapter Tuning (PEFT):**
- Only trains a small adapter module; base model unchanged.
- Much cheaper and faster (minutes to a couple of hours).
- Preserves base model's general knowledge.

### Vertex AI Endpoint (Online) vs Batch Prediction

**Vertex AI Endpoint (Online):**
- For real-time, user-facing applications.
- Low latency required (milliseconds to seconds).
- Pays per hour for serving infrastructure + per request.

**Batch Prediction:**
- For processing large volumes of data offline.
- Latency is not important (takes minutes or hours).
- Pays per node hour for the processing job.

### Model Garden vs Vertex AI Studio

**Model Garden:**
- A library/catalogue of available foundation models.
- You select a model from here to use or deploy.
- Does not have a direct tuning interface.

**Vertex AI Studio:**
- A web UI for interacting with and tuning models.
- You can write prompts, test outputs, and start tuning jobs.
- Integrated with Model Garden to access models.

## Common misconceptions

- **Misconception:** Fine-tuning and adapter tuning are basically the same process, just with different names. **Reality:** They are fundamentally different. Fine-tuning changes the entire model's weights, while adapter tuning only trains a small add-on module. Adapter tuning is much cheaper and faster but may not achieve the same peak performance for extremely niche tasks. (Beginners often see the word 'tuning' and assume it is a single method. The acronym 'PEFT' (Parameter-Efficient Fine-Tuning) is also confusing because it still contains 'fine-tuning'. The key distinction is 'does it change the whole brain, or just attach a new module?'.)
- **Misconception:** You must tune a model before you can deploy it. **Reality:** You can deploy a base, un-tuned foundation model directly from Model Garden to an Endpoint. Tuning is an optional optimisation step to make it better for a specific task. A lot of beginners believe tuning is mandatory for deployment, but it is not. (Marketers describe models as 'ready to use', which leads people to think tuning is an extra step you might skip. The reality is that for many simple tasks (like basic text generation), a base model is perfectly adequate. Tuning becomes essential only when you need a specific behaviour or domain expertise.)
- **Misconception:** Deploying a model to an endpoint is the same as hosting it on a normal web server. **Reality:** While both make a service accessible, Vertex AI Endpoints are specifically designed for machine learning inference. They handle autoscaling, load balancing, request batching, GPU acceleration, and model versioning automatically. A normal web server would struggle with the high compute demands and latency requirements of a large generative AI model. (People with a web development background assume any application runs on a standard server. They do not understand that AI models require specialised hardware (GPUs/TPUs) and an inference engine to run efficiently. The exam tests the understanding that Vertex AI Endpoints are a managed 'AI-specific' service, not a general-purpose compute instance.)
- **Misconception:** Fine-tuning a model will always make it better at every task. **Reality:** Fine-tuning can cause 'catastrophic forgetting'. If you only train the model on data about your company's return policy, it may forget how to answer general questions about other topics or lose common sense it previously had. Adapter tuning is better at preserving the base model's general knowledge. (This is a counter-intuitive concept. People believe 'more training = better model'. They do not realise that training is a trade-off. The exam loves testing this because it reveals a deeper understanding of model mechanics. The common phrase 'you can have too much of a good thing' applies directly here.)

## Key takeaways

- Model tuning specialises a pre-trained foundation model for a specific task by training it on your own data, improving accuracy and safety for your use case.
- Fine-tuning updates all the model's weights, offering the highest performance but requiring significant compute and risking catastrophic forgetting.
- Adapter tuning (PEFT) adds a small, trainable module to a frozen base model, making it much cheaper and faster than fine-tuning while preserving the model's general knowledge.
- Vertex AI Endpoints are used for real-time, low-latency inference for applications like chatbots, while Batch Prediction is used for high-volume, non-real-time processing.
- The choice between fine-tuning and adapter tuning depends on the trade-offs between performance, cost, speed, and the need to maintain the model's general capabilities.
- Deployment is the process of making a tuned or untuned model available for inference via an endpoint or batch job, and Vertex AI manages the infrastructure, scaling, and monitoring automatically.
- You can deploy a base foundation model from Model Garden without any tuning at all if it already performs adequately for your task.

## FAQ

**What is the difference between fine-tuning and prompt engineering?**

Prompt engineering is writing a clever, detailed instruction (the prompt) that tells the base model what to do at inference time, without changing the model itself. Fine-tuning actually updates the model's internal weights by training it on thousands of examples. One is a temporary instruction; the other is permanent learning.

**How much does fine-tuning a model on Vertex AI cost?**

Costs depend on the model size and the method. Fine-tuning a large model (e.g., Gemini 1.5 Pro) for multiple hours on powerful TPU machines can cost hundreds or thousands of dollars. Adapter tuning is significantly cheaper, often 10-50% of the cost of full fine-tuning. You are charged for compute time and storage of the dataset and tuned model.

**Can I deploy a model that was fine-tuned outside of Vertex AI (e.g., with Hugging Face)?**

Yes, you can. You can import a custom model (including weights from Hugging Face) into Vertex AI Model Registry and deploy it to an endpoint, as long as the model architecture is compatible. Vertex AI supports popular frameworks like PyTorch, TensorFlow, and JAX.

**What is 'catastrophic forgetting'?**

Catastrophic forgetting is when a model that is fully fine-tuned on a new, narrow task loses its ability to perform the original general tasks it was trained for. For example, a model fine-tuned only on medical texts might forget how to write a poem or answer common sense questions. Adapter tuning helps avoid this.

**Do I need to know how to code to tune a model in Vertex AI?**

No. The Google Cloud Console provides a no-code interface under Vertex AI Studio where you can upload a dataset, choose a model, and start a tuning job with clicks. However, for advanced control or automation, you can use the Vertex AI SDK or API with Python.

**What is the difference between an endpoint and a model?**

A model is the AI algorithm itself (the 'brain'). An endpoint is the live, accessible service that hosts that model (the 'mailbox'). You can have one endpoint that serves multiple versions of a model for A/B testing, or you can have many endpoints pointing to the same model.

---

Interactive version with quiz and diagrams: https://courseiva.com//learn/google-gen-ai-leader/model-tuning-and-deployment-on-vertex-ai