Foundation Models and Fine-Tuning

Foundation models are large-scale machine learning models trained on vast amounts of diverse data using self-supervised or semi-supervised learning. The term was popularized by the Stanford Institute for Human-Centered AI (HAI) in 2021. These models are typically based on transformer architectures and contain billions of parameters. Examples include GPT-4 (OpenAI), BERT (Google), Llama (Meta), and DALL-E (OpenAI for images). The key property: they are generalists. A single foundation model can perform multiple tasks—translation, summarization, question answering, code generation—without task-specific training, because the training data covers many domains.

Before foundation models, AI development required training a separate model from scratch for each task. This was expensive, time-consuming, and required large labeled datasets. Foundation models solve this by providing a single pre-trained model that can be adapted to many downstream tasks. This transfer learning approach drastically reduces the data and compute needed for new tasks. For example, training GPT-3 from scratch cost an estimated $4.6 million; fine-tuning it for a specific task costs a fraction of that.

Foundation models are trained using a transformer architecture. The training process involves: - Pre-training: The model is trained on a large corpus of text (or images) using objectives like masked language modeling (BERT) or autoregressive next-token prediction (GPT). The model learns statistical patterns, grammar, facts, reasoning abilities, and even some world knowledge. - Parameters: During training, the model adjusts its weights (parameters) to minimize prediction error. For example, GPT-3 has 175 billion parameters. These parameters encode the learned patterns. - Context window: The model can attend to a fixed number of previous tokens (e.g., GPT-4 Turbo has a 128K token context window). This determines how much context the model can consider when generating output. - Inference: When you input a prompt, the model processes it through its layers, generating a probability distribution over the vocabulary for the next token, then samples or selects the most likely token. This repeats autoregressively.

Fine-tuning is the process of taking a pre-trained foundation model and further training it on a smaller, task-specific dataset. This adapts the model's behavior to perform better on that particular task. For example, you might fine-tune GPT-4 on a dataset of customer support conversations to create a chatbot that responds in your company's tone and follows specific policies.

Fine-tuning typically follows these steps: 1. Prepare a dataset: Collect a labeled dataset of input-output pairs for the target task. For text generation, this might be prompt-response pairs. The dataset should be representative and large enough (hundreds to thousands of examples). 2. Initialize with pre-trained weights: Start with the foundation model's weights. This is crucial—you are not training from scratch. 3. Set a low learning rate: The learning rate is typically 1e-5 to 5e-5, much lower than the pre-training learning rate (which might be 1e-4 or higher). This prevents the model from forgetting its general knowledge (catastrophic forgetting). 4. Train on the new dataset: Run standard supervised learning: for each batch, compute the loss (e.g., cross-entropy between predicted tokens and target tokens), then backpropagate to update weights. The model learns to map inputs to desired outputs. 5. Evaluate and iterate: Monitor performance on a validation set. Fine-tuning may require only a few epochs (1-5) because the model already has strong priors.

Azure OpenAI Service supports fine-tuning for models like GPT-3.5-Turbo, GPT-4 (limited), and others. The process is: 1. Prepare training data: JSONL format with each line containing a conversation (messages array) with role (system, user, assistant) and content. 2. Upload data: Use Azure OpenAI Studio or the REST API. 3. Create a fine-tuning job: Specify the base model, training file, validation file (optional), and hyperparameters (learning_rate_multiplier, n_epochs, batch_size). 4. Monitor: Track training loss and validation loss via the portal. 5. Deploy: Once fine-tuned, deploy the model as a custom endpoint.

Example CLI command using Azure CLI:

az cognitiveservices account list --query "[?contains(name, 'myopenai')].{Name:name, ResourceGroup:resourceGroup}" -o table

But for fine-tuning, you typically use the REST API or Python SDK:

from openai import AzureOpenAI

client = AzureOpenAI(
    azure_endpoint="https://your-resource.openai.azure.com",
    api_key="your-api-key",
    api_version="2024-02-01"
)

client.fine_tuning.jobs.create(
    training_file="file-123abc",
    model="gpt-35-turbo",  # base model
    hyperparameters={
        "n_epochs": 3,
        "batch_size": 16,
        "learning_rate_multiplier": 0.1
    }
)

Fine-tuning is often compared to: - Prompt Engineering: No training; you craft prompts to guide the model. Cheaper but less powerful. - RAG (Retrieval Augmented Generation): Combines retrieval from a knowledge base with generation. No model retraining; adapts at inference time. - Transfer Learning: Fine-tuning is a form of transfer learning where the pre-trained model is the source. - Catastrophic Forgetting: If fine-tuning uses too high a learning rate or too many epochs, the model may overfit to the new task and lose general capabilities.