Generative AI for Business Leaders

Generative AI refers to a class of artificial intelligence models that can generate new content—text, images, audio, video, code, and more—based on patterns learned from training data. Unlike traditional AI models that classify or predict a label (e.g., 'is this email spam?'), generative models create novel outputs that mimic the statistical properties of the training data. The most prominent type of generative AI today is the large language model (LLM), such as Google's PaLM 2, Gemini, and GPT-4. These models are built on the Transformer architecture, introduced in the 2017 paper 'Attention Is All You Need' (Vaswani et al.).

At a high level, generative AI models are trained on massive datasets (e.g., trillions of tokens of text, billions of images). During training, the model learns to predict the next token (a word or subword) given the previous tokens. This is called autoregressive language modeling. The model's parameters (weights) are adjusted to minimize the difference between its predictions and the actual next token in the training data. Once trained, the model can generate text by repeatedly sampling from its predicted probability distribution over the vocabulary. Each new token becomes part of the input for the next prediction, creating a coherent sequence.

Key steps in generation: - Tokenization: Input text is split into tokens (e.g., 'Generative' might be split into 'Gen' and 'erative'). Each token is mapped to a unique ID. - Embedding: Each token ID is converted into a high-dimensional vector (e.g., 768 or 4096 dimensions) that captures semantic meaning. - Attention Mechanism: The model computes attention scores between every pair of tokens in the input sequence. This allows the model to weigh the importance of different tokens when predicting the next one. For example, in the sentence 'The cat sat on the mat,' the word 'cat' might have high attention to 'sat' and 'mat'. - Feed-Forward Layers: After attention, the token representations pass through feed-forward neural networks that apply non-linear transformations. - Output Layer: The final hidden state for the last token is projected to a vector of size equal to the vocabulary (e.g., 50,000 tokens). A softmax function converts these scores into probabilities. The model then samples from this distribution to pick the next token.

- Context Window: The maximum number of tokens the model can consider at once. For example, Gemini 1.5 Pro has a context window of up to 1 million tokens. Larger context windows allow the model to process longer documents or conversations. - Temperature: Controls randomness in generation. Low temperature (e.g., 0.1) makes the model more deterministic, picking the highest-probability token. High temperature (e.g., 1.0) increases randomness, leading to more creative but potentially less coherent outputs. - Top-K and Top-P (Nucleus Sampling): Alternative sampling strategies. Top-K limits the next token to the K most probable tokens. Top-P selects the smallest set of tokens whose cumulative probability exceeds P (e.g., 0.9). These techniques reduce the chance of generating nonsensical tokens. - Max Output Tokens: The maximum number of tokens the model can generate in a single response. This prevents excessively long outputs and controls cost. - Stop Sequences: Specific strings (e.g., ' ') that tell the model to stop generating when encountered.

Google Cloud offers pre-trained foundation models (e.g., PaLM 2, Gemini, Codey, Imagen) that can be used out of the box or customized. Customization options include: - Prompt Engineering: Crafting input prompts to elicit desired behavior without changing model weights. For example, adding 'Explain like I'm 5' changes the output style. - Fine-Tuning: Further training the model on a smaller, task-specific dataset. This adjusts model weights to improve performance on specific tasks. Google Cloud's Vertex AI supports supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF). - Grounding: Connecting the model to real-world data sources (e.g., Google Search, enterprise databases) to reduce hallucinations and provide up-to-date information. Vertex AI Agent Builder enables grounding with Google Search or private data. - Retrieval-Augmented Generation (RAG): Combines a retrieval system (e.g., vector database) with a generative model. The model retrieves relevant documents from a knowledge base and uses them as context for generation. This improves accuracy and reduces hallucination.

Using the gcloud CLI or Python SDK, you can interact with Vertex AI generative models. Example using gcloud:

gcloud ai models list --filter=display_name:PALM

To generate text using the PaLM API via Python:

import vertexai
from vertexai.language_models import TextGenerationModel

vertexai.init(project='my-project', location='us-central1')
model = TextGenerationModel.from_pretrained('text-bison@002')
response = model.predict(
    'What is the capital of France?',
    temperature=0.2,
    max_output_tokens=256,
    top_p=0.8
)
print(response.text)

Generative AI models can produce harmful, biased, or hallucinated outputs. Google Cloud provides tools to mitigate these risks: - Safety Attributes: Scores for categories like toxicity, harassment, and hate speech. These can be used to filter outputs. - Bias Detection: Tools to evaluate model outputs for demographic biases. - Human-in-the-Loop: Deploying models with human reviewers to catch errors. - Data Governance: Ensuring training data is properly sourced and anonymized.

A large e-commerce company wants to automate responses to common customer inquiries (e.g., order status, return policy). They use Vertex AI with PaLM 2 for text generation. First, they gather historical chat logs and FAQ documents. They fine-tune the model on a dataset of 10,000 question-answer pairs. They also ground the model by connecting it to their order database via a RAG pipeline: when a customer asks 'Where is my order?', the model retrieves the latest tracking info from BigQuery and generates a personalized response. The system handles 80% of inquiries without human intervention, reducing support costs by 60%. Common issues include the model occasionally hallucinating shipping dates when the database query fails—mitigated by adding a fallback message like 'I'm unable to retrieve your order details right now.' Latency is kept under 2 seconds by using a dedicated endpoint with autoscaling.

A media company uses Imagen (Google's image generation model) to create product images for their online catalog. They need to generate variations of product photos (e.g., different backgrounds, angles) without costly photoshoots. They use prompt engineering to specify style (e.g., 'photorealistic, bright lighting, white background'). They also use ControlNet techniques to ensure the generated image adheres to a product's shape. The system generates 100 images per hour, but quality varies—some images have distorted logos or unrealistic shadows. They implement a human review step where designers approve or reject images before publishing. Cost is $0.05 per image (Imagen pricing), which is cheaper than hiring a photographer.

A software company uses Codey (Code Generation API) to accelerate development. Developers describe a function in natural language (e.g., 'Write a Python function to calculate the Fibonacci sequence') and Codey generates the code. They integrate it into their IDE via a plugin. The model reduces boilerplate coding time by 30%. However, the generated code may contain security vulnerabilities (e.g., SQL injection). They use a static analysis tool to scan generated code before merging. The team also fine-tunes Codey on their internal codebase to improve accuracy for company-specific libraries.