How do you stop a helpful but sometimes wildly inaccurate AI from making up facts about your company's products? Generative AI models are brilliant at writing human-like text, but they have a fatal flaw: they can confidently state falsehoods, a problem called 'hallucination.' For the AI-102 exam, you must know two techniques to control this: prompt engineering (writing better questions) and Retrieval-Augmented Generation, or RAG (giving the AI a reference book to check before it answers).
Jump to a section
A simple way to picture Generative AI: Prompt Engineering and Retrieval-Augmented Generation
Have you ever been on hold with a company's customer service line, shouting your problem into the void, only to get a scripted response that has nothing to do with what you actually asked?
That is a language model without prompt engineering or Retrieval-Augmented Generation. Now imagine this: you call your insurance company about a cracked windscreen. Instead of just saying 'help me,' you say 'I need to file a claim for a cracked windscreen caused by a rock on the M25. My policy number is 12345.' That is prompt engineering — giving clear, specific instructions. But the agent still only knows what their basic training taught them.
Now imagine that before the agent answers, they can instantly pull up your exact policy details, your claim history, and the local garage's repair costs from a huge filing cabinet. That is Retrieval-Augmented Generation (RAG). The agent (the AI) does not guess — they find the right document first and then use it to give you a perfect, accurate answer. The filing cabinet is your company's private knowledge base. Without it, the agent makes things up. With it, they give you the correct, up-to-date answer every time.
To understand prompt engineering and Retrieval-Augmented Generation (RAG), you first need to grasp the core problem they solve: the hallucination problem.
A generative AI model, like the ones you use in Azure OpenAI Service, is essentially a massive, highly sophisticated pattern-matching machine. It was trained on billions of sentences from the internet, books, and other text. When you give it a prompt, it predicts the most likely next word, then the next, and so on, to build a reply. It does not 'know' facts the way a person does. It just knows that in its training data, the words 'capital of France' very often appear next to 'Paris.'
This means if you ask it about a very specific, private, or recent event — like 'What was the sales total for our Widget 3000 in Q3 2024?' — it cannot know the answer. It was not trained on your internal sales data. Instead of saying 'I do not know,' it will often invent a convincing-looking number. That is the hallucination. It sounds plausible, but it is completely wrong.
Prompt Engineering is the skill of crafting your question (the prompt) to guide the AI towards a useful and accurate response. It does not solve the knowledge gap entirely, but it dramatically reduces errors by setting rules.
For example:
- A bad prompt: 'Tell me about the new product.' (The AI invents a product.) - A good prompt: 'You are a product manager. Using only the information provided in the context below, answer the user's question. If the answer is not in the context, say 'I do not know.' Do not make up any information. Context: [Paste your internal document here].'
This is called 'system prompting' or 'meta-prompting.' You are giving the AI a role, a strict instruction about its knowledge boundaries, and explicit formatting rules. This is one of the most tested skills on AI-102.
Common prompt engineering techniques include:
- Few-shot prompting: Giving the AI 2-3 examples of the right answer format before you ask your real question. - Chain-of-thought prompting: Asking the AI to 'think step-by-step' before giving a final answer. This improves reasoning on complex maths or logic problems. - Temperature setting: A parameter you can set in Azure OpenAI. A low temperature (e.g., 0.0) makes the AI's answers very focused, repetitive, and safe. A high temperature (e.g., 1.0) makes it more creative and 'surprising,' which is useful for brainstorming but risky for factual answers. However, even the best prompt cannot give the AI knowledge it never had. That is where RAG comes in. Retrieval-Augmented Generation (RAG) is an architecture pattern. It does not retrain the AI model. Instead, it connects the AI to an external knowledge base (your company's documents, a database, a website) in real-time, right before the AI generates its answer. Here is how RAG works step-by-step: 1. Ingestion Phase (The Setup): You take all your trusted documents (e.g., product manuals, HR policies, legal contracts) and break them into small chunks (like paragraphs). You then turn each chunk into a 'vector embedding' — a long list of numbers that mathematically represents the meaning of that text. You store these embeddings in a vector database, like Azure Cosmos DB or Azure Cognitive Search. 2. Query Phase (When a User Asks): A user types a question. The system takes that question and converts it into the same type of vector embedding. 3. Retrieval Phase: The system searches your vector database for the chunks whose vector embeddings are most mathematically 'similar' to the question embedding. This finds the most relevant documents, not just keyword matches. 4. Augmentation Phase: The system takes the user's original question AND the retrieved relevant chunks and stuffs them both into a new, single prompt. This prompt looks like: 'Using ONLY the following context, answer the question. Context: [The relevant document chunks]. Question: [User's question].' 5. Generation Phase: This enriched prompt is sent to the generative AI model. Because the prompt now contains the exact factual data needed, the model simply has to read and rephrase. It does not need to guess. Hallucination risk plummets. On the AI-102 exam, you must understand that RAG does not change the underlying AI model. It is a pattern for feeding the model the right information at inference time (the moment it answers). You will be asked to select the correct Azure services to implement RAG: Azure OpenAI for the model, Azure Cognitive Search as the retriever (often with vector search enabled), and a data source like Blob Storage for your documents. The exam also tests your understanding of how RAG differs from fine-tuning. Fine-tuning is when you take the base model and give it extra training on your own data, which changes the model's weights permanently. RAG is cheaper, faster to update, and less risky because you never modify the base model. You just control what it 'reads' before it speaks.
1. Data Ingestion: Prepare Your Knowledge Base
Collect all trusted documents (PDFs, Word files, internal wikis) and store them in a centralised Azure storage service like Blob Storage. This is your 'source of truth' that the AI will consult.
2. Chunking and Indexing: Make the Data Searchable
Break the documents into small, semantically meaningful chunks (e.g., paragraphs). Use Azure Cognitive Search to create an index. For each chunk, generate a vector embedding using an embedding model (like text-embedding-ada-002) and store it. This turns your documents into a searchable numerical map.
3. Receive the User Query
A user types a question (e.g., 'What is the refund policy for cancellations made within 24 hours?'). The application captures this query and passes it to the orchestration layer.
4. Retrieve Relevant Chunks: The 'R' in RAG
The orchestration layer converts the user query into a vector embedding and sends it to Azure Cognitive Search. The search engine finds the top K most semantically similar chunks from your index. This step is critical — if retrieval fails, the answer will be wrong.
5. Augment the Prompt: Add the Context
The orchestration layer constructs a new prompt. It adds a system message (e.g., 'You are a helpful assistant. Use only the provided context to answer.') and then injects the retrieved document chunks as context, followed by the user's original question.
6. Generate the Answer
The enriched prompt is sent to the Azure OpenAI model (e.g., GPT-4) with a low temperature setting (e.g., 0.0). The model reads the context and generates an answer based exclusively on that information. The result is returned to the user.
Imagine you work for 'Contoso Insurance,' a company with a vast library of 10,000 different policy documents. Customers call in asking about their specific coverage. Without RAG, a generative AI chatbot would confidently make up coverage rules, exposing the company to massive legal and financial risk. Here is what an IT professional (that is you, after you pass AI-102) actually does to set this up:
1. Identify the Knowledge Source: First, you find all the trusted documents. In this scenario, it is the PDFs of every insurance policy, stored in Azure Blob Storage. You also have a PDF of 'Frequently Asked Questions' that the legal team has approved. 2. Set Up Azure Cognitive Search: You create an Azure Cognitive Search service. You configure it to index the Blob Storage. You enable a feature called 'semantic search' or 'vector search' so it understands meaning, not just keywords. 3. Create the Indexing Pipeline: You build a 'skillset' that chunks the PDFs into smaller pieces (e.g., 500 tokens each, with 100 tokens of overlap). You then run this skill to generate vector embeddings for every chunk. These embeddings are stored as an index in Azure Cognitive Search. 4. Deploy the Azure OpenAI Model: You deploy a GPT-4 or GPT-3.5 model in Azure OpenAI Studio. You note its endpoint URL and API key. You set the 'temperature' parameter low (e.g., 0.0) for this use case because you want precise, deterministic answers. 5. Build the Orchestration Logic (The 'Brain'): You write code (likely using Python or C# with the Azure OpenAI SDK and Azure Cognitive Search SDK) that does the following:
- Receives a user's question: 'Does my policy cover flood damage if I live in a flat?' - Calls Azure Cognitive Search to find the top 3 most relevant policy document chunks. - Constructs a new prompt: 'You are a helpful insurance assistant for Contoso. Use ONLY the context below to answer. If the context does not contain the answer, say 'I do not know.' Do not make up any information. Context: [policy chunk 1], [policy chunk 2], [policy chunk 3]. Question: Does my policy cover flood damage if I live in a flat?' - Sends this prompt to the Azure OpenAI model. - Returns the model's answer to the user. 6. Monitor and Iterate: You set up logging to capture every question and answer. You review logs to see if the RAG system is retrieving the correct documents. If a user asks about 'water damage from a pipe burst' but the system only returns documents about 'flood from rivers,' you need to improve your chunking strategy or your embedding model. The key takeaway for your job: You are not an AI trainer. You are a 'knowledge architect.' Your main task is to curate the right data source and build the retrieval pipeline. The AI is the engine, but your design of the RAG system determines whether the car crashes or drives smoothly.
The AI-102 exam (Exam AI-102: Designing and Implementing a Microsoft Azure AI Solution) dedicates a significant portion of domain 4 ('Implement knowledge mining and document intelligence') and domain 1 ('Plan and manage an Azure AI solution') to prompt engineering and RAG. Here is exactly what they test. 1. Distinguishing RAG from Fine-tuning This is the single most tested concept. The exam loves to give you a scenario and ask: 'Should you use fine-tuning or RAG?' - The Traps: They will describe a situation where you have new, proprietary knowledge (e.g., your internal HR handbook). They will suggest fine-tuning. The wrong answer is fine-tuning. - The Right Pattern: If the requirement is to ground the model in your specific, frequently updated data WITHOUT retraining the base model, the answer is always RAG. Fine-tuning is for changing the model's style, tone, or learning a new format (like code generation), not for injecting factual knowledge. 2. Understanding the Components of a RAG Solution They will ask you to select the correct Azure services for a RAG pattern. You need to know:
- The AI model (generator): Azure OpenAI service (specifically a GPT model).
- The data store (your documents): Azure Blob Storage, Azure Cosmos DB, or Azure Data Lake.
- The retriever (search index): Azure Cognitive Search (especially its vector search and semantic search capabilities).
- The embedding model: Often, you use Azure OpenAI's text-embedding-ada-002 model to turn text into vectors.
3. Know the Prompt Engineering Parameters
The exam tests your knowledge of the API parameters you can set:
- Temperature: Controls randomness. Low (0.0-0.3) for factual Q&A. High (0.7-1.0) for creative writing.
- Top P (or nucleus sampling): Similar to temperature, controls the pool of words the model considers. Low Top P makes answers more focused.
- Max Tokens: The maximum length of the response. Not the same as word count. A token is about 4 characters.
- Stop Sequences: Specific strings that tell the model to stop generating (e.g., 'END').
4. The Concept of 'Grounding'
The exam uses the term 'grounding' constantly. Grounding means providing the AI with verifiable, trusted source material so it cannot hallucinate. RAG is the primary technique for grounding. They will ask: 'Which technique ensures the AI model only answers from a specific set of documents?' The answer is RAG with a system prompt instructing the model to only use the provided context.
5. Exam Question Styles
- Case Studies: They give you a long description of a business problem (e.g., a legal firm needs an AI to answer contract questions). You must choose the correct combination of services. The trap is picking a pre-trained model without a retrieval step.
- Multiple Choice on Parameters: 'A chatbot for customer complaints needs to be very consistent. Which temperature setting should you use?' Answer: 0.0.
- Drag and Drop (in the exam): Order the steps of a RAG pipeline. It is always: 1. Index documents, 2. Receive user query, 3. Retrieve relevant chunks, 4. Augment prompt with chunks, 5. Generate answer.
Blunt Advice: If the question mentions 'up-to-date data,' 'company-specific knowledge,' or 'avoid hallucination,' your brain should instantly think 'RAG.' If it mentions 'changing the model's personality' or 'teaching it a new language,' think 'fine-tuning.'
Prompt engineering is the skill of crafting precise instructions to guide an AI's behaviour and reduce hallucinations.
Retrieval-Augmented Generation (RAG) is an architecture that supplies an AI with relevant documents from your own data at query time.
RAG prevents hallucination by grounding the AI's answer in verifiable, real-time source material.
Fine-tuning changes the model itself; RAG does not modify the model, it only controls what data the model 'reads' before answering.
In Azure, you implement RAG using Azure OpenAI for the model, Azure Cognitive Search as the retriever, and Azure Blob Storage for the data.
The temperature parameter set to 0.0 is best for factual, consistent answers, while 1.0 is better for creative tasks like brainstorming.
A system prompt can enforce strict rules like 'if the answer is not in the context, say you do not know,' which is a key exam concept.
These come up on the exam all the time. Here's how to tell them apart.
RAG (Retrieval-Augmented Generation)
Injects factual knowledge at query time without retraining the model.
Cheaper and faster to update because only the document store changes.
Does not modify the base model, so the model remains generic and reusable.
Fine-tuning
Permanently changes the model's weights through retraining on new data.
Expensive and time-consuming to update; requires a new training run for any change.
Creates a custom model that loses the generic capabilities of the base model.
Low Temperature (0.0-0.3)
Produces deterministic, focused, and repetitive answers.
Ideal for factual Q&A, customer support, and legal applications.
Minimises creativity and variation, reducing risk of hallucination.
High Temperature (0.7-1.0)
Produces creative, varied, and surprising outputs.
Ideal for brainstorming, story writing, or generating marketing copy.
Increases risk of hallucination and inconsistency.
System Prompt
Sets the role and behavioural rules for the AI (e.g., 'You are a helpful assistant.').
Persists across the entire conversation session.
Used to enforce constraints like 'only use the provided context.'
User Prompt
Contains the specific user question or task for a single turn.
Is specific to each individual query.
Is the variable input that triggers the retrieval and generation steps.
Semantic Search (Vector Search)
Understands the meaning and intent behind the query, not just keywords.
Uses vector embeddings to find documents with similar thematic concepts.
Can find relevant information even if the exact words do not match.
Keyword Search (Lexical Search)
Matches exact words or phrases in the query against the index.
Is fast and simple but misses relevant content that uses different vocabulary.
Requires manual synonym mapping to expand query coverage.
Mistake
Prompt engineering is just about 'asking nicely' and does not require any technical skill.
Correct
Prompt engineering is a structured, technical skill involving specific techniques like few-shot prompting, chain-of-thought, and setting system messages with explicit constraints.
Because the interface is text-based, people assume it is just conversation. In reality, it is akin to writing precise, deterministic API instructions.
Mistake
RAG and fine-tuning achieve the same result, so you can use either one.
Correct
They solve fundamentally different problems. RAG adds factual knowledge at query time without modifying the model. Fine-tuning permanently changes the model's weights to alter its behaviour or style.
Many beginners see both as 'giving the AI data' and do not grasp the architectural difference between retrieval-time access versus training-time learning.
Mistake
Once you set up a RAG system, the AI will always give the right answer because it has all the documents.
Correct
RAG depends heavily on the quality of your document chunking, the relevance of retrieved chunks, and the prompt enforcing 'use only provided context.' It can still fail if retrieval retrieves the wrong chunks.
People overestimate the 'magic' of search and underestimate the engineering needed to make retrieval accurate. They assume having the data is enough, ignoring the retrieval quality.
Mistake
Prompt engineering is only useful for getting better answers, not for security.
Correct
Prompt engineering is a critical security layer. Well-structured prompts can prevent 'prompt injection' attacks, where a user tricks the AI into ignoring its instructions or revealing sensitive data.
Most beginners think of prompts purely as question-formulation, not as a defence mechanism against malicious users who try to jailbreak the AI.
Mistake
The temperature setting controls how 'intelligent' the AI is.
Correct
Temperature controls the randomness of the output, not the intelligence. Low temperature makes the model's answers more deterministic and focused; high temperature makes them more creative and diverse.
The word 'temperature' is abstract and non-intuitive. People attach their own meaning to it, often conflating it with general capability or confidence.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
No. Use Retrieval-Augmented Generation (RAG) instead. You keep the base model unchanged and simply feed it relevant documents at query time.
RAG adds knowledge temporarily at query time without changing the model. Fine-tuning permanently changes the model's weights through additional training. Use RAG for factual knowledge; use fine-tuning for changing the model's style or behaviour.
Grounding means providing the AI with a trusted, real-time source of truth (like a retrieved document) so it cannot hallucinate. RAG is the primary method to achieve grounding.
Yes, but in Azure, RAG is commonly implemented using Azure OpenAI and Azure Cognitive Search. The pattern works with any generative model that accepts a prompt containing context.
A vector embedding is a long list of numbers (a mathematical representation) that captures the 'meaning' of a piece of text. It allows the search engine to find documents with similar meaning, not just matching keywords.
A well-designed prompt instructs the model to say 'I do not know' if the context does not contain the answer. This prevents the AI from making up a false fact based on no information.
You've finished Generative AI: Prompt Engineering and Retrieval-Augmented Generation. Continue through the AI-102 study guide to build a complete picture of the exam.
Done with this chapter?