System Messages and Grounding Prompts

System messages and grounding prompts are two mechanisms to control the behavior of large language models (LLMs) in Azure OpenAI Service. They are part of the broader concept of prompt engineering, which is the practice of crafting inputs to elicit desired outputs from generative AI models.

System messages are instructions that set the overall context, tone, and rules for the model's responses. They are sent as part of the API call's system role in the chat completions endpoint. The system message persists throughout the entire conversation session, influencing every response the model generates. For example, a system message might state: "You are an AI assistant that helps users find information about Azure services. Always respond in a professional tone and cite your sources."

Grounding prompts (or grounding data) are external sources of information provided to the model to anchor its responses in facts, reducing the likelihood of hallucinations. In Azure OpenAI, grounding is achieved through the Azure OpenAI On Your Data feature, which connects the model to a data source (e.g., Azure Cognitive Search index, blob storage, or a database). The model retrieves relevant chunks from that data source and uses them as context when generating responses. Grounding prompts are not part of the system message; they are injected into the prompt as context or retrieved via a retrieval-augmented generation (RAG) pattern.

LLMs are trained on vast public datasets but have no inherent knowledge of proprietary data, recent events, or specific business rules. Without guidance, they can produce outputs that are off-brand, factually incorrect, or unsafe. System messages provide a way to constrain the model's behavior without retraining. Grounding prompts ensure the model's responses are based on authoritative data rather than its internal (and potentially outdated) knowledge.

When you call the Azure OpenAI chat completions API, you send an array of messages with roles: system, user, and assistant. The system message is typically the first element and sets the stage. For example:

{
  "messages": [
    {"role": "system", "content": "You are a helpful assistant. Answer only from the provided context. If you don't know, say you don't know."},
    {"role": "user", "content": "What is the capital of France?"}
  ]
}

The model processes the entire conversation history (including the system message) as a single sequence of tokens. The system message is not hidden from the model; it is part of the input. The model learns to follow the system instruction because it has been trained on examples where similar instructions were used. In practice, the system message is most effective when it is clear, concise, and placed at the beginning of the conversation.

Key parameters: - temperature: Controls randomness. Lower values (e.g., 0.0) make the model more deterministic, which is desirable for grounded responses. - top_p: Nucleus sampling. Also controls randomness. - max_tokens: Limits the length of the response. - stop: Sequences that stop generation.

Grounding via Azure OpenAI On Your Data uses a RAG architecture: 1. Indexing: Your data is indexed into a search service (e.g., Azure Cognitive Search). The index stores chunks of text with embeddings. 2. Query: When a user asks a question, Azure OpenAI converts the user's query into an embedding vector. 3. Retrieval: The vector is used to search the index for semantically similar chunks. The top-k (e.g., 5) chunks are retrieved. 4. Augmentation: Those chunks are inserted into the prompt as context, typically after the system message and before the user's question. The model then generates a response based on that context. 5. Response: The model's output is constrained by the retrieved data, reducing hallucinations.

The grounding prompt is not a separate API parameter; it is the combination of the retrieved chunks and the system message. You can also manually provide grounding data by including it in the user message (e.g., "Based on this article: [text], answer: ...").

To configure system messages and grounding in Azure OpenAI, you use the Azure Portal or API.

Azure Portal (Chat Playground): 1. Go to Azure OpenAI Studio. 2. Select your deployment. 3. In the Chat Playground, you can set the system message in the "System message" field. 4. To enable grounding, click "Add your data" and connect to a data source (e.g., Azure Cognitive Search). 5. Configure the retrieval parameters (e.g., number of chunks, strictness).

API Call (Python SDK):

import openai

openai.api_type = "azure"
openai.api_base = "https://your-resource.openai.azure.com/"
openai.api_version = "2023-12-01-preview"
openai.api_key = "your-key"

response = openai.ChatCompletion.create(
  engine="gpt-4",
  messages = [
    {"role": "system", "content": "You are an AI assistant that helps with Azure. Only answer from the provided context."},
    {"role": "user", "content": "What is Azure Cognitive Search?"}
  ],
  temperature=0.0,
  max_tokens=500
)

To verify that grounding is working, check the response's context field (if using On Your Data) which contains the retrieved chunks.

System messages and grounding prompts work alongside: - Content filtering: Azure OpenAI has built-in content filters that can block harmful outputs. System messages can also instruct the model to avoid certain topics. - Prompt engineering: System messages are a core prompt engineering technique. Others include few-shot learning (providing examples) and chain-of-thought prompting. - Fine-tuning: System messages can be used even with fine-tuned models. However, fine-tuning changes the base model, so system messages may need adjustment. - Azure AI Content Safety: Can be integrated to add an additional layer of filtering on top of system messages.

Enterprise Scenario 1: Customer Support Chatbot for a Software Company A software company deploys a customer support chatbot using Azure OpenAI. They have a large knowledge base of troubleshooting articles, release notes, and FAQs. The system message is: 'You are a support agent for Acme Software. Only answer from the provided knowledge base. If you don't have an answer, suggest opening a support ticket.' Grounding is configured with Azure Cognitive Search indexing all support articles. The chatbot handles thousands of queries daily. A common issue is that the model may retrieve irrelevant chunks due to poor indexing or vague queries. To mitigate, the team sets strictness to 4 and uses semantic search. They also add a fallback system message: 'If the context does not contain the answer, say: I need to escalate this to a human agent.' Performance considerations: latency increases with the number of retrieved chunks; they use 3 chunks to balance speed and accuracy.

Enterprise Scenario 2: Internal HR Policy Assistant A large enterprise uses an AI assistant to answer employee questions about HR policies. The data source includes PDFs of employee handbooks, benefit documents, and compliance guidelines. The system message is: 'You are an HR assistant. Provide accurate information based on the company's official policies. Do not give legal advice or speculate.' Grounding is essential to avoid outdated or incorrect answers. The HR team updates the data source quarterly. A misconfiguration occurred when the indexing pipeline failed to update after a policy change, causing the assistant to give old information. To prevent this, they set up a scheduled indexer with a 24-hour refresh. They also monitor the assistant's responses for policy citations. The strictness is set to 5 to ensure the model never fabricates policies.

Enterprise Scenario 3: Medical Information Bot (Non-diagnostic) A healthcare provider creates a bot that answers general medical questions using trusted medical textbooks and guidelines. The system message explicitly states: 'You are an informational assistant. Do not provide diagnoses or treatment recommendations. Always advise users to consult a doctor.' Grounding uses a curated set of medical literature. A critical issue is that the model might retrieve a chunk that is out of context and provide misleading information. To address this, the team uses a custom chunking strategy with 512-token chunks and 50-token overlap, and they manually review the top 10 retrieved chunks for quality. They also implement a content filter that blocks any response containing certain keywords (e.g., 'diagnosis'). The system message includes: 'If the context is ambiguous, say: Please consult your healthcare provider.'