# Monitor Data Storage and Processing

> Chapter 12 of the Courseiva DP-203 curriculum — https://courseiva.com/learn/dp-203/monitor-data-storage-and-processing

**Official objective:** 3.3 — Secure, monitor, and optimize data storage and processing

## Introduction

How do you know if your data storage and processing systems are running smoothly, or if they are about to break down? This chapter explains the tools and practices Azure provides to watch over your data workloads, so you can spot problems early and keep everything running reliably for your users.

## The Home Security Camera System Analogy

First, you install security cameras around your house to watch for anything unusual, which leads to you being able to check the footage whenever you want to see who came and went, or if something went wrong.

Imagine you run a busy café from your home kitchen. You have a fridge full of ingredients (your data storage) and an oven that bakes cakes (your data processing). To make sure nothing ruins your business, you set up a few small cameras: one in the fridge to see if the door is left open too long and the milk goes warm (monitoring storage temperature, or latency), one over the oven to count how many cakes you bake each hour (monitoring processing throughput), and one at the door to log every delivery person who walks in (monitoring access, or security logs). These cameras don't stop a power cut or a broken mixer, but they give you a recording of exactly what happened, so you can fix the root cause fast. In the same way, Azure sends you metrics—like how fast data is being read from a database (IOPS) or how long a query takes to run—just like your camera feed. If the fridge camera shows the door was open for ten minutes, you know why your cream curdled. If Azure Monitor shows a sudden spike in processing time, you know your data pipeline has a bottleneck. The camera system itself is your monitoring tool, like Azure Monitor or Log Analytics, and the footage is the log data you can search through to troubleshoot problems before they ruin your cakes.

## Core explanation

Monitoring data storage and processing means continuously watching how your data systems behave. It is like having a health monitor for your digital fridge and oven. Without monitoring, you would only find out something broke when a customer complains they cannot access their report. With monitoring, you see the temperature rising before the milk spoils.

The core idea is that every Azure service that stores or processes data emits telemetry—that is just a fancy word for automatic measurements and logs. For example, Azure SQL Database sends metrics about how many queries it handles per second, how much CPU it uses, and how long each query takes. Azure Data Lake Storage sends metrics about how much data you are writing and reading each minute. These measurements give you a live picture of health and performance.

The main Azure tool for this is called Azure Monitor. Think of Azure Monitor as the control centre for all your camera feeds. It collects the telemetry from every Azure resource (a resource is anything you create in Azure—a storage account, a database, a virtual machine). Azure Monitor stores this information in two main places: Metrics and Logs. Metrics are numerical values tracked over time, like average CPU usage at 2 PM. Logs are detailed event records, like a note that says 'User John updated the customer table at 2:05 PM.'

To make sense of all this data, you use Log Analytics. This is a query tool inside Azure Monitor, where you write Kusto Query Language (KQL) sentences to search through logs. For example, you could query 'Show me all failed logins in the last hour' and get a list back instantly. For viewing metrics visually, you use Azure Monitor dashboards and Workbooks, which are like custom scoreboards showing your key performance indicators (KPIs) such as storage latency or query success rate.

For specific types of storage, there are specialised monitoring features. Azure Storage Analytics gives you metrics for blob, file, queue, and table storage—things like total requests, average egress (data leaving Azure), and error counts. It also logs every authenticated request, so you can audit who accessed what. For data processing pipelines built with Azure Data Factory or Synapse Pipelines, you monitor the pipeline runs, see which activities succeeded or failed, and check how long each step took. Synapse Studio has a built-in monitoring hub that shows you a list of all running and completed Spark jobs and SQL queries.

Alerts are the final critical piece. You set up alert rules that say, 'If CPU usage stays above 90% for five minutes, send an email.' This way, you do not have to stare at dashboards all day; the system wakes you up when something needs attention. You can send alerts to email, SMS, or even trigger an automated action like scaling up a database.

Before cloud monitoring, IT teams had to manually check servers. If a disk filled up overnight, nobody knew until someone arrived in the morning and found the application broken. Monitoring replaces that blind guesswork with real-time, always-on visibility, so you can prevent downtime instead of just reacting to it.

## Real-world context

Imagine Maria works as a data engineer for an online retail company. Every night, a pipeline built in Azure Data Factory (ADF) processes millions of sales transactions from the day and loads them into Azure Synapse Analytics for reporting. One morning, the sales director cannot open the daily dashboard because the data from last night is missing. Maria needs to figure out what went wrong.

First, she opens Azure Monitor and looks at the 'Pipeline runs' view in Data Factory. She sees that last night's pipeline failed at the step called 'TransformSalesData.' She clicks on that failed run and sees the error message: 'Out of memory.' This tells her the processing job tried to use more memory than was allocated.

To understand why, Maria checks the metrics for the Synapse SQL pool that was being written to. She opens Azure Monitor Metrics and selects 'DWU (Data Warehouse Unit) usage.' She sees a spike to 100% at exactly the time the pipeline was writing, meaning the data warehouse had no capacity left. Further, she looks at the 'Wait' metrics—how long queries had to wait for resources—and sees a high wait time. That confirms the pipeline was competing with other queries.

Maria then sets up an alert: an email notification if Synapse DWU usage exceeds 80% for more than 2 minutes. She also schedules the pipeline to run at 2 AM, when fewer users are querying the warehouse. She adds a diagnostic setting to the storage account that logs all read/write operations, so if a similar issue happens, she can see exactly which files were being accessed.

The real work does not stop at fixing one problem. Maria creates a dashboard in Azure Monitor Workbooks that shows four key cards: number of failed pipeline runs this week, average query execution time, storage account latency, and number of active alerts. She shares this dashboard with her team so everyone can spot trends before they cause failures.

The step-by-step process Maria follows each morning is: check the Alert inbox, open the pipeline runs monitoring view, examine any failed runs for error messages, review related resource metrics (CPU, memory, IOPS), and finally adjust configurations or scale resources to prevent recurrence. This is exactly what DP-203 expects you to be able to do: use Azure Monitor and its child tools to investigate and resolve performance and availability issues in data storage and processing.

## Exam focus

The DP-203 exam tests monitoring in three main areas: knowing what metrics and logs each Azure data service emits, understanding how to set up alerts and diagnostic settings, and being able to interpret monitoring data to troubleshoot problems.

First, the exam asks you to identify which monitoring tool fits a given scenario. For example, a question might say: 'You need to see a historical chart of average query duration for an Azure SQL database over the last month.' The correct answer is 'Metrics in Azure Monitor' because metrics store numerical time-series data. A trap answer would be 'Log Analytics'—logs store detailed events but not aggregated numeric charts. Another common question: 'You need to search for all delete operations on a storage account from a specific user.' That requires 'Storage logs in Log Analytics' because you need to query the detailed request logs.

Second, the exam loves diagnostic settings. You must know that to send platform logs from a resource (like a storage account) to Log Analytics, you enable Diagnostic Settings on that resource and choose the destination. A common trap is a question that says 'Enable diagnostic logs on the storage account' but then offers 'turn on firewall rules' as a distractor. Firewall rules are about security, not monitoring.

Third, you need to memorise specific monitoring features for key services:
- Azure Storage: 'Storage Analytics logs and metrics'—includes 'SuccessE2ELatency' and 'Availability' metrics.
- Azure Synapse Analytics: 'DMVs (Dynamic Management Views)' like sys.dm_pdw_exec_requests to see running queries.
- Azure Data Factory: 'Monitor pipeline runs' and 'Activity runs' views.
- Azure Data Lake Storage: 'Azure Monitor integration' for hierarchical namespace metrics.

The exam often presents a scenario with slow performance and asks which metric to check first. Correct pattern: check 'Average IOPS' for disk bottlenecks, 'DTU or DWU consumption' for database limits, or 'Pipeline run duration' for processing delays. Wrong patterns: checking network bandwidth for a database query slowdown—that is a distractor because network is rarely the bottleneck in same-region queries.

Another favourite: alerts. You must know that alert rules consist of a condition (e.g., metric > threshold), a severity (Sev 0-4), and an action group (list of recipients or automation runbooks). A trap question will ask you to create an alert but offer 'Create a Log Analytics workspace' as a step—you need the workspace to store logs, but you do not need to create one if you already have one.

Finally, the exam expects you to know the difference between basic and advanced monitoring. Basic monitoring is free and included with every resource (like activity logs, which record who created/deleted resources). Advanced monitoring requires enabling diagnostic settings and paying for log ingestion. A question will present a cost-conscious scenario: 'You want to collect SQL server logs but minimise costs.' The answer would be 'Use basic metrics and activity logs only' or 'Send only error logs, not all logs.'

## Step by step

1. **Enable diagnostic settings** — For each Azure data resource (storage account, SQL database, Synapse workspace), go to 'Diagnostic settings' and turn on sending metrics and logs to a Log Analytics workspace. Without this step, monitoring data will not be collected.
2. **Set up metric alerts** — Create alert rules for critical thresholds like high DTU consumption (above 80%) or high storage latency (above 10 ms). Define an action group (email your team or trigger a runbook) so you are notified automatically.
3. **Build a monitoring dashboard** — Use Azure Monitor Workbooks to create a live view showing pipeline success rate, top resource consumers, and recent alert history. This gives you a quick snapshot every morning without running individual queries.
4. **Query logs with KQL** — Write Kusto queries to search for specific events, such as failed logins or storage delete operations. Practise common queries like counting errors per hour or listing all operations by a specific user.
5. **Analyse pipeline runs** — In Azure Data Factory or Synapse Pipelines, open the 'Monitor' tab to see each pipeline run's status, duration, and activity-level errors. Click into a failed run to see the exact error message and input/output parameters.
6. **Review and adjust thresholds** — After observing monitoring data for a week, adjust alert thresholds to reduce noise (false alarms) and ensure critical issues are caught early. This is an ongoing improvement cycle.

## Comparisons

### Azure Monitor Metrics vs Azure Monitor Logs

**Azure Monitor Metrics:**
- Numeric data points (e.g., CPU %, IOPS)
- Stored in a time-series database for fast charting
- Best for real-time dashboards and alert rules

**Azure Monitor Logs:**
- Text event records (e.g., error messages, user actions)
- Stored in Log Analytics workspace as tables
- Best for forensic search with KQL queries

### Activity Logs vs Resource Diagnostic Logs

**Activity Logs:**
- Record management-plane events (e.g., create/delete resource)
- Generated automatically for all resources at no cost
- Cannot be disabled, retained for 90 days by default

**Resource Diagnostic Logs:**
- Record data-plane events (e.g., read/write to storage blob)
- Must be manually enabled via diagnostic settings
- Can be sent to Log Analytics, storage, or Event Hub

### Azure Data Factory Pipeline Runs vs Synapse Pipeline Runs

**Azure Data Factory Pipeline Runs:**
- Orchestrates activities across multiple services
- Monitored in ADF Monitor tab
- Shows pipeline status, activity runs, and trigger history

**Synapse Pipeline Runs:**
- Orchestrates activities within Synapse analytics
- Monitored in Synapse Studio Monitor hub
- Also shows Spark job and SQL script execution status

### Alert Rule Condition vs Alert Rule Action Group

**Alert Rule Condition:**
- Defines when the alert fires (e.g., metric > 90% for 5 min)
- Evaluated by Azure Monitor every minute
- Can be a metric condition or log search result

**Alert Rule Action Group:**
- Defines what happens when alert fires (e.g., send email)
- Contains a list of notification channels and/or automation actions
- Can be reused across multiple alert rules

## Diagram

_Flowchart showing how Azure resources send telemetry to Azure Monitor, which splits it into metrics and logs, then feeds dashboards and alerts._

```mermaid
flowchart TD
    A[Azure Resources] -->|Emit metrics and logs| B[Azure Monitor]
    B --> C[Metrics]
    B --> D[Logs]
    C -->|Create dashboards| E[Workbooks]
    C -->|Configure alerts| F[Alert Rules]
    D -->|Query| G[Log Analytics]
    F -->|Notify| H[Action Group: Email/SMS/Runbook]
    G -->|Result| I[KQL Query Results]
    E --> J[Monitoring Dashboard]
```

## Common misconceptions

- **Misconception:** Monitoring just means checking a dashboard once a day to see if everything is green. **Reality:** Monitoring involves continuous collection of metrics and logs, automated alerts that notify you of issues in real-time, and proactive analysis to find trends before they become problems. (Beginners think dashboards are passive status boards, but monitoring is an active, automated process that wakes you up when something breaks.)
- **Misconception:** If a metric is below the threshold, the system is fine and there is no issue. **Reality:** Some problems, like an implicit throttle or a slow degradation, do not trigger a high-metric alarm. You also need to check error counts, wait times, and query plan changes. (People assume green numbers equal perfect health, but silent failures like slow queries or increased latency can happen without breaking any static threshold.)
- **Misconception:** Log Analytics is the same as Azure Monitor Metrics, just stored differently. **Reality:** Metrics are numerical time-series data optimised for real-time charting and alerts. Log Analytics stores text-based events and allows complex KQL queries for deep forensic analysis. They serve different purposes. (Both come from Azure Monitor, so beginners conflate them, but the exam expects you to know exactly when to use each one.)
- **Misconception:** Enabling diagnostic logs on a storage account will automatically alert you if a file is deleted. **Reality:** Diagnostic logs capture the event of the deletion, but you must also set up an alert rule that queries those logs to trigger an alert. Logging alone does not notify anyone. (People think 'if I turn on logging, problems will magically notify me.' You have to deliberately connect logs to alerts or create a scheduled query rule.)
- **Misconception:** Monitoring a single resource is enough to ensure the whole pipeline is healthy. **Reality:** A data pipeline involves multiple resources (storage, compute, database, network). You must monitor each component and the end-to-end flow, because a delay in one part can cascade. (Newbies focus on one box, forgetting that data moves through a chain. The exam tests end-to-end thinking with multi-resource scenarios.)

## Key takeaways

- Azure Monitor is the central tool for collecting, storing, and analysing metrics and logs from all Azure data services.
- Metrics are numerical values tracked over time (like CPU percentage) and are best for real-time dashboards and alert rules.
- Logs are detailed event records (like a specific error message) and are best for deep forensic querying using Kusto Query Language (KQL).
- Diagnostic settings must be explicitly enabled on each resource to send its platform logs to a Log Analytics workspace.
- Alerts consist of a condition (metric or log search), a severity rating, and an action group that defines who gets notified.
- To troubleshoot a failed pipeline, check the pipeline runs monitor, examine error messages, then review metrics of dependent resources to find the root cause.

## FAQ

**What is the difference between metrics and logs in Azure Monitor?**

Metrics are numeric data points (like CPU usage percentage) collected at regular intervals and stored for quick charting. Logs are detailed event records (like 'User X deleted file Y at time Z') that you can search using KQL.

**Do I need to enable diagnostic settings on every storage account?**

Yes. By default, diagnostic settings are disabled for most resources. You must manually enable them for each storage account, database, or compute resource to send logs to a Log Analytics workspace or storage account for retention.

**Why can't I see any pipeline runs in Azure Data Factory Monitor?**

You might be looking at a different Data Factory instance, or the pipeline may never have been triggered. Check that you are in the correct Data Factory resource and that the pipeline has been executed at least once.

**How do I create an alert when a storage blob is deleted?**

First, enable diagnostic logs for the storage account and send them to Log Analytics. Then create a scheduled query alert rule that runs a KQL query like 'StorageBlobLogs | where OperationName == 'DeleteBlob'' and triggers an action group.

**What does 'DWU consumption' mean in Synapse metrics?**

DWU stands for Data Warehouse Unit, a measure of compute resources in Synapse. High DWU consumption means your queries are using a lot of processing power, which can cause queuing if it hits 100%.

**Can I monitor on-premises data storage with Azure Monitor?**

Yes, using Azure Arc enabled servers. You can install the Azure Monitor agent on on-premises machines to collect metrics and logs, as long as they have network connectivity to Azure.

## Check your understanding

1. **What Azure service is the central collection point for all metrics and logs from Azure resources?**

   Answer: Azure Monitor is the central service that collects, stores, and provides analytics over metrics and logs from all Azure resources.

2. **You need to search for all delete operations that happened on a storage account in the last hour. Should you use Metrics or Logs?**

   Answer: You should use Logs (specifically Log Analytics), because logs contain detailed events like delete operations, while metrics only track numeric values over time.

3. **What must you enable on a resource before its platform logs are sent to a Log Analytics workspace?**

   Answer: You must enable Diagnostic Settings on the resource and configure the destination as the Log Analytics workspace.

4. **What three components make up an alert rule in Azure Monitor?**

   Answer: An alert rule consists of a condition (e.g., metric threshold), a severity level (Sev 0-4), and an action group that defines who gets notified or what automation runs.

5. **Name one metric you would check to determine if an Azure SQL database is overworked. kql**

   Answer: DTU consumption or CPU percentage are key metrics to check if a SQL database is overworked, because they indicate how close the database is to its resource limits.

---

Interactive version with quiz and diagrams: https://courseiva.com/learn/dp-203/monitor-data-storage-and-processing