Without scheduled searches and alerts, you would have to sit at your desk staring at log files 24 hours a day, waiting for something to break. That is obviously impossible, so Splunk does the waiting for you. In the SPLK-1003 exam, you must know how to tell Splunk to run searches automatically at specific times and then notify you when something important happens.
Jump to a section
A simple way to picture Scheduled Searches and Alerts
At exactly 07:55 every weekday, 1 minute before your alarm goes off, your automatic coffee machine grinds 30 grams of beans, heats 500ml of water to 95 degrees Celsius, and fills your favourite mug. You do not stand there pressing buttons each morning. You programmed it once. This is exactly how a scheduled search works. You tell Splunk, 'At exactly 08:00 every morning, look at the logs from the last 24 hours and count how many times the error code XYZ appeared.' Splunk does this automatically, on its own, without you having to manually run the search each time.
Now imagine that yesterday the error count jumped from a normal 2 to 200. You need to know immediately, not tomorrow morning. This is where alerts come in. You add a condition to your scheduled search: 'If the count is greater than 50, send me an email right now.' Your coffee machine has a burst pipe? You want an alert, not a scheduled brew. The coffee machine is your scheduled search. The leak detector is your alert. One runs on a schedule for information. The other runs on a schedule but additionally checks a condition and acts instantly.
Scheduled searches are pre-written Splunk queries that the system runs automatically at a time you define, without any human clicking the search button. You set a schedule like 'every hour' or 'every Monday at 9am'. The search runs itself. The result is saved, and you can look at it later.
An alert is a scheduled search with a condition attached. The condition is called a trigger condition. When the scheduled search finishes, Splunk checks its results against this condition. If the condition is true, Splunk performs an action. That action is called the alert action. Common alert actions include sending an email, running a script, or posting a message to a web service (webhook).
To set up a scheduled search, you start from any regular search in Splunk. You click 'Save As' and choose 'Alert'. A window opens with several settings you must configure.
Title – A name for your alert, like 'High Error Rate Alert'. Description – Optional but useful text explaining what the alert does. Permissions – Controls who can see and edit the alert.
Next, you set the schedule. You have two options: - Run on Cron Schedule – This uses a cron expression. A cron expression is a string of five numbers and symbols that defines exactly when to run. For example, '0 */2 * * *' means 'run every two hours at minute zero'. You do not need to memorise cron syntax for the exam, but you must know that this option exists and that it gives you the most precise control. - Schedule Interval – This is simpler. You pick from dropdowns: 'Run every 10 minutes', 'Run every hour', 'Run daily', etc.
Then you set the Time Range. This tells Splunk how far back to look each time the search runs. If you schedule a search to run every hour but set the time range to 'last 60 minutes', each run looks at the data from that specific hour. The time range is relative to the time the search runs.
Next is the Trigger Condition. This is the most important part for the exam. You have three options:
- Number of Results – Trigger if the search returns at least X results. Example: 'Trigger if there are more than 10 results'.
- Custom Condition – Write a condition using a Splunk search language function like if or where. Example: search count > 50.
- Always – Trigger every time the search runs, regardless of results.
Finally, you configure the Alert Actions. You can choose one or more actions: - Email – Sends an email to one or more people. You can include the search results in the email body or as an attachment. - Webhook – Sends a message to a web service URL, like Slack or a custom API. - Run a Script – Executes a custom script on the Splunk server.
When the alert triggers, Splunk immediately executes the actions you configured. The alert also creates a record in the Triggered Alerts page, which you can view later.
Scheduled searches and alerts replace constant manual monitoring. Without them, you would need a person watching the logs every second. With them, you can go home at 5pm and wake up to an email if something broke at 3am. In the exam, you need to know the difference between a scheduled search and an alert, the settings in the alert creation window, and how trigger conditions work.
Write Your Search
Start by writing a search in the Splunk search bar that answers a specific question, like 'How many 404 errors occurred in the last hour?' This search will be the basis of your alert. In the exam, you need to know that any valid SPL query can become an alert.
Save As Alert
Click the 'Save As' button above the search results and select 'Alert'. This opens the Alert creation window. This step transitions a one-time search into a persistent, automated monitor. The exam tests that you know the correct menu option to create an alert.
Configure the Schedule
Set how often the search runs. Choose either a Schedule Interval (e.g., every 10 minutes) or a Cron Schedule for exact times (e.g., every Monday at 9am). The exam will test your understanding of when to use each option, especially that cron allows specific days and times.
Set the Time Range
Define which data the search examines each time it runs. For example, if the schedule is every hour, set the time range to 'Last 60 minutes'. This ensures each run looks at a fresh, non-overlapping window. The exam loves to trap you with mismatched time ranges and schedules.
Define the Trigger Condition
Choose when the alert should fire. For most alerts, you use 'Number of Results > X' to trigger when a metric exceeds a threshold. Alternatively, you can write a custom SPL condition or choose 'Always'. This step determines whether the alert becomes meaningful or useless. The exam quizzes you on the difference between the three options.
Add Alert Actions and Throttling
Select what happens when the alert triggers: send an email, call a webhook, or run a script. Also set throttling to limit how often notifications are sent, e.g., once per hour. The exam tests your knowledge of each action type and the purpose of throttling.
Review and Save
Give the alert a name, description, and permissions. Check all settings once more, then click Save. After saving, you can view the alert on the 'Settings > Searches, Reports, and Alerts' page. The exam may ask where you find your saved alerts or how to edit them.
An IT professional named Sam supports a company's online store. The store has a log file called 'checkout.log' that records every time a customer tries to pay. If the checkout process fails, an error message 'PAYMENT_FAILED' appears. Sam needs to know quickly if many customers are failing to pay, because that means lost revenue.
Sam creates a scheduled search that runs every 10 minutes. The search looks at the last 10 minutes of data from checkout.log and counts how many lines contain 'PAYMENT_FAILED'. Sam saves this as an alert.
In the alert configuration, Sam sets the trigger condition to 'Number of Results > 5'. This means: if more than 5 payment failures happen in 10 minutes, something is probably wrong. Sam sets the alert action to send an email to the team's distribution list.
Now let's say it's 2am. A software bug starts causing every payment attempt to fail. Between 2:00 and 2:10, twenty customers get an error. At 2:10, Splunk runs the scheduled search. It finds 20 results. That is greater than 5, so the alert triggers. Splunk sends an email to the team. At 2:11, Sam's phone buzzes. Sam sees the email subject: 'Alert: High Payment Failure Rate'. Sam logs in from home, sees the spike, and calls the on-call engineer. By 2:45, the bug is fixed. Without the alert, nobody would have noticed until 9am, and hundreds of customers would have been unable to buy anything.
In real companies, IT professionals also use alerts for: - Disk space – Trigger when disk usage exceeds 90%. - Server downtime – Trigger if a server stops sending logs for more than 5 minutes. - Security violations – Trigger if a user tries to log in 10 times with the wrong password in one minute. - Business metrics – Trigger if the number of completed orders drops by 50% compared to the same time yesterday.
Sam also uses the Triggered Alerts page to review old alerts. This page shows a history of every time the alert fired, including the time and the search results. This is useful for auditing and troubleshooting.
One important detail: Sam must set the correct time range. If Sam sets the time range to 'last 60 minutes' but the schedule is 'every 10 minutes', each search looks at the same 60-minute window overlapping with previous runs. Sam would get duplicate results. The correct configuration is to match the time range to the schedule interval: every 10 minutes looks at last 10 minutes' data.
Scheduled searches also have a Backfill feature. When you enable backfill, Splunk runs the search for any missed scheduled times. For example, if you scheduled a search to run every hour but the Splunk server was down for 3 hours, when it comes back up, it can run the search for those 3 missed hours. This is useful for reporting, but for alerts you usually do not want backfill, because you do not want to be alerted about a problem that happened hours ago.
The SPLK-1003 exam tests your ability to configure scheduled searches and alerts correctly, not just to understand the concept. Expect multiple-choice questions that present a scenario and ask you to choose the correct settings.
Key exam topics: - The difference between a scheduled search and an alert. A scheduled search just runs and saves results. An alert also checks a condition and performs actions. - The three trigger condition types: Number of Results, Custom Condition, Always. Know which to use in each scenario. - Alert action types: Email, Webhook, Run a Script. You need to know what each does. - Cron schedule versus Schedule Interval. You should know that cron gives more precise control. - Understanding the time range relative to the schedule. This is a common trap. - The concept of throttling. Throttling prevents the alert from triggering too frequently. For example, if an error happens every minute but you set throttling to 1 hour, Splunk will only send one email per hour. You can set throttle based on a field value, like hostname. - Severity levels. You can assign a severity (low, medium, high, critical) to alerts. This helps prioritise response.
Common traps the exam sets: - They give you a scenario where the schedule is every 5 minutes but the time range is 'last hour'. The correct answer is that this is wrong because the searches will overlap. The correct time range should match the schedule interval. - They ask: 'What happens if you set a scheduled search with no trigger condition?' The answer: It is just a scheduled search, not an alert. It will save results but not notify anyone. - They ask: 'Which alert action sends a message to a chat service like Slack?' Answer: Webhook. - They ask: 'What is the default throttle interval?' Answer: There is no default; you must set it explicitly, but common intervals are 1 hour or 24 hours. The exam may ask you to select a throttle interval that makes sense for a given scenario.
Concepts you must memorise: - The exact names of the trigger conditions: 'Number of Results', 'Custom Condition', 'Always'. - The exact names of alert actions: 'Email', 'Webhook', 'Run a Script'. - The meaning of cron expression and when to use it. - The purpose of backfill and when it is appropriate (reporting, not alerts). - The difference between per-result and per-time-window alerting (you typically use per-time-window for summarised metrics).
If you see a question about 'Alert on a scheduled search', the answer usually involves configuring the trigger condition and alert actions. Do not confuse scheduled searches with real-time alerts. Real-time alerts trigger within seconds of a matching event entering Splunk. Scheduled alerts run on a timer. The exam explicitly tests the difference.
Finally, know that you can edit an existing alert. You can change the schedule, the trigger condition, or the actions. You do not need to delete and recreate. This is a common trick: they ask 'How do you change the email recipient of an existing alert?' The answer is to edit the alert, not create a new one.
A scheduled search runs automatically on a timer, while an alert adds a trigger condition and action to that search.
You must match the time range of a scheduled search to the schedule interval to avoid overlapping or missing data.
The three trigger conditions are Number of Results, Custom Condition, and Always.
Throttling prevents an alert from sending too many notifications in a short period; you configure it explicitly.
Alert actions include Email, Webhook, and Run a Script, each serving a different notification or automation purpose.
A cron schedule lets you define exact minutes, hours, or days for running a search, which is more precise than the Schedule Interval dropdown.
Do not enable backfill for alerts because alerting on past events is useless; backfill is only for scheduled reports.
Real-time alerts trigger instantly when data arrives, whereas scheduled alerts run at fixed intervals, regardless of when data arrives.
These come up on the exam all the time. Here's how to tell them apart.
Scheduled Search
Runs automatically on a set timer.
Saves results without checking any condition.
Used for regular reporting or data gathering.
Alert
A scheduled search with a trigger condition attached.
Performs an action only when the condition is true.
Used for proactive monitoring and immediate notification.
Schedule Interval
Simple dropdown to pick every X minutes/hours/days.
Less flexible; cannot set specific days of month.
Easier for beginners to configure quickly.
Cron Schedule
Uses a 5-field cron expression for precise timing.
Can run on specific weekdays or month days.
More powerful but requires understanding cron syntax.
Number of Results Trigger
Triggers when search result count exceeds a threshold.
Simplest to configure: just enter a number.
Cannot evaluate fields or calculated values within results.
Custom Condition Trigger
Uses a Splunk search condition like 'where avg > 100'.
Can evaluate specific fields, calculations, or complex logic.
Requires writing SPL; more powerful but more complex.
Throttling Enabled
Limits notifications to one per set time window.
Prevents email floods during repeated triggers.
Best for high-frequency alerts to avoid spam.
Throttling Disabled
Every trigger sends a notification immediately.
Risk of overwhelming recipients with repeated messages.
Suitable for critical alerts that need every instance notified.
Mistake
A scheduled search and an alert are the same thing.
Correct
A scheduled search simply runs a query on a schedule and saves the results. An alert is a scheduled search that also has a trigger condition and performs an action when the condition is met.
The words 'scheduled search' and 'alert' are often used interchangeably in casual conversation, but the exam defines them differently. Beginners see the same creation window and assume they are identical.
Mistake
Setting the time range to 'last 30 minutes' on a search that runs every hour will only look at the most recent 30 minutes of data each time.
Correct
It will look at the 30 minutes immediately before the search runs. If you run it every hour, you get a 30-minute window that shifts each hour, missing the other 30 minutes of data. You should match the time range to the schedule interval.
Beginners often think 'last 30 minutes' means 'the last 30 minutes of data in the system' rather than 'the 30 minutes leading up to now'. This causes them to configure overlapping or incomplete data windows.
Mistake
If an alert triggers, it will keep sending emails every second until someone turns it off.
Correct
By default, alerts do not throttle. However, you can configure throttling to limit how often the alert sends notifications, typically once per hour or once per day. Without throttling, an alert that fires every time the search runs could send many emails quickly.
Beginners have seen stories of email floods from monitoring systems and incorrectly assume this is the normal, unchangeable behaviour. They do not know throttling exists as a configurable option.
Mistake
You can only use 'Number of Results' as a trigger condition.
Correct
Splunk offers three trigger conditions: Number of Results, Custom Condition (using SPL), and Always. Number of Results is the most common, but the others exist for different use cases.
Beginners often learn 'trigger when there are more than X results' first because it is the simplest. They then never explore the other options and assume that is the only way.
Mistake
If you set an alert to run every 10 minutes, it looks at data from the past 10 minutes without any overlap.
Correct
It looks at the last 10 minutes from the exact time the search starts. If the search takes 30 seconds to run, the next run starts 10 minutes after the previous start, so there is a 30-second gap of unsearched data. This is usually acceptable, but it can cause missed events if you need complete coverage.
Beginners think the search window is perfectly aligned with the schedule, but Splunk schedules are based on start time, not end time. This leads to misunderstandings about data completeness.
Mistake
Alerts are only useful for IT problems like server crashes.
Correct
Alerts can monitor any metric in your data, including business metrics like sales conversion rates, customer signup rates, or error counts in a mobile app. They are used by marketing, finance, and product teams, not just IT.
The term 'alert' sounds technical and urgent, so beginners associate it exclusively with IT operations. In reality, Splunk is used across entire companies for all kinds of monitoring.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
No, Splunk's minimum schedule interval is 1 minute. You cannot set a scheduled search to run every 30 seconds. If you need faster monitoring, use a real-time alert instead.
A report is a saved search that generates a visualisation or table you can view later. An alert is a scheduled search that checks a condition and performs an action. A report does not trigger actions; an alert does.
Yes. Go to Settings > Searches, Reports, and Alerts, find your alert, and click Edit. You can change any setting except the search itself in some cases. You do not need to recreate the alert for minor changes.
If the trigger condition is 'Number of Results > 0', the alert will not fire because 0 is not greater than 0. If the trigger condition is 'Always', the alert will fire even with zero results. The search still runs and saves the results either way.
Configure throttling in the alert settings. For example, set throttling to '1 hour' so that even if the alert triggers every minute, Splunk only sends one email per hour. This prevents notification fatigue.
A webhook is a URL that Splunk sends an HTTP POST request to when an alert triggers. This is commonly used to send notifications to chat platforms like Slack, Microsoft Teams, or custom applications without needing an email server.
Yes. Use the Cron Schedule option with an expression like '0 0 1 * *' which means 'at midnight on the first day of every month'. The Schedule Interval dropdown cannot achieve this level of specificity.
Backfill runs the scheduled search for any previous time windows that were missed (e.g., during a server outage). For alerts, you should not enable backfill because you do not want to be notified about problems from hours ago. Only use backfill for report generation.
You've finished Scheduled Searches and Alerts. Continue through the SPLK-1003 study guide to build a complete picture of the exam.
Done with this chapter?