CIM Data Normalization and Tagging

When you manage a company's IT systems, you collect logs — records of events — from dozens of different sources: firewalls, servers, laptops, cloud services, databases. Each of these sources was built by a different vendor, and each vendor chose their own names for the same concepts. For example, a firewall might record a user's activity in a field called 'src_user', a Windows server might call it 'UserName', and a custom application might use 'account_id'. If you search Splunk for 'user=jdoe', you'll only see results from the sources that used 'user' as a field name. You'll completely miss the data from the firewall and the Windows server. This is a disaster for security investigations and troubleshooting — you need to see the full picture.

The Common Information Model (CIM) fixes this by defining a standard set of field names that all Splunk data should use. Think of it as an agreement: "From now on, every log that describes a user incident will put the username in a field called 'user'." Splunk maintains this list of standard field names across different categories of data, called "data models." A data model is like a blueprint for a specific type of event. The most common data models you will encounter on the SPLK-1003 exam are: the Authentication data model (for login successes and failures), the Network Traffic data model (for connections between computers), the Web data model (for web server requests), and the Endpoint data model (for processes and files on a computer). Each data model has a list of required and optional fields. For instance, the Authentication data model requires a field called 'action' to store whether the login was 'success' or 'failure', and a field called 'user' to store the username.

So how does this actually work? The process of converting your raw, messy logs into CIM-compliant data is called "normalisation." It happens through two main tools in Splunk: field extractions and calculated fields. Field extractions are rules that tell Splunk how to pull out a piece of information from the raw log text. For example, a raw firewall log might look like this: "Dec 15 10:00:00 192.168.1.1 allow TCP 10.0.0.5 443" . A field extraction rule could tell Splunk: "when you see the word 'allow', put that value into a field called 'action'." Calculated fields take it a step further — they let you transform a value as you extract it. For example, if a Windows log stores the username as 'DOMAIN\jdoe', you could use a calculated field to strip the 'DOMAIN\' part and store just 'jdoe' in the 'user' field.

But normalisation alone is not enough. Your data now has standard field names, but you still need a way to group related events together. This is where tagging comes in. A tag is a label that you attach to a data source, a field value, or an event type. Tags help you categorise data so that you can find it later. For example, you might tag all logs coming from your firewalls with the tag 'firewall', and all logs from your web servers with the tag 'webserver'. Then, if you only want to search for web traffic, you can use the tag in your search: index=main tag=webserver. Tags are also essential for some of Splunk's advanced features, like the Splunk App for Enterprise Security, which relies on specific tags to automatically identify the type of data it's looking at.

The key insight is that CIM normalisation and tagging work together. The CIM normalisation gives you a consistent vocabulary for your data (standard field names), while tagging gives you a filing system to organise it. Without the CIM, tags would be useless because you'd be tagging fields that don't exist in most of your data. Without tags, the CIM would give you neat data but no way to easily separate "firewall events" from "server events." Together, they enable the consistent searching that the SPLK-1003 exam objective 3.5 demands. When you master this, you can write one search that finds all authentication failures across your entire IT environment, regardless of which systems generated them.

Imagine you are a security analyst at a mid-sized company called 'Apex Retail'. Your job is to monitor for suspicious login attempts. You have three different systems: a Windows Active Directory server that logs every user logon, a VPN appliance that logs remote connections, and a cloud-based HR app that logs employee logins to their payroll portal. Each system stores its logs in a different location and uses different field names. The Windows server stores the username in a field called 'User', the VPN stores it in 'user_name', and the HR app stores it in 'login_id'. Furthermore, the Windows server calls a failed login 'Logon Failure', the VPN calls it 'Authentication Error', and the HR app calls it 'Invalid Credentials'. Without CIM normalisation and tagging, you cannot run a single search to see all failed logins in one place. You would have to run three separate searches and manually stitch the results together — a slow, error-prone process.

As a Splunk Power User, here is what you would actually do to fix this. First, you would work with an administrator to configure the Splunk Universal Forwarder on each source system. A Universal Forwarder is a lightweight piece of software installed on the machine sending logs; it collects the logs and sends them to the Splunk indexer. Once the logs arrive in Splunk, you would inspect a sample of raw events from each source. You would look at the raw text to see the exact format. For example, a sample raw event from the HR app might read: '2024-03-15 10:30:00 LOGIN_FAILURE for user jdoe from IP 10.0.0.100'.

Then, you would create field extractions. For the HR app's logs, you would create a regular expression (a pattern-matching rule) that says: 'when you see the word 'LOGIN_FAILURE', set the field 'action' to 'failure'.' You would create another extraction that says: 'when you see the word 'user', grab the next word and call it 'user'.' You would do the same for the VPN logs and the Windows logs, ensuring the 'user' field contains the username and the 'action' field contains 'success' or 'failure' in every case.

After setting up the extractions, you would use calculated fields to normalise the data further. For example, if the Windows logs store the username as 'APEX\jdoe', you would create a calculated field that removes 'APEX\' and sets the 'user' field to just 'jdoe'. Once the data is normalised, you would create tags. You would tag all events coming from the Windows server with 'authentication' and 'windows'. You would tag all events from the VPN with 'authentication' and 'vpn'. You would tag all events from the HR app with 'authentication' and 'hr-app'.

Now, when a security incident occurs, you can run a single search: 'tag=authentication action=failure | stats count by user'. This search looks across all three systems, finds every event that has the 'authentication' tag and where the 'action' field is 'failure', and gives you a single table showing which usernames had the most failed logins. This is exactly what the SPLK-1003 exam expects you to understand: how to design and use normalisation and tagging to solve a real-world problem. Without this skill, you are flying blind in a multi-source environment.