Custom Sentinel Data Connectors

Microsoft Sentinel provides over 100 built-in data connectors for popular security solutions like Azure Active Directory, Microsoft 365 Defender, and AWS CloudTrail. However, many organizations use proprietary or less common security tools—such as custom-built applications, legacy IDS/IPS systems, or IoT device logs—that lack a native connector. Custom data connectors fill this gap by allowing you to ingest any log source into Sentinel.

Custom connectors are not a single product but a set of methods to bring data into a Log Analytics workspace. The primary methods include: - Azure Logic Apps: For REST API sources. - Logstash: For any source Logstash supports. - Azure Functions: For serverless ingestion. - Syslog/CEF: For Linux-based sources. - Direct API ingestion: Using the Log Analytics Ingestion API.

Each method has specific use cases, latency characteristics, and configuration steps. The SC-200 exam focuses on when to use each and how to set them up.

All custom data connectors ultimately deliver log data to a Log Analytics workspace via the Log Analytics Ingestion API (also known as the Data Collector API). The API accepts data in JSON or CSV format and requires a workspace ID and a shared key for authentication. The data is then stored in custom tables (tables with names ending in _CL for custom logs) or in standard tables like Syslog or CommonSecurityLog.

The workflow is: 1. Data Source: The external system generates logs (e.g., application logs, firewall logs). 2. Connector Agent: The custom connector (e.g., a Logic App, a Logstash plugin) collects the logs. It may pull from an API or receive pushed data. 3. Transformation: The connector may transform the data into a schema that Sentinel expects. For example, converting CSV to JSON, or mapping fields to the CommonSecurityLog schema. 4. Authentication: The connector authenticates to the Log Analytics workspace using the workspace ID and shared key (or Azure AD authentication for newer APIs). 5. Ingestion: The connector sends the data via HTTPS POST to the Ingestion API endpoint: https://<workspace-id>.ods.opinsights.azure.com/api/logs?api-version=2016-04-01. 6. Storage: The data is stored in a custom table or a standard table. Custom tables are created automatically on first ingestion. 7. Detection: Sentinel analytic rules can query the custom table just like any other table.

For Logstash: 1. Install the plugin: bin/logstash-plugin install microsoft-sentinel-logstash-output-plugin 2. Configure the output in your Logstash config file:

output {
  microsoft-sentinel-logstash-output-plugin {
    workspace_id => "your-workspace-id"
    workspace_key => "your-shared-key"
    table_name => "MyCustomLog_CL"
    time_generated_field => "TimeGenerated"
  }
}

For Logic Apps: 1. Create a Logic App with a trigger (e.g., HTTP request, recurrence). 2. Add an action: Azure Log Analytics Data Collector. 3. Provide the workspace ID and shared key (use Azure Key Vault for security). 4. Set the JSON payload in the request body. The schema must match the custom table. 5. Test by running the Logic App and querying the custom table in Log Analytics: MyCustomLog_CL | take 10.

For Azure Functions: 1. Create a function with an HTTP trigger or timer trigger. 2. Use the Microsoft.Azure.OperationalInsights NuGet package (C#) or equivalent Python/Node.js SDK. 3. Send logs using the LogAnalyticsClient class. 4. Monitor function logs for errors.

Custom data connectors often work with: - Azure Monitor Agent (AMA): For Syslog and CEF, you can use AMA to collect logs from Linux machines and send them to Sentinel. This is not a custom connector per se, but it's an alternative for standard formats. - Logstash: Commonly used with Elastic stack; the Sentinel output plugin allows seamless integration. - Azure Event Hubs: For high-volume data ingestion, you can send logs to Event Hubs and then use a Logic App or Azure Function to process and forward to Sentinel. - Azure Data Lake Storage: For historical data, you can use Azure Data Factory to copy logs to a staging location and then ingest via Logic App.

Important Exam Note: The SC-200 exam does not require you to write code for custom connectors, but you must understand the architecture and configuration steps. You should know that custom connectors are created in the Sentinel portal under Data connectors > Create data connector (which opens a Logic App template) or by using the Log Analytics Ingestion API directly.

A financial services company has a proprietary trading application that logs transaction data to a REST API. The logs include fields like trade_id, symbol, quantity, price, and timestamp. There's no built-in Sentinel connector for this application. The security team needs to ingest these logs to detect suspicious trading patterns. They choose Azure Logic Apps because the API is well-documented and the volume is moderate (about 1000 transactions per minute).

Configuration: They create a Logic App with a recurrence trigger set to run every minute. The Logic App calls the REST API, parses the JSON response, and then uses the Azure Log Analytics Data Collector action to send the data to a custom table named TradingLogs_CL. They store the API key in Key Vault and the workspace shared key in Key Vault as well.

Performance Considerations: The Logic App must handle API pagination (e.g., the API returns a next page token). They implement a loop to fetch all pages within the 120-second action timeout. The Log Analytics API limit of 500 MB per minute is not an issue at this volume.

Common Pitfalls: If the API changes its JSON schema, the Logic App may fail to parse. They set up error handling to send alerts to the security operations center. Also, they ensure the TimeGenerated field is in UTC to avoid timezone confusion.

A large enterprise has hundreds of legacy network firewalls that send syslog data in a proprietary format (not CEF). The volume is high—about 50,000 events per second. They cannot use the built-in Syslog connector because it expects RFC 3164 or RFC 5424 format. They decide to use Logstash running on a Linux server to collect the syslog, parse it, and forward it to Sentinel.

Configuration: They install Logstash with the microsoft-sentinel-logstash-output-plugin. The input is a syslog listener on port 514. They write a custom filter to parse the proprietary format into key-value pairs. The output sends to the custom table FirewallLogs_CL. They tune the batch size to 5000 events and flush interval to 2 seconds to balance latency and API limits.

Scale Considerations: The Logstash server must have sufficient CPU and memory to handle 50,000 EPS. They deploy multiple Logstash instances behind a load balancer. They monitor the Logstash queue depth to ensure it doesn't grow unbounded. The Log Analytics API rate limit of 500 MB per minute may be a bottleneck; they compress the data (gzip) to reduce size.

What Goes Wrong: If the shared key is rotated without updating Logstash, ingestion stops. They automate key rotation using Azure Key Vault and a script that restarts Logstash with the new key. Another issue: if the syslog format changes, the filter fails, and events are lost. They implement a dead-letter queue to capture unparsed events.