Cloud Logging Exports and Log Sinks

Cloud Logging collects logs from Google Cloud services, applications, and on-premises systems. By default, logs are retained in Cloud Logging for a limited period (30 days for most logs, 400 days for audit logs). For long-term retention, compliance, or analysis, you need to export logs to external destinations. Log sinks are the mechanism that performs this export. A log sink is a configuration object that defines a filter and a destination. When a log entry matches the filter, Cloud Logging writes a copy of that log entry to the destination. The original log remains in Cloud Logging.

When a log entry is ingested into Cloud Logging, the logging pipeline evaluates all sinks in the project, folder, or organization. Each sink has a filter expression (using the Cloud Logging query language). If the log entry matches the filter, Cloud Logging creates a delivery task for that sink. The delivery task is processed asynchronously — there is no guarantee of real-time delivery, but typically logs arrive at the destination within seconds to minutes. For high-volume sinks, Google Cloud may batch entries before writing to the destination.

- Sink ID: A user-defined name (e.g., "my-storage-sink"). Must be unique within the resource hierarchy level (project, folder, or organization). - Filter: A string using Cloud Logging filtering syntax (e.g., severity>=ERROR). Filters can include log name, resource type, labels, and more. If empty, all logs are exported. - Destination: The target resource. Supported destinations: - Cloud Storage bucket: Must be in the same project or a different project. The bucket must exist and the sink's writer identity must have storage.objectCreator and storage.objectViewer roles on the bucket. - BigQuery dataset: Must be in the same project or a different project. The dataset must exist and the writer identity must have bigquery.dataEditor role. - Pub/Sub topic: Must be in the same project or a different project. The topic must exist and the writer identity must have pubsub.publisher role. - Another Cloud Logging project (log bucket): The sink can route logs to a log bucket in a different project. This is useful for centralised log aggregation. - Writer Identity: A service account automatically created by Cloud Logging when the sink is created. Format: p1234567890-123456@gcp-sa-logging.iam.gserviceaccount.com. This identity must have appropriate permissions on the destination. - Exclusion Filters: You can create exclusion filters on log buckets to prevent certain logs from being ingested. Exclusions are applied before sinks are evaluated, so excluded logs are never exported.

#### Creating a sink via Console 1. Navigate to Logging > Log Router. 2. Click "Create Sink". 3. Provide a name, filter, and select destination type. 4. Choose or create the destination resource. 5. The console shows the writer identity; you must grant permissions on the destination.

#### Creating a sink via gcloud

gcloud logging sinks create my-sink \
    storage.googleapis.com/my-bucket \
    --log-filter='severity>=ERROR' \
    --project=my-project

This command creates a sink that exports all ERROR and higher severity logs to a Cloud Storage bucket named my-bucket. The command outputs the writer identity, which you must then authorize:

gsutil iam ch serviceAccount:WRITER_IDENTITY:objectCreator \
    gs://my-bucket

#### Verifying a sink - List sinks: gcloud logging sinks list --project=my-project - Describe a sink: gcloud logging sinks describe my-sink --project=my-project - Check sink metrics: In Cloud Monitoring, use the logging.googleapis.com/sink/entries_delivered metric.

A financial services company must retain all Admin Activity audit logs for 7 years to meet SEC regulations. They create an organization-level sink with an empty filter (to export all logs) to a Cloud Storage bucket in a locked-down project with a retention policy of 2555 days. The bucket uses the COLDLINE storage class to minimize cost. The sink's writer identity is granted storage.objectCreator on the bucket. At scale, the company exports millions of log entries per day; the bucket automatically organizes logs into folders by date. Misconfiguration: if the bucket is deleted, logs are silently dropped — the company must set up an alert on the sink's entries_delivered metric to detect delivery failures.

A large e-commerce company with 500 projects wants to analyze ERROR-level logs across all projects in real-time. They create folder-level sinks in each folder (or a single organization-level sink) that export logs with severity>=ERROR to a central BigQuery dataset in a dedicated analytics project. Data analysts use SQL queries to identify patterns. Performance consideration: BigQuery streaming inserts are used for near-real-time ingestion, but there is a daily limit on streaming inserts (currently 2 TB per day for standard edition). The company partitions the table by day and sets a table expiration of 30 days. Common issue: if the dataset is deleted, logs are lost — the sink does not automatically recreate it.

A gaming company needs to respond to critical errors (e.g., database connection failures) within minutes. They create a project-level sink with filter severity=CRITICAL to a Pub/Sub topic. A Cloud Function subscribes to the topic and sends alerts to PagerDuty. The sink must have pubsub.publisher permission. At scale, high volumes of critical logs can overwhelm the Pub/Sub topic; the company sets up a dead-letter queue and configures the Cloud Function to batch messages. Misconfiguration: if the Pub/Sub topic is deleted, logs are lost — the sink should be deleted or updated before the topic is removed.