S3 Event Notifications and S3 Object Lambda

S3 Event Notifications allow you to trigger actions automatically when specific events occur in an S3 bucket. Supported event types include: - s3:ObjectCreated:* (all create events) - s3:ObjectCreated:Put - s3:ObjectCreated:Post - s3:ObjectCreated:Copy - s3:ObjectCreated:CompleteMultipartUpload - s3:ObjectRemoved:* (all delete events) - s3:ObjectRemoved:Delete - s3:ObjectRemoved:DeleteMarkerCreated - s3:ObjectRestore:* (restore events) - s3:ObjectRestore:Post - s3:ObjectRestore:Completed - s3:ReducedRedundancyLostObject (for RRS) - s3:Replication:* (replication events) - s3:LifecycleExpiration:* - s3:LifecycleExpiration:Delete - s3:LifecycleExpiration:DeleteMarkerCreated - s3:LifecycleTransition - s3:IntelligentTiering - s3:ObjectTagging:* - s3:ObjectTagging:Put - s3:ObjectTagging:Delete - s3:ObjectAcl:Put

Notifications can be delivered to: - Amazon Simple Notification Service (SNS) – fan out to multiple subscribers. - Amazon Simple Queue Service (SQS) – decouple processing with a queue. - AWS Lambda – invoke a function directly. - Amazon EventBridge – for advanced routing and filtering (via S3 on EventBridge).

Key configuration details: - Notifications are configured at the bucket level, with optional prefix and suffix filters. - You can have up to 100 notification configurations per bucket (each configuration can have multiple event types and destinations). - The destination resource (SQS queue, SNS topic, Lambda function) must have a resource-based policy that allows S3 to publish to it. - Notifications are typically delivered within seconds, but there is no SLA on delivery time. - If the destination is unavailable, S3 will retry for up to 24 hours, then discard the notification.

How it works internally: 1. An object event occurs (e.g., PUT). 2. S3 evaluates all notification configurations for the bucket. 3. For matching configurations (based on event type, prefix, suffix), S3 publishes a JSON message to the destination. 4. The message includes details like bucket name, object key, event name, size, etag, and version ID.

Example notification message (to SQS/SNS):

{
  "Records": [
    {
      "eventVersion": "2.1",
      "eventSource": "aws:s3",
      "awsRegion": "us-east-1",
      "eventName": "ObjectCreated:Put",
      "s3": {
        "s3SchemaVersion": "1.0",
        "bucket": {
          "name": "my-bucket",
          "arn": "arn:aws:s3:::my-bucket"
        },
        "object": {
          "key": "images/photo.jpg",
          "size": 1024,
          "eTag": "d41d8cd98f00b204e9800998ecf8427e",
          "versionId": "abc123",
          "sequencer": "0055AED6DCD90281E5"
        }
      }
    }
  ]
}

For Lambda invocation, the event structure is similar but wrapped in a Lambda event.

Common use cases: - Automatically trigger a Lambda function to resize uploaded images. - Send a notification to an SNS topic when a new file is added to process orders. - Enqueue messages to SQS for asynchronous processing.

AWS now recommends using Amazon EventBridge for advanced event-driven patterns. S3 can send events directly to EventBridge (enable S3 on EventBridge at the bucket level). EventBridge provides:

Exam tip: If the question mentions filtering by object metadata or size, or using multiple destinations, think EventBridge. If it's a simple notification to Lambda/SQS/SNS, S3 Event Notifications suffice.

S3 Object Lambda allows you to add custom code to modify the data returned by standard S3 GET, HEAD, and LIST requests. You define a Lambda function that transforms the object on the fly, and clients access the transformed data via a special S3 Object Lambda Access Point. The original object remains unchanged in S3.

Key components: - S3 Object Lambda Access Point: A special type of access point that is associated with a Lambda function and a supporting access point. - Supporting Access Point: A standard S3 access point that points to the bucket. - Lambda function: The transformation code that receives the original object (or its S3 reference) and returns the transformed version. - S3 Object Lambda alias: A DNS name (e.g., my-access-point-123456789012.s3-object-lambda.us-east-1.amazonaws.com) used by clients.

How it works: 1. Client sends a GET request to the Object Lambda Access Point URL. 2. S3 Object Lambda invokes the associated Lambda function, passing an event with details about the request and a pre-signed URL to fetch the original object. 3. The Lambda function can:

- Fetch the original object using the pre-signed URL. - Transform it (e.g., resize, redact, compress, convert format). - Return the transformed object to S3 Object Lambda. 4. S3 Object Lambda returns the transformed object to the client.

Example Lambda event (simplified):

{
  "xAmzRequestId": "request-id",
  "getObjectContext": {
    "inputS3Url": "https://s3.amazonaws.com/...",
    "outputRoute": "...",
    "outputToken": "..."
  },
  "configuration": {
    "accessPointArn": "arn:aws:s3-object-lambda:us-east-1:123456789012:accesspoint/my-ap",
    "supportingAccessPointArn": "arn:aws:s3:us-east-1:123456789012:accesspoint/my-supporting-ap",
    "payload": "{}"
  },
  "userRequest": {
    "url": "https://my-ap-123456789012.s3-object-lambda.us-east-1.amazonaws.com/key",
    "headers": { "Host": "...", "Accept": "*/*" }
  },
  "userIdentity": { "type": "AssumedRole", "arn": "..." },
  "protocolVersion": "1.00"
}

Key Lambda function requirements: - The function must call WriteGetObjectResponse API to send the transformed object back. - It can also read the original object using the provided inputS3Url (pre-signed URL). - The function must handle errors gracefully; if it fails, S3 Object Lambda returns an error to the client. - The function can be written in any Lambda-supported runtime. - The function must have IAM permissions to call s3-object-lambda:WriteGetObjectResponse.

Use cases: - Redacting personally identifiable information (PII) from documents. - Resizing images on the fly based on query parameters. - Converting file formats (e.g., .doc to .pdf). - Watermarking or adding annotations. - Compressing large files before download.

Performance considerations: - Lambda execution time adds to the request latency. For large objects, consider streaming responses (if supported by the runtime). - Lambda concurrency limits can throttle requests; use reserved concurrency if needed. - S3 Object Lambda supports payload sizes up to 5 GB (for transformed object). - The original object size can be up to 5 GB as well.

Pricing: - You pay for S3 Object Lambda requests and data transfer. - Standard S3 GET requests are not charged when using Object Lambda (the access point handles the request). - Lambda execution time and data transfer costs apply.

These two features can be combined: Event Notifications can trigger Lambda functions that update metadata or trigger downstream processing, while Object Lambda transforms data at read time. For example:

To use Object Lambda, you need: 1. Lambda function execution role with permissions to: - s3-object-lambda:WriteGetObjectResponse - s3:GetObject on the supporting access point (if fetching original object via SDK) 2. Client IAM policy to allow s3:GetObject on the Object Lambda Access Point ARN. 3. Bucket policy to allow the supporting access point to read objects (if bucket is not public).

aws s3control create-access-point --bucket my-bucket --name my-supporting-ap --account-id 123456789012

aws s3control create-access-point-for-object-lambda \
  --account-id 123456789012 \
  --name my-object-lambda-ap \
  --configuration "{SupportingAccessPoint: arn:aws:s3:us-east-1:123456789012:accesspoint/my-supporting-ap, CloudWatchMetricsEnabled: true, AllowedFeatures: [GetObject-Range, GetObject-PartNumber], TransformationConfigurations: [{Actions: [GetObject], ContentTransformation: {AwsLambda: {FunctionArn: arn:aws:lambda:us-east-1:123456789012:function:my-function, FunctionPayload: '{}'}}}]}"

Scenario 1: Image Processing Pipeline

A social media platform allows users to upload high-resolution profile pictures. When a user uploads an image, S3 Event Notification triggers a Lambda function that creates three resized versions (thumbnail, small, medium) and stores them in separate prefixes. This is configured with an event notification on the 'uploads/' prefix for 's3:ObjectCreated:Put'. The Lambda function uses the Pillow library to resize and writes to the same bucket under 'thumbnails/', 'small/', and 'medium/'. The destination is a Lambda function. The bucket policy allows the function's execution role to put objects. Common issues: if the Lambda fails due to timeouts (default 3 seconds), increase timeout to 30 seconds. Also, ensure the function has enough memory (e.g., 1024 MB) for image processing.

Scenario 2: PII Redaction for Compliance

A healthcare company stores patient records in S3. When a doctor retrieves a document, S3 Object Lambda invokes a Lambda function that redacts sensitive fields like Social Security numbers and patient names using regular expressions, then returns the redacted PDF. The supporting access point is configured to allow only VPC access for security. The Lambda function uses PyMuPDF to modify the PDF. Performance: each request adds ~500ms for redaction. To handle high concurrency, the Lambda function is configured with reserved concurrency of 100. The Object Lambda Access Point is integrated with AWS CloudTrail for audit logging. Misconfiguration: if the Lambda function fails to call WriteGetObjectResponse, the client receives a 500 error. Ensure the function has proper error handling.

Scenario 3: Dynamic Watermarking for Media

A stock photo website uses S3 Object Lambda to add watermarks to images on-the-fly based on the user's subscription level. The client passes a 'watermark=true' query parameter. The Lambda function checks the user identity from the request headers, then either adds a watermark or returns the original image. The Object Lambda Access Point is configured with AllowedFeatures: ['GetObject-Range'] to support range requests for partial downloads. The Lambda function uses the AWS SDK to call WriteGetObjectResponse with the modified image. If the function is misconfigured (e.g., missing IAM permissions), clients get an access denied error. At scale, Lambda concurrency limits can cause throttling; use provisioned concurrency for predictable performance.