S3 Batch Operations

S3 Batch Operations is a managed service that enables you to perform bulk, asynchronous operations on billions of S3 objects with a single API request, CLI command, or console action. It eliminates the need to write custom scripts that iterate over objects, handle retries, and manage concurrency. The service automatically scales to process objects in parallel across multiple AWS-managed resources, providing progress tracking, completion notifications, and detailed failure reports.

Before S3 Batch Operations, performing bulk actions (like tagging, copying, or restoring objects) required custom code using the S3 API or SDK, often involving pagination, rate limiting, error handling, and multi-threading. This was time-consuming and error-prone. S3 Batch Operations abstracts away these complexities, offering a serverless, scalable, and auditable solution. It is especially useful for large-scale data management, compliance, and cost optimization tasks.

When you create a batch job, you provide:

The service reads the manifest, divides the objects into chunks, and processes them in parallel using AWS-managed compute resources. For each object, it executes the specified operation. If the operation fails, it retries up to the configured number of times (default 1, maximum 5). After all objects are processed (or failed), the service generates a completion report in CSV format, stored in a specified S3 bucket. You can also receive an SNS notification upon job completion.

- Manifest: Required. Can be a CSV object listing bucket, key, and version ID, or an S3 Inventory report (CSV or ORC format). The manifest must be in the same AWS region as the target bucket. - Operations: Supported operations include: - Put object tagging – add/replace tags - Copy – copy objects within or across buckets (same region or cross-region) - Restore – restore objects from S3 Glacier or S3 Glacier Deep Archive - Invoke AWS Lambda – run a Lambda function on each object - Put object ACL – update ACLs (legacy) - Put object legal hold – set legal hold - Put object retention – set retention period - Initiate multipart upload – start multipart upload (rarely used alone) - Lambda invocation: You can specify a Lambda function ARN. The function receives an event with object details and must return a result (success/failure). The Lambda function must be in the same region as the job. - Retry mode: You can set the number of retries (0-5). Default is 1. The retry count applies to each individual task (object operation). - Priority: Jobs with higher priority (1-1000) are processed first. Default is 0 (lowest). - Completion report: Optional. If enabled, a CSV report is written to a specified bucket after job completion (including partial failures). The report contains status for each object (succeeded/failed) and error codes. - SNS notification: Optional. You can configure an SNS topic to receive a notification when the job completes (success or failure). - Job statuses: Active, Cancelling, Cancelled, Complete, Completing, Failed, Failing, New, Paused, Pausing, Preparing, Ready, Suspended (due to permissions issues). - Permissions: The service requires an IAM role with permissions to read the manifest, perform the operation on objects, write the completion report, and invoke Lambda (if used). The role must be passed in the job creation. - Concurrent job limit: Up to 100,000 active jobs per account per region. Each job can process up to billions of objects. - Pricing: You pay per object processed (successful or failed), plus any S3 API costs (e.g., tagging, copying) and Lambda invocation costs.

To create a batch job using the AWS CLI:

aws s3control create-job \
    --account-id 123456789012 \
    --operation '{"S3PutObjectTagging": {"TagSet": [{"Key": "department", "Value": "finance"}]}}' \
    --manifest '{"Spec": {"Format": "S3BatchOperations_CSV_20180820", "Fields": ["Bucket", "Key"]}, "Location": {"ObjectArn": "arn:aws:s3:::my-manifest-bucket/manifest.csv", "ETag": "abc123"}}' \
    --report '{"Bucket": "arn:aws:s3:::my-report-bucket", "Format": "Report_CSV_20180820", "Enabled": true, "Prefix": "reports", "ReportScope": "AllTasks"}' \
    --priority 10 \
    --role-arn arn:aws:iam::123456789012:role/S3BatchOperationsRole \
    --region us-east-1

To monitor job status:

aws s3control describe-job --account-id 123456789012 --job-id <job-id> --region us-east-1

To list jobs:

aws s3control list-jobs --account-id 123456789012 --job-statuses Active Complete --region us-east-1

To update job priority:

aws s3control update-job-priority --account-id 123456789012 --job-id <job-id> --priority 50 --region us-east-1

To cancel a job:

aws s3control cancel-job --account-id 123456789012 --job-id <job-id> --region us-east-1

1. Adding Tags to Millions of Objects for Cost Allocation

A large enterprise uses S3 to store petabytes of data across multiple accounts and regions. To implement cost allocation, they need to add a 'CostCenter' tag to every object. Instead of writing a Python script that uses list-objects and put-object-tagging with pagination and rate limiting, they create an S3 Inventory report for each bucket, then submit a batch job with the 'Put object tagging' operation. The job processes 50 million objects in a few hours, automatically retrying failures. The completion report shows 99.9% success; the remaining 0.1% are objects with permissions issues, which they fix and rerun a smaller job. This approach saved weeks of development and avoided throttling.

2. Bulk Restore from Glacier Deep Archive for Compliance Audit

A financial services company must retrieve archived data from S3 Glacier Deep Archive for a regulatory audit. They have 10 million objects archived. Using batch operations with the 'Restore' operation, they submit a job specifying a restore tier (Expedited, Standard, Bulk) and a retention period. The job automatically initiates restore requests for each object. However, they must ensure that the restore completes before accessing the data. The batch job does not wait for restoration to complete; it only initiates the request. To verify, they can use S3 Inventory to check the restore status. Common mistake: assuming the batch job waits for restore completion. It does not.

3. Copying Objects Between Buckets for Data Migration

A media company needs to copy 200 TB of video files from an old bucket to a new bucket with different encryption settings. They use the 'Copy' operation with the destination bucket and optional parameters like storage class, encryption, and metadata. The batch job handles copying, including large objects (up to 5 GB per copy; for larger objects, use S3 Batch Operations with Lambda to initiate multipart copy). However, cross-region copy is supported but incurs data transfer costs. They set a priority of 100 to expedite the job. After completion, they verify the copy using the completion report and then delete the source objects with a separate batch job.