S3 Pre-Signed URLs and Multipart Upload

A pre-signed URL is a URL that grants temporary access to a specific S3 object for a specific operation (GET, PUT, DELETE, etc.). It is generated using the requester's AWS credentials (access key and secret key) to sign the URL. Anyone with the URL can perform the operation within the expiration window, without needing AWS credentials. This solves the problem of granting granular, time-limited access without sharing long-term keys.

Pre-signed URLs are commonly used for:

A pre-signed URL is constructed by appending query parameters to the standard S3 REST API endpoint. The signature is computed using HMAC-SHA256 with the secret access key. The URL includes:

When a client uses the URL, S3 recalculates the signature using the same algorithm and checks it against the provided signature. If they match and the expiration time has not passed, the request is allowed.

Important: The signature is tied to the HTTP method. A GET pre-signed URL cannot be used for PUT. The URL must be generated with the exact method you intend to permit.

To generate a pre-signed URL using the AWS CLI:

aws s3 presign s3://my-bucket/my-object --expires-in 3600

This outputs a URL valid for 1 hour. To generate for PUT (upload):

aws s3 presign s3://my-bucket/my-object --expires-in 3600 --method PUT

Using Python SDK (boto3):

import boto3
from botocore.config import Config

s3 = boto3.client('s3', config=Config(signature_version='s3v4'))
url = s3.generate_presigned_url(
    ClientMethod='get_object',
    Params={'Bucket': 'my-bucket', 'Key': 'my-object'},
    ExpiresIn=3600
)

Verifying a pre-signed URL: You can test it with curl:

curl -v "<pre-signed-url>"

If the URL is expired or malformed, S3 returns a 403 Forbidden with an error like "SignatureDoesNotMatch" or "AccessDenied".

Multipart upload allows you to upload a single object as a set of parts. It is designed for large objects (recommended for objects > 100 MB, required for objects > 5 GB). Benefits:

The process has three phases: 1. Initiation: You call CreateMultipartUpload which returns an upload ID. 2. Part upload: You upload each part using UploadPart, specifying the upload ID and part number (1 to 10,000). Each part must be at least 5 MB (except the last part). You receive an ETag for each part. 3. Completion: You call CompleteMultipartUpload with the list of part numbers and ETags. S3 assembles the parts in order.

Alternatively, you can abort the upload with AbortMultipartUpload, which deletes all uploaded parts.

Using AWS CLI to perform multipart upload:

# Initiate
aws s3api create-multipart-upload --bucket my-bucket --key largefile.iso
# Upload part
aws s3api upload-part --bucket my-bucket --key largefile.iso --part-number 1 --body part1 --upload-id <upload-id>
# Complete
aws s3api complete-multipart-upload --bucket my-bucket --key largefile.iso --upload-id <upload-id> --multipart-upload file://parts.json

Using boto3:

import boto3

s3 = boto3.client('s3')
response = s3.create_multipart_upload(Bucket='my-bucket', Key='largefile.iso')
upload_id = response['UploadId']

# Upload part
with open('part1', 'rb') as f:
    response = s3.upload_part(Bucket='my-bucket', Key='largefile.iso', PartNumber=1, UploadId=upload_id, Body=f)
    etag1 = response['ETag']

# Complete
s3.complete_multipart_upload(
    Bucket='my-bucket',
    Key='largefile.iso',
    UploadId=upload_id,
    MultipartUpload={'Parts': [{'ETag': etag1, 'PartNumber': 1}]}
)

Verification: Use aws s3api list-parts to see uploaded parts, or aws s3api list-multipart-uploads to see in-progress uploads.

You can generate pre-signed URLs for each part of a multipart upload. This allows a client to upload large files directly to S3 without credentials. Steps: 1. Server initiates a multipart upload (using AWS credentials) and gets an upload ID. 2. Server generates pre-signed URLs for each part using generate_presigned_url with method upload_part. 3. Client uploads each part directly to S3 using the pre-signed URLs. 4. After all parts are uploaded, server completes the multipart upload.

This pattern is common in serverless file upload applications where the client cannot hold AWS credentials.

You can use generate_presigned_post (boto3) to create a pre-signed POST form that enforces additional conditions like content-length range, success redirect, and metadata. This is used for browser-based uploads.

Example:

s3 = boto3.client('s3')
conditions = [
    ["content-length-range", 0, 10485760],  # 10 MB max
    {"acl": "public-read"}
]
fields = {
    "acl": "public-read"
}
response = s3.generate_presigned_post(
    Bucket='my-bucket',
    Key='${filename}',
    Conditions=conditions,
    Fields=fields,
    ExpiresIn=3600
)

This returns a URL and form fields that the client can use to upload via HTML form.

A healthcare SaaS platform allows doctors to upload patient MRI scans (often 200-500 MB) to S3 for analysis. The platform cannot expose AWS credentials to the browser. Solution: The backend (Node.js on Elastic Beanstalk) generates a pre-signed URL for each upload using a Lambda function. For large files, the backend initiates a multipart upload and generates pre-signed URLs for each part (e.g., 10 parts of 50 MB). The frontend JavaScript uses these URLs to upload parts in parallel using the browser's Fetch API. The backend completes the upload after receiving success callbacks. Common issue: If the pre-signed URL expires before all parts are uploaded (e.g., slow connection), the upload fails. Mitigation: Set expiration based on estimated upload time (e.g., 24 hours) and monitor progress. Also, implement a retry mechanism for failed parts.

A media company ingests user-submitted videos (up to 2 GB) for transcoding. Users upload via a web form. The backend creates a pre-signed POST form that enforces a maximum file size of 2 GB and requires a specific content type. After upload, an S3 event triggers Lambda to start a transcoding job (Elastic Transcoder or MediaConvert). For very large videos, the backend uses multipart upload with pre-signed URLs. The client is a mobile app that uploads parts sequentially to avoid memory issues. The backend tracks progress in DynamoDB. Performance consideration: Use S3 Transfer Acceleration for uploads from distant regions. Misconfiguration: Not setting a lifecycle policy to abort incomplete multipart uploads can lead to orphaned parts accumulating storage costs. Best practice: Set lifecycle rule to abort multipart uploads after 7 days.

A financial analytics firm needs to share daily reports (CSV files) with partner companies. They generate pre-signed URLs for each report and email them to partners. The URLs expire after 24 hours. To avoid regenerating URLs for each partner, they use a single pre-signed URL that is unique per object. However, they must ensure that the bucket policy does not override the pre-signed URL permissions. They also implement CloudTrail logging to audit access. Common mistake: Sharing pre-signed URLs via unencrypted email, which exposes the URL. Mitigation: Use a secure portal where partners log in to retrieve URLs. Also, note that pre-signed URLs cannot be restricted to specific IPs unless using CloudFront signed URLs with IP restrictions.