Migrating Batch Processing to AWS with SFTP, SQS, and Auto Scaling
A company is migrating a large-scale batch processing system from on-premises to AWS. The system processes hundreds of thousands of files daily, each file being a few MB in size. The files are received via SFTP and need to be processed within 4 hours. The company wants to use AWS services to build a fully managed, scalable, and cost-effective solution. The current on-premises system uses a single server with a scheduled task that processes files sequentially. The migration should improve throughput and reduce processing time. Which architecture should the company implement?
Quick Answer
The correct architecture uses AWS Transfer Family for SFTP, Amazon S3, SQS, and an Auto Scaling group of EC2 instances. This solution works because S3 event notifications can publish file arrival messages directly to an SQS queue, decoupling ingestion from processing, while the Auto Scaling group scales EC2 instances based on queue depth to handle hundreds of thousands of files within the 4-hour window. On the SAP-C02 exam, this scenario tests your understanding of decoupled, event-driven batch processing migration patterns, where the key trap is choosing a single EC2 instance (which doesn't improve throughput) or AWS Lambda (which has a 15-minute timeout unsuitable for multi-MB files). The exam often presents AWS Batch with Fargate as a distractor, but for long-running file processing, EC2-based auto scaling with SQS provides better cost control and scalability. Memory tip: think "SFTP to S3, SQS to scale" — the queue depth drives the scaling decision, not the file count alone.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Use AWS Transfer Family for SFTP, store files in Amazon S3, configure an S3 event notification to publish messages to an Amazon SQS queue, and use an Auto Scaling group of EC2 instances to process messages from the queue.
It uses AWS Transfer Family for managed SFTP, stores files in S3, and then leverages S3 event notifications to send messages to an SQS queue. An Auto Scaling group of EC2 instances can then process messages from the queue, scaling based on queue depth (e.g., using CloudWatch alarms). This provides scalable, parallel processing, improving throughput over the on-premises sequential system. Option A is incorrect because AWS Batch with Fargate may incur higher costs for long-running jobs and is subject to concurrency limits. Option B is incorrect because AWS Lambda has a 15-minute timeout and is not suitable for files that may take longer to process, and also may not be cost-effective for high-volume processing. Option C is incorrect because a single EC2 instance processes sequentially, similar to on-premises, and does not improve throughput.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Use AWS Transfer Family for SFTP, store files in Amazon S3, and use AWS Batch with Fargate launch type to process files in parallel.
Why it's wrong here
AWS Batch with Fargate has limits on concurrent jobs and is less cost-effective for long-running jobs.
- ✗
Use AWS Transfer Family for SFTP, store files in Amazon S3, and use AWS Lambda functions triggered by S3 events to process each file.
Why it's wrong here
Lambda has a 15-minute timeout; file processing may exceed that.
- ✗
Use AWS Transfer Family for SFTP, store files in Amazon S3, and configure a single EC2 instance to poll S3 for new files and process them sequentially.
Why it's wrong here
Single instance does not improve throughput; still sequential.
- ✓
Use AWS Transfer Family for SFTP, store files in Amazon S3, configure an S3 event notification to publish messages to an Amazon SQS queue, and use an Auto Scaling group of EC2 instances to process messages from the queue.
Why this is correct
SQS decouples and scales processing; Auto Scaling based on queue depth improves throughput.
Quick reference
AWS S3 Storage Class Comparison
| Storage Class | Min Duration | Retrieval | Use Case |
|---|---|---|---|
| S3 Standard | None | Immediate | Frequently accessed data |
| S3 Standard-IA | 30 days | Immediate | Infrequent access, rapid retrieval |
| S3 One Zone-IA | 30 days | Immediate | Non-critical infrequent data |
| S3 Intelligent-Tiering | None | Immediate–hours | Unknown or changing access patterns |
| S3 Glacier Instant | 90 days | Milliseconds | Archive with instant retrieval |
| S3 Glacier Flexible | 90 days | Minutes–hours | Archive, flexible retrieval |
| S3 Glacier Deep Archive | 180 days | Hours | Long-term compliance archive |
Go deeper
Related to this question
About these practice questions
Courseiva writes every SAP-C02 question from scratch — 1,660 in total, each with an explanation and a wrong-answer breakdown. None are copied from real exams or dumps. Learn why practice questions differ from exam dumps →
Same concept, more angles
8 more ways this is tested on SAP-C02
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. A company is migrating a large-scale batch processing system from on-premises to AWS. The system runs millions of short-lived jobs each day. The company wants to minimize operational overhead and cost. Which AWS compute service should the company use?
medium- A.Amazon EC2 with Spot Fleet
- B.Amazon ECS with AWS Fargate
- C.AWS Lambda
- ✓ D.AWS Batch
Why D: AWS Batch is specifically designed for batch computing workloads, handling millions of short-lived jobs efficiently by automatically provisioning compute resources and scaling based on job demand. It integrates with Spot Instances to reduce costs. Option A is wrong because EC2 with Spot Fleet requires manual management of instances and scaling, increasing operational overhead. Option B is wrong because ECS with Fargate is optimized for containerized applications, but AWS Batch provides more specialized features for batch job scheduling and cost optimization. Option C is wrong because Lambda has a maximum execution timeout of 15 minutes and is intended for short, event-driven functions, not suitable for batch processing.
Variation 2. A company is migrating a large-scale batch processing workload from on-premises to AWS. The workload runs on a schedule and processes data files from an FTP server. The company wants to use AWS services that are serverless and event-driven to reduce operational overhead. The data files will be uploaded to an Amazon S3 bucket. Which solution meets these requirements?
hard- A.Use Amazon S3 Event Notifications to invoke an AWS Batch job that processes the file
- ✓ B.Use Amazon S3 Event Notifications to invoke an AWS Lambda function that processes the file
- C.Use AWS Glue to crawl the S3 bucket and run an ETL job on a schedule
- D.Use Amazon S3 Event Notifications to start an AWS Step Functions workflow that runs processing on Amazon EC2
Why B: Amazon S3 Event Notifications can directly invoke an AWS Lambda function to process the file upon upload. This is serverless and event-driven, meeting the requirements to reduce operational overhead. Option A (AWS Batch) is not serverless because it requires provisioning compute resources and does not directly trigger from S3 events in a serverless manner. Option C (AWS Glue) is designed for ETL and crawling, not for general batch processing of files. Option D (Step Functions) is used for orchestrating workflows, but the actual processing would still need a compute resource like EC2 or Lambda; using it with EC2 adds overhead and is not fully serverless.
Variation 3. A company is migrating a batch processing workload to AWS. The workload runs a Java application that processes files from an S3 bucket. The company wants to minimize operational overhead. Which AWS service should the company use to run the Java application?
medium- A.Amazon EC2
- B.Amazon ECS with EC2 launch type
- C.AWS Batch
- ✓ D.AWS Lambda
Why D: (AWS Lambda) is correct because it allows running Java code in response to S3 events with no server management, minimizing operational overhead. AWS Lambda supports Java runtime, can be triggered by S3 object creation events, and automatically scales. Options A (Amazon EC2) and B (Amazon ECS with EC2 launch type) require managing EC2 instances or cluster, increasing overhead. Option C (AWS Batch) is designed for batch jobs but still requires compute environments (EC2 or Fargate) and is more complex than Lambda for simple file processing.
Variation 4. A company is migrating a large-scale batch processing workload to AWS. The workload runs on a schedule and processes terabytes of data. The migration must be cost-effective. Which AWS service is best suited for this workload?
hard- A.Amazon EMR
- B.AWS Lambda
- ✓ C.AWS Batch
- D.Amazon EC2 with Auto Scaling
Why C: AWS Batch is designed for batch computing and can automatically provision the optimal quantity and type of compute resources. Option A is wrong because Amazon EMR is optimized for big data frameworks like Spark, not general batch processing. Option B is wrong because AWS Lambda has a 15-minute timeout and limited memory, making it unsuitable for large-scale batch processing. Option D is wrong because Amazon EC2 with Auto Scaling requires more manual management of instances compared to AWS Batch.
Variation 5. A company is migrating a batch processing workload to AWS. The workload runs nightly and takes approximately 2 hours on a single on-premises server. The company wants to reduce the processing time to under 30 minutes. Which THREE steps should the solutions architect recommend?
hard- ✓ A.Split the workload into smaller batches and run them in parallel on multiple EC2 instances.
- ✓ B.Use a larger, compute-optimized EC2 instance type.
- C.Use EBS io2 volumes with high IOPS for the instance.
- D.Use EC2 Spot Instances to reduce cost.
- ✓ E.Use AWS Batch with multiple compute environments to orchestrate parallel processing.
Why A: Using a larger instance (compute optimized) can speed up processing. Parallel processing by splitting the workload across multiple EC2 instances reduces time. Using a distributed processing framework like AWS Batch with multiple compute environments enables parallel execution. EBS io2 volumes improve I/O but may not reduce time by 75%. Spot Instances can reduce cost but not processing time directly.
Variation 6. A company is migrating a batch processing workload to AWS. The workload runs on a schedule and processes large files stored on a network file system. The company wants to use a serverless architecture to reduce costs. Which combination of AWS services should the company use?
medium- A.AWS Step Functions, Amazon EMR, and Amazon EFS.
- B.Amazon CloudWatch Events, AWS Lambda, and Amazon Kinesis Data Firehose.
- ✓ C.AWS Step Functions, AWS Lambda, and Amazon S3.
- D.Amazon CloudWatch Events, Amazon EC2, and Amazon EBS.
Why C: AWS Step Functions can orchestrate the workflow, AWS Lambda can process files in a serverless manner, and Amazon S3 can store the large files. Option A is wrong because Amazon EMR is not a serverless service and EFS is a network file system, not ideal for serverless batch processing. Option B is wrong because Amazon Kinesis Data Firehose is designed for streaming data, not batch processing of large files. Option D is wrong because Amazon EC2 and EBS are not serverless.
Variation 7. A company is migrating a batch processing workload to AWS. The workload runs on a schedule and processes large files stored on a network file system. The company wants to use serverless services. Which TWO services should the company use? (Choose TWO.)
easy- A.Amazon EMR
- B.Amazon EC2
- ✓ C.Amazon S3
- D.Amazon EFS
- ✓ E.AWS Lambda
Why C: Amazon S3 (Option C) is a serverless object storage service that can store the large files. AWS Lambda (Option E) is a serverless compute service that can process the files, for example by triggering on S3 events. Options A (Amazon EMR) requires a cluster of EC2 instances and is not serverless. Option B (Amazon EC2) requires provisioning servers and is not serverless. Option D (Amazon EFS) is a managed file system that is serverless, but it does not provide compute capabilities and is not the primary serverless compute service needed for processing. Therefore, the correct answers are C and E.
Variation 8. A company is migrating a batch processing workload to AWS. The workload runs on a scheduled basis and processes large files from an on-premises NFS share. The company wants to use AWS services to replace the on-premises server and minimize operational overhead. Which solution is MOST suitable?
medium- A.Use AWS Lambda to directly read from the NFS share over VPN and process the files.
- ✓ B.Mount the on-premises NFS share to Amazon EFS using AWS Direct Connect, and use AWS Batch with AWS Fargate to process the files.
- C.Use AWS Snowball Edge to transfer files to Amazon S3 and then process with EC2 instances.
- D.Use AWS DataSync to copy files to Amazon S3 and trigger an AWS Lambda function to process.
Why B: AWS Batch with AWS Fargate is a serverless compute service that integrates with Amazon EFS. By mounting the on-premises NFS share to EFS via AWS Direct Connect, files are accessible in AWS without copying. AWS Batch can then process the files on a scheduled basis, minimizing operational overhead. Option A is wrong because AWS Lambda has execution time and file size limits, and directly reading from NFS over VPN may be unreliable. Option C is wrong because Snowball Edge is a one-time transfer tool, not suitable for ongoing scheduled processing. Option D is wrong because while DataSync can copy files to S3, triggering Lambda has similar limitations, and using EC2 adds management overhead.
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This SAP-C02 practice question is part of Courseiva's free Amazon Web Services certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the SAP-C02 exam.