Without getting data into the cloud, even the most powerful analytics tools are useless. This chapter covers the three key ways to move data into AWS S3—the 'hard drive' of the cloud—using S3 Transfer, AWS DataSync, and the Snow Family. For the DEA-C01 exam, you must understand which service to use when, because choosing wrongly can cost time and money.
Jump to a section
A simple way to picture Data Ingestion into AWS: S3 Transfer, AWS DataSync, and Snow Family
A Moving Company Manager is the person who organises getting a household's belongings from their old home to their new home, using different methods depending on the situation. For a regular, ongoing move of a few boxes every week, the manager uses a courier service called S3 Transfer. They simply hand the boxes to the courier who drives them directly to the new house. For a single, massive move of an entire warehouse, the manager uses a dedicated moving truck called AWS DataSync. The truck is designed to handle huge volumes, checks each item, and takes the most efficient route. For a move where the old house has no internet or is in a remote area, the manager sends a specialised moving container called the Snow Family. The container is physically delivered, you load it up with all your data, and it gets shipped back to the new house where the contents are unloaded. Each method solves the same problem—getting data from point A to point B—but the moving company manager chooses the right tool based on the size, frequency, and connectivity of the move. The job is to ensure nothing gets lost, everything arrives safely, and the move stays on schedule and within budget, matching the data ingestion needs for AWS.
Data ingestion is the process of moving data from where it lives (like a company's on-premises server or a laptop) into a storage service in the cloud, such as Amazon S3 (Simple Storage Service). Think of S3 as an infinitely large, secure, and always-available filing cabinet in the cloud. For a DEA-C01 candidate, understanding how to fill that cabinet efficiently is a core skill. AWS offers three main services for this: S3 Transfer, AWS DataSync, and the Snow Family. Each serves a different scenario.
First, S3 Transfer Acceleration. This is a feature you enable on an S3 bucket (a bucket is just a container you create in S3 to hold your files). When you upload a file to a bucket with Transfer Acceleration enabled, your file is not sent directly to the bucket's location. Instead, it goes to a nearby AWS 'edge location'—a data centre positioned close to you geographically, like a local post office. From there, it travels over the optimised, private AWS network to your bucket. This avoids the slower, congested public internet. The tool you use for the upload (like the AWS CLI, a command-line tool) automatically talks to this edge location. Use this when you have users all over the world uploading files to a single bucket and you need the upload to be fast. It adds a small cost per gigabyte uploaded.
Second, AWS DataSync. This is a separate service designed for large, one-time or recurring transfers between on-premises storage (like a Network Attached Storage device, or NAS) and AWS. A NAS is a dedicated file-storage device on a local network. DataSync uses an agent—a small piece of software that runs on a virtual machine or a physical device on your local network. This agent connects to your storage, reads the data, compresses it (squeezes it to make it smaller), and sends it to AWS over the internet or a dedicated connection like AWS Direct Connect (a private network cable from your office to AWS). DataSync can also verify the integrity of each file by checking the checksum (a mathematical fingerprint of the file data) to ensure nothing was corrupted during transfer. It can schedule transfers to happen automatically, like every night at 2 AM. Use this for migrating data from an old file server to Amazon EFS (Elastic File System) or S3.
Third, the AWS Snow Family. This is a set of physical devices that AWS ships to you when your data is massive (terabytes or petabytes) or your network is too slow or unreliable. A petabyte is 1,000 terabytes. Imagine trying to upload 100 terabytes over a 100 Mbps internet connection—it would take over 100 days. The Snow Family solves this with 'sneakernet': physically shipping storage. The main devices are:
Snowcone: A small, rugged device (about the size of a tissue box) that holds up to 14 terabytes. It has a built-in battery and screen. You plug it into your local network, load data onto it via a local device, and then ship it back to AWS.
Snowball Edge: A larger suitcase-sized device that holds up to 80 terabytes. It has more computing power and can even run AWS compute services (like EC2, which is a virtual computer) on the device itself. This is useful if you need to process data before sending it back, for example, filtering out bad images from a dataset.
Snowmobile: A literal shipping container on a truck that can hold up to 100 petabytes. This is used for massive data centre migrations. Amazon drives the truck to your facility, you load the data onto it, and they drive it to an AWS data centre.
For the Snow Family, after you ship the device back, AWS copies the data into your S3 bucket. You use the AWS Management Console (a web interface) to order the device, and you track its shipping status.
The key distinction for the exam is: use S3 Transfer for optimising standard uploads over the internet for speed. Use DataSync for automating and securing bulk transfers between on-premises and AWS over the network. Use the Snow Family when the data volume is very high (10s of terabytes or more) or the network connection is too weak to transfer the data in a reasonable time frame.
Assess Data Volume and Network Speed
The data engineer first measures the total data size (e.g., in terabytes) and the available internet upload speed. This determines if a network transfer is feasible or if the Snow Family is required. For example, 50 TB on a 100 Mbps link would take over a month, so Snow is a better choice.
Choose the Ingestion Service
Based on the assessment, the engineer selects one of three: S3 Transfer Acceleration (for faster internet uploads), AWS DataSync (for automated, scheduled transfers from on-premises), or the Snow Family (for physical shipping). This is the critical exam-relevant decision.
Prepare the Source and Target
The engineer configures the source (e.g., on-premises NAS drive) and the target (e.g., an S3 bucket or Amazon EFS file system). For DataSync, this involves installing the agent and granting permissions. For Snow, this involves ordering the device from the AWS Console.
Execute the Transfer or Ship the Device
For network transfers, the engineer initiates the copy, possibly setting up a schedule. For Snow Family, the engineer loads data onto the device using the client software, then ships it back to AWS. This step includes monitoring progress and verifying no failures.
Verify Data Integrity and Completion
After the transfer, the engineer checks that all files arrived correctly in the AWS storage. This may involve comparing file counts, sizes, and checksums using AWS CLI commands or the DataSync dashboard to ensure no data loss occurred during transit.
Clean Up and Optimise
The engineer decommissions the Snow device (if used) from the job, deletes any temporary resources, and verifies that the source can be decommissioned if desired. For recurring DataSync tasks, the engineer sets up monitoring alerts for failures.
A medium-sized company called 'RetailBox' has been running for 15 years and has all its customer transaction data and inventory records stored on a local file server in its main office. The server is five years old and running out of space. The IT director decides to move everything to AWS S3 to save money and improve reliability. Here is what an IT professional—let's call her Priya—actually does.
First, Priya assesses the data. She checks the file server and finds 50 terabytes of data. She also checks the office internet connection; it is a standard 200 Mbps broadband connection. She calculates the transfer time: 50 TB at 200 Mbps would take over 23 days, assuming the line is fully dedicated, which it is not. The business cannot afford that downtime or risk. So she rules out a plain internet upload. She considers AWS DataSync, but the calculation shows it would still take too long on the existing connection. Priya decides a Snowball Edge device is the right choice.
Priya logs into the AWS Management Console and goes to the AWS Snow Family service page. She clicks 'Create Job' and orders a Snowball Edge device. She specifies that the data should ultimately go to an S3 bucket she has already created for 'RetailBox-Archive'. AWS confirms the order, and the device is shipped to the office address. Priya tracks the delivery.
When the Snowball Edge arrives (a heavy suitcase with a handle), Priya connects it to the office local network using an ethernet cable. She then downloads and installs the 'Snowball Edge Client' software on a laptop. She uses the client to unlock the device with a code provided by AWS. Priya then uses the client command-line tool to start the data copy from the file server to the Snowball Edge. The software shows a progress bar. The copy takes a few days as it copies 50 terabytes over the local network. Priya can pause and resume if needed.
Once the copy is complete, Priya uses the client software to 'power off' the device, disconnects the cables, and packs it back in its shipping case. She goes to the AWS Console and marks the job as 'Ready for Return'. The shipping label is pre-generated. She ships it back via a courier.
A few days later, AWS receives the device. They copy the data from the Snowball Edge into the designated S3 bucket. Priya gets a notification that the data import is complete. She logs into the S3 console and sees all the files. She runs a verification script (using AWS CLI commands) to check that the file count and sizes match the original server. It all matches.
For ongoing weekly backups of new transaction files (which are much smaller, only a few gigabytes), Priya sets up AWS DataSync. She creates a schedule to run every Saturday night. The DataSync agent runs on a small server in the office. It automatically picks up new files, compresses and encrypts them, and sends them to a separate S3 bucket. Priya monitors the DataSync dashboard to see successful runs. For ad-hoc uploads from remote sales offices, she enables Transfer Acceleration on the main S3 bucket and directs the remote staff to use that endpoint for faster uploads.
The DEA-C01 exam tests your ability to select the correct data ingestion service for a given scenario. You will not be asked to configure them step-by-step, but you must know the key features, limitations, and use cases. The exam loves to present a scenario and ask: 'Which service should the data engineer use?'
Key concepts and traps:
The biggest trap is confusing S3 Transfer Acceleration with DataSync. Transfer Acceleration is for optimising uploads over the internet using edge locations. It does NOT handle on-premises file servers, nor does it schedule recurring transfers. DataSync does handle on-premises storage and scheduling. A common question: 'A company needs to migrate a 1 TB folder from a NAS drive to Amazon EFS every day.' The correct answer is DataSync, not Transfer Acceleration.
Another trap: using the Snow Family when a simple internet transfer would suffice. If the scenario says the data is 500 GB and the network connection is 10 Gbps (very fast), the transfer would take minutes, so Snow is overkill and more expensive. Conversely, if the network is very slow (100 Mbps) and the data is 1 TB, Snow is the correct answer because the internet transfer would take over a day.
The exam also tests recognition of Snow Family device specifications. Memorise the approximate storage capacities:
- Snowcone: 8-14 TB (usable) - Snowball Edge: 80 TB (usable) - Snowmobile: 100 PB - A tricky detail: Snowcone can be used for data transfer even in locations without internet, because it has a built-in display and can act as a local endpoint. Snowball Edge can also run compute jobs (like EC2 instances) directly on the device, useful for pre-processing data before sending it. - For DataSync, remember it can transfer data to S3, Amazon EFS (file storage), and Amazon FSx (specialised file systems like Windows File Server). It can also transfer from any of these to another. It uses an agent, and the agent can be deployed on a virtual machine (VMware, Hyper-V) or on an AWS provided hardware device (the DataSync appliance). - For Transfer Acceleration, the exam tests that it uses AWS edge locations and that it only works for uploads (not downloads) to S3. It also costs extra per GB. - A word on security: all three services encrypt data in transit. The Snow Family devices use hardware encryption (the data is encrypted on the device itself using a key you manage). DataSync can use encryption in transit (TLS, a security protocol) and can integrate with AWS KMS (Key Management Service) for encrypting data at rest. - The exam will also test cost awareness: Snow Family has a per-job fee plus shipping costs. DataSync has a per-GB transferred cost. Transfer Acceleration has a per-GB upload fee on top of standard S3 charges.
To remember the choices, consider a simple decision tree:
Is the data over 10 TB or is the network very slow? If yes, use Snow Family.
Is it a recurring or one-time transfer between on-premises and AWS? If yes, use DataSync.
Is it a standard internet upload that needs to be faster? If yes, enable Transfer Acceleration on the bucket.
Exam questions often present a scenario with two plausible answers, but one has a fatal flaw, such as 'Snowball Edge costs less than DataSync' (false, typically the opposite for small data). Being able to identify the tool for the volume is key.
S3 Transfer Acceleration uses AWS edge locations to speed up uploads over the internet for globally distributed uploaders.
AWS DataSync automates and schedules bulk data transfers between on-premises storage and AWS, with built-in validation and encryption.
The Snow Family (Snowcone, Snowball Edge, Snowmobile) is for physically moving large datasets (10s of TB or more) when network transfer is too slow or not possible.
Snowball Edge devices can run AWS compute (EC2) instances locally, allowing data pre-processing before shipping.
DataSync requires an agent to be installed on your network to connect to on-premises storage sources.
Snowmobile is a shipping container on a truck used for exabyte-scale migrations of entire data centres.
Transfer Acceleration adds a per-GB cost, so it is not suitable for small or infrequent uploads where speed is not critical.
DataSync can verify file integrity by comparing checksums to ensure no data corruption during transfer.
These come up on the exam all the time. Here's how to tell them apart.
S3 Transfer Acceleration
Only speeds up uploads over the internet using edge locations.
No agent or on-premises software needed; works with any S3-compatible client.
Best for ad-hoc, user-initiated uploads from distant locations.
AWS DataSync
Automates and schedules transfers between on-premises storage and AWS.
Requires an agent installed on the local network to connect to file servers.
Best for large, recurring migrations or backups from a corporate data centre.
Snowball Edge (Snow Family)
Physically shipped device; used when network is too slow or unavailable.
Supports up to 80 TB of storage per device.
Can run compute jobs locally (EC2 instances) on the device.
AWS DataSync
Transfers data over the network (internet or dedicated connection).
No limit on data volume, but speed depends on bandwidth.
Does not provide local compute capabilities.
Snowcone (Snow Family)
Smallest device, holds up to 14 TB.
Has a built-in display and battery; ideal for remote or edge locations.
Cannot run AWS compute instances.
Snowball Edge (Snow Family)
Larger device, holds up to 80 TB.
Requires external power and network connection.
Can run EC2 instances for local data processing.
S3 Transfer Acceleration
Routes uploads through nearest edge location for faster speeds.
Costs extra per GB uploaded.
Best for global teams uploading to a single bucket.
Standard S3 Upload
Uploads directly to bucket over the internet (no edge routing).
No additional cost beyond standard S3 upload charges.
Best for local uploads or when speed is not critical.
Mistake
S3 Transfer Acceleration can be used to download data from S3 faster.
Correct
S3 Transfer Acceleration only speeds up uploads to S3, not downloads.
The name 'Transfer' sounds bidirectional, but AWS designed it specifically to optimise the upload path using edge locations; downloads already benefit from global infrastructure.
Mistake
AWS DataSync requires you to set up a VPN or Direct Connect to work.
Correct
DataSync can work over the public internet using TLS encryption. Direct Connect or VPN is optional for increased security or bandwidth.
Beginners assume that any on-premises to cloud tool needs a private network, but DataSync is designed for internet-based transfer.
Mistake
The Snow Family devices are shipped pre-loaded with your data.
Correct
They are shipped empty; you load the data onto the device at your location, then ship it back to AWS for ingestion.
The term 'snow' might imply it already has data, but it is a blank container you fill.
Mistake
Using the Snow Family is always cheaper than transferring over the internet.
Correct
For small amounts of data (e.g., under 10 TB), internet transfer is cheaper. Snow is cost-effective only for very large volumes or where internet is impractical.
The physical shipping costs and job fees make it expensive for small datasets; the exam tests cost awareness.
Mistake
DataSync can only transfer data to S3, not to other AWS storage services.
Correct
DataSync can transfer to S3, Amazon EFS, and Amazon FSx (including Windows File Server and Lustre).
The name 'DataSync' sounds generic, but beginners often think it is S3-only, missing its broader file-system support.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
No, Transfer Acceleration only speeds up uploads to S3, not downloads. To speed up downloads, consider using Amazon CloudFront, which is a content delivery network.
No, DataSync can work over the public internet using TLS encryption for security. However, using Direct Connect or VPN can provide a more consistent and faster network path.
Use Snowcone for small amounts of data (up to 14 TB) or in remote locations. Use Snowball Edge for 10s of terabytes. Use Snowmobile for petabyte-scale (100 PB) migrations only.
Yes, Snow Family devices use hardware-based encryption with keys you manage through AWS Key Management Service (KMS). Your data is encrypted on the device and remains encrypted during shipping.
Yes, DataSync can transfer data to an S3 bucket or EFS file system in any AWS region, provided you have adequate permissions and network connectivity.
Yes, AWS charges for shipping the device to you and back, though it is included in the job pricing for some configurations. Check the AWS pricing page for current details.
Transfer Acceleration can handle files up to 5 GB in a single upload. For larger files, you must use multipart upload, which breaks the file into smaller parts.
You've finished Data Ingestion into AWS: S3 Transfer, AWS DataSync, and Snow Family. Continue through the DEA-C01 study guide to build a complete picture of the exam.
Done with this chapter?