A data engineer needs to run a one-time complex data transformation job on a large dataset (10 TB) stored in Cloud Storage. The job will take approximately 8 hours and is not fault-tolerant. The engineer wants the cheapest possible compute option that can reliably complete the job. What should they use?
Standard on-demand VMs provide guaranteed availability for as long as you need them, with no upfront commitment or termination risk. You can select an instance size matching your transformation's requirements, run the job, and then delete the VM immediately afterward, paying only for the precise compute time consumed. This approach offers direct control over the environment, simplifies debugging, and is cost-effective for a one-time workload because there are no hidden service fees or prolonged obligations. For a job that must complete reliably once, a standard VM that is deleted after use is the most straightforward and dependable choice.
Why this answer
Preemptible VMs can be terminated at any time, so they are not reliable for a non-fault-tolerant job. Standard VMs are reliable and can be stopped after the job to save costs. Committed use discounts require long-term commitment.
Dataflow is a fully managed service but may have a minimum cost; however, the question asks for the cheapest compute option, and standard VMs (with proper sizing) can be cheaper than Dataflow for a one-time job.