Azure Batch for HPC Workloads

Azure Batch is a platform service that schedules and orchestrates compute-intensive work across a managed pool of virtual machines. It is designed for HPC and batch computing workloads where you have a large number of similar, independent tasks that can be executed in parallel. Instead of manually provisioning VMs, installing software, and managing job queues, you define the compute environment and the work to be done, and Batch handles the rest.

Key use cases include: - Rendering – rendering frames of a movie or animation in parallel. - Financial risk modeling – running Monte Carlo simulations across thousands of scenarios. - Image processing – applying filters or transformations to a large set of images. - Gene sequencing – aligning DNA sequences in parallel. - Any embarrassingly parallel workload – tasks that don't require inter-task communication.

The Batch architecture consists of four primary components: Batch account, pool, job, and task. The service runs on top of Azure compute resources (VMs), but the user does not directly interact with those VMs.

Execution Flow: 1. User creates a Batch account. 2. User creates a pool with desired VM configuration. 3. User creates a job targeting that pool. 4. User adds tasks to the job. Each task specifies a command line (e.g., myapp.exe --input file1.txt --output result1.txt). 5. Batch service schedules tasks onto available nodes. The scheduler considers task dependencies, node availability, and resource requirements (CPU, memory). 6. Each node runs the Batch agent, which pulls tasks from the service, executes them, and reports status. 7. Upon completion, task output can be retrieved from the node or uploaded to Azure Storage. 8. After all tasks complete, the job is finished. The pool can be deallocated or scaled down.

Using Azure CLI:

# Create a Batch account
az batch account create --name mybatchaccount --resource-group myrg --location eastus

# Create a pool (using JSON configuration)
az batch pool create --account-name mybatchaccount --account-endpoint https://mybatchaccount.eastus.batch.azure.com --json-file pool-config.json

Example pool-config.json:

{
  "id": "mypool",
  "vmSize": "Standard_D2_v3",
  "virtualMachineConfiguration": {
    "imageReference": {
      "publisher": "microsoftwindowsserver",
      "offer": "windowsserver",
      "sku": "2022-datacenter",
      "version": "latest"
    },
    "nodeAgentSkuId": "batch.node.windows amd64"
  },
  "targetDedicatedNodes": 3,
  "targetLowPriorityNodes": 0
}

# Create a job
az batch job create --account-name mybatchaccount --account-endpoint https://mybatchaccount.eastus.batch.azure.com --id myjob --pool-id mypool

# Add a task
az batch task create --account-name mybatchaccount --account-endpoint https://mybatchaccount.eastus.batch.azure.com --job-id myjob --task-id task1 --command-line "echo hello"

# List tasks
az batch task list --account-name mybatchaccount --account-endpoint https://mybatchaccount.eastus.batch.azure.com --job-id myjob

# Delete a pool
az batch pool delete --account-name mybatchaccount --account-endpoint https://mybatchaccount.eastus.batch.azure.com --pool-id mypool

To verify pool status:

az batch pool show --account-name mybatchaccount --account-endpoint https://mybatchaccount.eastus.batch.azure.com --pool-id mypool --query "allocationState"

A small animation studio needs to render a 10-minute short film. Each frame is an independent task. They use Azure Batch with a pool of Standard_NV6 VMs (GPU-enabled) running Windows. They configure the pool with 20 dedicated nodes and auto-scaling based on pending tasks. The job contains 14,400 tasks (10 min * 24 fps * 60 sec). Each task runs Blender to render a single frame. Input files (scene files) are stored in Azure Blob Storage; output frames are uploaded back to Blob. The entire job completes in 2 hours. Without Batch, they would need to buy 20 powerful workstations. Cost: ~$5 per hour for the pool (using low-priority VMs for 80% savings). Misconfiguration: If they forget to set the output file upload, frames are lost when nodes are deallocated. They learned to always specify output files in the task resource files.

A bank runs Monte Carlo simulations for value-at-risk calculations. They have 10,000 scenarios, each requiring 5 minutes of CPU time on a single core. They use a Batch pool of Standard_D2_v3 VMs (2 cores each) with Linux OS. They set max tasks per node to 2 (one per core). The pool auto-scales between 10 and 100 nodes based on pending tasks. The job completes in 25 minutes. They use job preparation tasks to copy the simulation binary to each node. They also use low-priority VMs to cut costs, but they implement a retry strategy for preempted tasks. Common pitfall: They initially used a fixed pool of 100 nodes and paid for idle time when tasks were finishing. Auto-scaling solved this.

An e-commerce company processes millions of product images daily. When a new image is uploaded to Blob Storage, an Azure Function triggers a Batch task to resize, compress, and watermark the image. They use a pool of Standard_F2s_v2 VMs (compute optimized) with auto-scaling from 0 to 50 nodes. The scaling formula uses $PendingTasks metric. Tasks are short (2-3 seconds each). They set task retention period to 1 hour. They use Azure Files for shared output directory. Challenge: The pool was slow to scale up from 0 because node allocation takes 5-10 minutes. They solved it by keeping a minimum of 2 idle nodes always running (warm pool).