Cloud Cost Management [2026] Serverless GPU Spot Cheat Sheet

Decision Matrix

“Serverless GPU spot” usually means a managed scheduler or job service placing work onto interruptible GPU-backed nodes for you. That makes AWS Batch and Google Cloud Batch the cleanest defaults. Azure still has strong spot economics, but the operating model is more VM-centric for many GPU workloads.

Choose managed batch when

• Your workload already enters the system through a queue or job spec.
• You can split work into shards, scenes, prompts, files, or epochs.
• You want retries, logs, and scheduling without maintaining a full Kubernetes control plane.

Avoid all-spot designs when

• You have strict end-user latency targets.
• You cannot checkpoint model state or partial outputs.
• Capacity spikes are deadline-bound and missed work is more expensive than on-demand compute.

Commands by Purpose

Keep these grouped by operator intent: discover, provision, submit, inspect, and clean up. Before sharing logs, billing exports, or account-scoped snippets in tickets, scrub them with the Data Masking Tool.

Discover capacity

aws ec2 describe-spot-price-history \
  --instance-types g5.xlarge g6.xlarge \
  --product-descriptions Linux/UNIX \
  --start-time 2026-05-03T00:00:00Z

gcloud compute accelerator-types list \
  --zones=us-central1-a

Provision the scheduling layer

aws batch create-compute-environment \
  --compute-environment-name gpu-spot-ce \
  --type MANAGED \
  --state ENABLED \
  --service-role AWSBatchServiceRole \
  --compute-resources type=SPOT,allocationStrategy=SPOT_PRICE_CAPACITY_OPTIMIZED,minvCpus=0,maxvCpus=256,instanceTypes=g5,g6,subnets=subnet-AAA,securityGroupIds=sg-BBB,instanceRole=ecsInstanceRole,spotIamFleetRole=AmazonEC2SpotFleetTaggingRole

az group create -n gpu-spot-rg -l eastus
az vm create \
  --resource-group gpu-spot-rg \
  --name gpu-spot-vm \
  --image Ubuntu2204 \
  --size GPU_VM_SIZE \
  --admin-username azureuser \
  --generate-ssh-keys \
  --priority Spot \
  --max-price -1 \
  --eviction-policy Deallocate

Submit GPU work

aws batch submit-job \
  --job-name gpu-inference-spot \
  --job-queue gpu-queue \
  --job-definition gpu-jobdef \
  --container-overrides resourceRequirements=[{type=GPU,value=1},{type=VCPU,value=8},{type=MEMORY,value=32768}]

gcloud batch jobs submit gpu-spot-job \
  --location=us-central1 \
  --config=job.json

Inspect and re-queue

aws batch describe-jobs --jobs JOB_ID

gcloud batch jobs describe gpu-spot-job --location=us-central1

Clean up idle cost

aws batch update-compute-environment \
  --compute-environment gpu-spot-ce \
  --state DISABLED

gcloud batch jobs delete gpu-spot-job --location=us-central1

Configuration Patterns

The safest pattern is a small, explicit config surface: multiple GPU families, interruption-tolerant queues, and no hidden defaults around capacity type.

AWS Batch Spot GPU compute environment

{
  "computeEnvironmentName": "gpu-spot-ce",
  "type": "MANAGED",
  "state": "ENABLED",
  "serviceRole": "AWSBatchServiceRole",
  "computeResources": {
    "type": "SPOT",
    "allocationStrategy": "SPOT_PRICE_CAPACITY_OPTIMIZED",
    "minvCpus": 0,
    "maxvCpus": 256,
    "instanceTypes": ["g5", "g6"],
    "subnets": ["subnet-AAA"],
    "securityGroupIds": ["sg-BBB"],
    "instanceRole": "ecsInstanceRole",
    "spotIamFleetRole": "AmazonEC2SpotFleetTaggingRole"
  }
}

• Prefer families over a single SKU so the scheduler has room to move.
• Use g5 and g6 only if both fit your driver and CUDA constraints.
• Keep minvCpus at zero for purely opportunistic queues.

Google Cloud Batch Spot GPU job

{
  "taskGroups": [
    {
      "taskSpec": {
        "runnables": [
          {
            "container": {
              "imageUri": "us-docker.pkg.dev/PROJECT_ID/images/gpu-worker:latest"
            }
          }
        ]
      },
      "taskCount": 8,
      "parallelism": 4
    }
  ],
  "allocationPolicy": {
    "instances": [
      {
        "installGpuDrivers": true,
        "policy": {
          "provisioningModel": "SPOT",
          "reservation": "NO_RESERVATION",
          "accelerators": [
            {
              "type": "nvidia-tesla-t4",
              "count": 1
            }
          ]
        }
      }
    ],
    "location": {
      "allowedLocations": ["regions/us-central1"]
    }
  },
  "logsPolicy": {
    "destination": "CLOUD_LOGGING"
  }
}

• Set installGpuDrivers when you want Batch to install the required drivers for the specified GPU type.
• Use allowedLocations to restrict scheduling to regions that actually expose the accelerator you want.
• Block reservations explicitly with NO_RESERVATION when you want clean spot behavior.

Operator defaults that pay off

• Checkpoint model state, scene progress, or shard offsets to object storage.
• Make outputs idempotent so replayed tasks overwrite or skip cleanly.
• Keep retry logic in the job layer, not only in application code.

Live Search + Shortcuts

Cheat sheets get used under pressure. Add a tiny client-side filter so readers can jump to commands by provider, task, or keyword.

const input = document.getElementById('gpu-spot-filter');
const rows = [...document.querySelectorAll('[data-filter-row]')];

function runFilter(value) {
  const q = value.trim().toLowerCase();
  rows.forEach((row) => {
    row.style.display = row.dataset.filterRow.includes(q) ? '' : 'none';
  });
}

input?.addEventListener('input', (e) => runFilter(e.target.value));

document.addEventListener('keydown', (e) => {
  if (e.key === '/') {
    e.preventDefault();
    input?.focus();
  }
  if (e.key.toLowerCase() === 'g') {
    document.getElementById('commands-by-purpose')?.scrollIntoView({behavior: 'smooth'});
  }
  if (e.key.toLowerCase() === 'c') {
    document.getElementById('config-patterns')?.scrollIntoView({behavior: 'smooth'});
  }
  if (e.key === 'Escape') {
    input.value = '';
    runFilter('');
    input.blur();
  }
});

Keyboard shortcuts

Advanced Usage

Once the basic queue works, the remaining savings come from reducing wasted retries and improving scheduling flexibility.

Capacity strategy

• Split work into a spot lane and a deadline lane instead of forcing every task onto interruptible capacity.
• Diversify across GPU families and regions when model requirements allow it.
• Use one queue for cheap opportunistic work and another for recovery or SLA-sensitive replay.

Retry and checkpoint design

• Checkpoint by completed batch, shard, epoch, or frame range.
• Write deterministic output paths so retries do not duplicate artifacts.
• Persist retry metadata outside the worker node so eviction does not erase progress history.

Cost telemetry that actually matters

• Track successful work per dollar, not only raw hourly price.
• Measure interruption rate by GPU family and zone.
• Separate queue wait time from execution time so cheap capacity does not hide slow delivery.
• Correlate requeues with specific capacity pools before expanding them.

Practical guardrails

• Cap per-job runtime if your platform struggles with very large checkpoint intervals.
• Do not treat historical spot price alone as a placement policy.
• Budget for a controlled on-demand fallback instead of emergency ad hoc recovery.