New BlogDistributed tracing without code changes using eBPF and OpenTelemetryRead the post
Skip to main content

Case Study

Celery to Argo Workflows Migration: How an AI Red Teaming Platform Cut Compute Costs 11x


In this Celery to Argo Workflows migration, CloudRaft moved an AI red teaming platform's job engine from Celery to Argo Workflows on Kubernetes. Each red teaming job now runs in its own right-sized pod, on spot capacity, and survives restarts and deploys. Compute costs dropped 11x: the client now runs eleven jobs for the price of one. The application code did not change.

The client's customers attack their own AI models on purpose to find weak spots before anyone else does. A single test can run for hours on expensive hardware. When one fails halfway, the client pays twice: once for the failed run and again for the rerun.

This case study covers why the old setup broke at scale and how we rebuilt it. For the engineering detail, including the NATS and Argo Events configuration, read our companion post: How We Replaced Celery with Argo Workflows.

About the client

The client runs an AI security platform with automated red teaming at its core. Customers point it at their models and applications. The platform then runs attack campaigns across many categories, such as jailbreaks, prompt injection, and data leakage.

These jobs are long, uneven, and bursty. A quick check finishes in minutes, while a multi-step attack chain can run for six hours. Demand rises and falls with customer test schedules.

Business impact

OutcomeResult
Compute cost per job11x lower: eleven jobs now run for the cost of one
Jobs lost to deploysRestarts and rollouts no longer kill running jobs
Failure blast radiusOne failing job no longer takes down the others
Job operationsManual restarts replaced by dashboard stop, retry, rerun
Application changesNone: the existing app sends job requests exactly as before

Why Celery stopped working for long-running AI jobs

Every red teaming job ran through one Python service that used Celery and also held much of the platform's business logic. As customers and test lengths grew, it broke in four ways:

  • Lost jobs: every restart or rollout killed running jobs. One six-hour scan had to be rerun from scratch.
  • Wasted memory: every job got a fixed 11 GB, and the setup could not use cheaper spot capacity.
  • No isolation: all jobs shared one pod, so one failure stopped everything.
  • Risky recovery: restarting jobs by hand often caused out-of-memory errors and crashed the service again.

The goal: run more jobs, stop losing them, and spend less, without rewriting the app.

Choosing a workflow engine for long-running AI jobs

We compared three options against the client's job profile.

OptionStrengthWhy it did not fit, or did
Apache Airflow (KubernetesExecutor)Mature scheduling and pipeline UIBuilt for scheduled pipelines, not on-demand jobs
TemporalDurable, code-first long-running workflowsOne more server to run and a new model for the team to learn
Argo WorkflowsKubernetes-native, one pod per stepChosen: the client already ran Kubernetes and other Argo tools

The client already used Argo CD, so Argo Workflows added the least new operational surface.

How the new architecture works

Every job now runs in its own pod, sized to that job. If a job crashes, only that job is affected, and Argo retries it.

Event-driven red teaming architecture: a custom proxy publishes job requests to NATS JetStream; an Argo Events EventSource, EventBus, and Sensor trigger the Argo Workflow Controller, which runs each red teaming task in its own pod.

Figure 1. Job requests flow from a custom proxy through NATS JetStream and Argo Events. Argo Workflows then runs each red teaming task as its own pod.

  • Thin proxy, no rewrite: a small proxy picks up the job requests the app already sends, adds job metadata, and publishes them as events.
  • Durable queue: NATS JetStream holds each request, so nothing is lost if downstream components restart.
  • Event-driven triggers: Argo Events submits one Argo Workflow per job request.
  • Right-sized pods on spot capacity: each job gets only the memory it needs. Isolated, retryable jobs can safely run on cheaper spot nodes.
  • Dashboard over manual restarts: the team stops, retries, and reruns jobs from the Argo Workflows UI.

Results: Celery vs Argo Workflows

The application code did not change. The way jobs run did.

Before (Celery)After (Argo Workflows)
Job executionAll jobs inside one shared service podOne isolated pod per job
Memory per jobFixed 11 GB for every jobSized to each job type
Node pricingOn-demand onlySpot capacity
Compute costBaseline11x lower
Service restart or deployRunning jobs lost, scans rerun from zeroRunning jobs unaffected; failed steps retried
A single job crashesCan take down every jobOnly that job is affected
ScalingFixed, sized for peakScales up with demand and back down when quiet
Recovering failed jobsManual restarts that often caused OOMsStop, retry, and rerun from the Argo UI

Most of the savings came from two changes: sizing each job individually and running on spot capacity. Neither was possible while every job shared one long-lived pod.

What we would tell other teams running long AI jobs

  • Match the tool to job duration. A task queue built for short background work struggles when jobs run for hours. Check how long jobs take and what a failure costs before you pick the tool.
  • Size jobs one by one. Giving every job the same resources is easy but expensive. Per-job sizing drove most of the savings.
  • Isolate jobs to contain failures. Separate pods stop one bad job from becoming an outage.
  • Skip the rewrite when you can. A small proxy in front of the existing app let us change everything behind it.
  • Use tools your team already knows. Argo Workflows fit because the team already ran Argo.

Technology stack

LayerTechnology
PlatformKubernetes
IngestionCustom proxy service (request enrichment, event publishing)
MessagingNATS JetStream (work-queue stream)
Event triggeringArgo Events (EventSource, EventBus, Sensor)
Workflow executionArgo Workflows (one pod per job, retries, UI)
ComputeSpot node capacity with per-job resource sizing
ReplacedCelery workers inside a single Python service

Running long, expensive AI jobs on Kubernetes?

If your AI jobs get lost on restarts or pay for resources they never use, we can help. CloudRaft designs Kubernetes platforms and cloud-native backends that run long AI workloads reliably and at lower cost.