In this Celery to Argo Workflows migration, CloudRaft moved an AI red teaming platform's job engine from Celery to Argo Workflows on Kubernetes. Each red teaming job now runs in its own right-sized pod, on spot capacity, and survives restarts and deploys. Compute costs dropped 11x: the client now runs eleven jobs for the price of one. The application code did not change.
The client's customers attack their own AI models on purpose to find weak spots before anyone else does. A single test can run for hours on expensive hardware. When one fails halfway, the client pays twice: once for the failed run and again for the rerun.
This case study covers why the old setup broke at scale and how we rebuilt it. For the engineering detail, including the NATS and Argo Events configuration, read our companion post: How We Replaced Celery with Argo Workflows.
About the client
The client runs an AI security platform with automated red teaming at its core. Customers point it at their models and applications. The platform then runs attack campaigns across many categories, such as jailbreaks, prompt injection, and data leakage.
These jobs are long, uneven, and bursty. A quick check finishes in minutes, while a multi-step attack chain can run for six hours. Demand rises and falls with customer test schedules.
Business impact
| Outcome | Result |
|---|---|
| Compute cost per job | 11x lower: eleven jobs now run for the cost of one |
| Jobs lost to deploys | Restarts and rollouts no longer kill running jobs |
| Failure blast radius | One failing job no longer takes down the others |
| Job operations | Manual restarts replaced by dashboard stop, retry, rerun |
| Application changes | None: the existing app sends job requests exactly as before |
Why Celery stopped working for long-running AI jobs
Every red teaming job ran through one Python service that used Celery and also held much of the platform's business logic. As customers and test lengths grew, it broke in four ways:
- Lost jobs: every restart or rollout killed running jobs. One six-hour scan had to be rerun from scratch.
- Wasted memory: every job got a fixed 11 GB, and the setup could not use cheaper spot capacity.
- No isolation: all jobs shared one pod, so one failure stopped everything.
- Risky recovery: restarting jobs by hand often caused out-of-memory errors and crashed the service again.
The goal: run more jobs, stop losing them, and spend less, without rewriting the app.
Choosing a workflow engine for long-running AI jobs
We compared three options against the client's job profile.
| Option | Strength | Why it did not fit, or did |
|---|---|---|
| Apache Airflow (KubernetesExecutor) | Mature scheduling and pipeline UI | Built for scheduled pipelines, not on-demand jobs |
| Temporal | Durable, code-first long-running workflows | One more server to run and a new model for the team to learn |
| Argo Workflows | Kubernetes-native, one pod per step | Chosen: the client already ran Kubernetes and other Argo tools |
The client already used Argo CD, so Argo Workflows added the least new operational surface.
How the new architecture works
Every job now runs in its own pod, sized to that job. If a job crashes, only that job is affected, and Argo retries it.
Figure 1. Job requests flow from a custom proxy through NATS JetStream and Argo Events. Argo Workflows then runs each red teaming task as its own pod.
- Thin proxy, no rewrite: a small proxy picks up the job requests the app already sends, adds job metadata, and publishes them as events.
- Durable queue: NATS JetStream holds each request, so nothing is lost if downstream components restart.
- Event-driven triggers: Argo Events submits one Argo Workflow per job request.
- Right-sized pods on spot capacity: each job gets only the memory it needs. Isolated, retryable jobs can safely run on cheaper spot nodes.
- Dashboard over manual restarts: the team stops, retries, and reruns jobs from the Argo Workflows UI.
Results: Celery vs Argo Workflows
The application code did not change. The way jobs run did.
| Before (Celery) | After (Argo Workflows) | |
|---|---|---|
| Job execution | All jobs inside one shared service pod | One isolated pod per job |
| Memory per job | Fixed 11 GB for every job | Sized to each job type |
| Node pricing | On-demand only | Spot capacity |
| Compute cost | Baseline | 11x lower |
| Service restart or deploy | Running jobs lost, scans rerun from zero | Running jobs unaffected; failed steps retried |
| A single job crashes | Can take down every job | Only that job is affected |
| Scaling | Fixed, sized for peak | Scales up with demand and back down when quiet |
| Recovering failed jobs | Manual restarts that often caused OOMs | Stop, retry, and rerun from the Argo UI |
Most of the savings came from two changes: sizing each job individually and running on spot capacity. Neither was possible while every job shared one long-lived pod.
What we would tell other teams running long AI jobs
- Match the tool to job duration. A task queue built for short background work struggles when jobs run for hours. Check how long jobs take and what a failure costs before you pick the tool.
- Size jobs one by one. Giving every job the same resources is easy but expensive. Per-job sizing drove most of the savings.
- Isolate jobs to contain failures. Separate pods stop one bad job from becoming an outage.
- Skip the rewrite when you can. A small proxy in front of the existing app let us change everything behind it.
- Use tools your team already knows. Argo Workflows fit because the team already ran Argo.
Technology stack
| Layer | Technology |
|---|---|
| Platform | Kubernetes |
| Ingestion | Custom proxy service (request enrichment, event publishing) |
| Messaging | NATS JetStream (work-queue stream) |
| Event triggering | Argo Events (EventSource, EventBus, Sensor) |
| Workflow execution | Argo Workflows (one pod per job, retries, UI) |
| Compute | Spot node capacity with per-job resource sizing |
| Replaced | Celery workers inside a single Python service |
Running long, expensive AI jobs on Kubernetes?
If your AI jobs get lost on restarts or pay for resources they never use, we can help. CloudRaft designs Kubernetes platforms and cloud-native backends that run long AI workloads reliably and at lower cost.