Cloud-Native · Data Engineering
Scale with the work that is waiting, not just with CPU
Background workers add capacity automatically when messages pile up in a queue, and scale down to save money when it is quiet.
Proof of concept: Event-driven autoscaling with KEDA
Where this helps
- Background jobs and queue processing
- Cutting the cost of idle servers
- Absorbing traffic spikes without anyone stepping in
The rest of this page is the technical deep dive — how it was built and what its limits are.
The technical problem
Queue-driven workloads often sit idle or fall behind because CPU and memory are poor signals for how much work is waiting.
How the architecture works
- 01 · Produce
A producer pushes random messages to SQS or Kafka.
- 02 · Measure
KEDA scalers watch queue size, Kafka lag or CPU usage.
- 03 · Scale
ScaledObjects adjust consumer replicas, with TriggerAuthentication for AWS access.
What was being explored
How far Kubernetes autoscaling can move from resource metrics to business signals such as “messages waiting”, and how the same consumer can be scaled by different triggers.
What was implemented
Containerised producer and consumer code with Poetry, deployments and config maps, three ScaledObject variants (SQS, Kafka, CPU) and the trigger authentication needed for SQS.
Limitations and next steps
- Demonstrates scaling behaviour, not tuning for a particular workload.
- AWS credentials are supplied as a Kubernetes secret; IAM roles for service accounts would be preferred on EKS.
Related writing
Event-Driven Autoscaling with KEDA: Scaling Kubernetes Applications in Real Time with AWS SQS, Kafka, and CPU Metrics
When CPU and memory are the wrong signals: scaling consumers on queue depth in SQS and Kafka, and on CPU load, using KEDA scaled objects.