n8n Autoscaling: Scaling Workers via Redis Queue Without Kubernetes

This system provides dynamic scaling for n8n worker containers based on the length of your Redis queue. It eliminates the need for Kubernetes or heavy orchestration by using a lightweight script to monitor the queue and adjust container counts on the fly. Built on a Docker Compose foundation, the setup has been verified on an 8-core, 16GB VPS and is capable of managing hundreds of concurrent executions.

The architecture connects several key components: the primary n8n instance pushes jobs into a Redis queue, while a dedicated autoscaler and a Redis monitor track the queue's activity. Depending on the queue depth, the autoscaler adds or removes worker containers as needed. PostgreSQL serves as the persistent data store, and the main n8n instance is configured to communicate with an n8n webhook service.

Key Features of the Autoscaling System

1. Reactive Scaling. Worker counts adjust automatically in response to the growth or shrinkage of the Redis queue.
2. Granular Configuration. Define your own scaling thresholds and limits to match specific workload requirements.
3. Queue Visibility. A built-in monitor provides real-time insights into the Redis queue, so you always know exactly how many jobs are pending.
4. Simple Deployment. The entire stack is managed through a standard Docker Compose file, making it easy to deploy and maintain.
5. Stability and Health Monitoring. Every service includes integrated health checks to ensure the system remains reliable under heavy load.

Before You Start

Ensure that Docker and Docker Compose are installed on your system. If you are on Ubuntu, you can use the official convenience script; otherwise, Docker Desktop is recommended for local environments.

1. Clone or copy the repository files into your preferred directory.
2. Configure your environment variables in the .env file. While the defaults are functional for testing, you should update passwords and security tokens for any production use.
3. Launch the stack by running docker compose up -d.

Detailed Configuration

1. Security Defaults. Update the default passwords and encryption keys in the .env file before going live.
2. Worker Concurrency. By default, each worker is configured to handle 10 tasks simultaneously. You can modify this by changing N8N_CONCURRENCY_PRODUCTION_LIMIT=10 in the docker-compose.yml file.
3. Shutdown Timeouts. Adjust N8N_QUEUE_BULL_GRACEFULSHUTDOWNTIMEOUT=300 and N8N_GRACEFUL_SHUTDOWN_TIMEOUT=300 (in seconds). These should be set to a duration longer than your longest-running workflow to prevent data loss during scaling events.
4. Core Environment Variables

5. Mandatory n8n Settings. To enable queue mode, ensure the following are included in your n8n environment configuration:

EXECUTIONS_MODE=queue

QUEUE_BULL_REDIS_HOST=redis

QUEUE_HEALTH_CHECK_ACTIVE=true

How the Scaling Logic Works

The autoscaler evaluates the Redis queue at every POLLING_INTERVAL_SECONDS.

Scaling Up: If the queue length exceeds the SCALE_UP_QUEUE_THRESHOLD and the worker count is currently below MAX_REPLICAS, the autoscaler launches additional workers.

Scaling Down: If the queue length drops below the SCALE_DOWN_QUEUE_THRESHOLD and the worker count is above MIN_REPLICAS, the autoscaler terminates excess containers.

A cooldown period is enforced after every scaling event to prevent "thrashing"—the rapid, unnecessary starting and stopping of containers due to minor fluctuations in the queue.

Monitoring and Troubleshooting

Monitoring. The system includes a dedicated Redis monitor service. You can track the health of all services via Docker health checks, and the autoscaler provides detailed activity logs for auditing.

Troubleshooting Tips

• Review container logs: docker-compose logs [service_name]
• Confirm Redis connectivity: docker-compose exec redis redis-cli ping
• Manual queue check: To see the number of waiting jobs, run: docker-compose exec redis redis-cli LLEN bull:jobs:wait

Important Note on Webhook URLs: When configuring webhooks within this setup, use the Docker service name rather than localhost. For example: http://n8n-webhook:5678/webhook/[unique-id].