D2 technology · first-party infrastructure proof
Self-hosted n8n production infrastructure on Docker Swarm.
This is the production architecture behind D2 Group's own n8n runtime. It separates ingress, request handling, queue execution, Code-node runners and durable state — with guarded deployment, encrypted recovery and operational change control around the workflows themselves.
n8n.d2group.co
Docker Swarm production topology
Upstream TLS proxy
Sets origin access token
Caddy origin
Routing · origin guard · headers
n8n main
Editor · REST · OAuth · test endpoints
Webhook pool ×2
Webhook · form · MCP production
Redis / Bull
Shared execution queue
Worker 1
Queue consumer · concurrency 8 default
Worker 2
Queue consumer · concurrency 8 default
Runner · main
Isolated task runners · Code execution
Runner · worker 1
One-to-one broker pairing
Runner · worker 2
One-to-one broker pairing
PostgreSQL 16
Workflow state · execution data · binary data · durable database state
Docker Secrets
Encryption · DB · Redis · runner auth
1
main / editor
Control plane · REST · OAuth
2
webhook processors
Production webhook · form · MCP
2
worker services
Queue consumers · concurrency 8 default
3
external runner pairs
Main + one-to-one worker runners
Production engineering
This is not a demo container.
The system is built around the failure modes that appear after automation becomes operational infrastructure: concurrent executions, Code-node isolation, secret handling, database migrations, stale backups, direct-origin access and workflow drift.
Code task blocks
External runners + explicit task timeout
Worker/process failure
Queue execution + health-gated services
Secret exposure
Docker Secrets + runtime bridge
Unsafe upgrade
Immutable images + recovery point + guarded migration
Backup exists but fails
Verification + disposable restore drill
Live workflow diverges
Private Git export + drift detection
Runtime topology
One physical host. Multiple production responsibilities.
Docker Swarm provides service orchestration on one manager node. Queue mode separates request handling from execution, while dedicated external task runners isolate JavaScript and Python Code-node work from the n8n processes that coordinate it.
n8n.d2group.co
Docker Swarm production topology
Upstream TLS proxy
Sets origin access token
Caddy origin
Routing · origin guard · headers
n8n main
Editor · REST · OAuth · test endpoints
Webhook pool ×2
Webhook · form · MCP production
Redis / Bull
Shared execution queue
Worker 1
Queue consumer · concurrency 8 default
Worker 2
Queue consumer · concurrency 8 default
Runner · main
Isolated task runners · Code execution
Runner · worker 1
One-to-one broker pairing
Runner · worker 2
One-to-one broker pairing
PostgreSQL 16
Workflow state · execution data · binary data · durable database state
Docker Secrets
Encryption · DB · Redis · runner auth
Network model
edge
Caddy ↔ main / webhook processors
backend
PostgreSQL · Redis · n8n internals · runner brokers
egress
Outbound access for workers / runners
Queue semantics
One shared Bull queue.
Worker 1 and Worker 2 are named services for stable one-to-one runner pairing. They are not separate workload queues.
Execution path
A production request does not execute inside the webhook process.
Production webhook, form and MCP traffic enters through the guarded origin, is handled by the webhook pool, queued through Redis/Bull and consumed by workers. Workflow state and binary/execution data remain durable in PostgreSQL.
Caddy
guard + route
Webhook pool
receive request
Redis / Bull
queue job
Worker
consume execution
Task runner
Code isolation
PostgreSQL
durable state
Test traffic
webhook-test · form-test · mcp-test → main/editor
Production traffic
webhook · form · MCP → horizontally-scaled webhook pool
Execution
workers consume the shared queue; Code runs in external runners
Production invariants
Reliability starts before the first workflow node.
D2 treats runtime identity, health, secrets, recovery and deployment preconditions as code-level operating rules rather than documentation-only recommendations.
Immutable runtime identity
n8n, runner and proxy images are pinned by immutable identity so a normal deploy cannot silently change the runtime version.
Health gates
Deployment waits for real service health, including n8n readiness plus runner launcher and broker health — not merely container state.
Docker Secrets bridge
Sensitive values stay outside Git and the Swarm environment specification, then are bridged into supported n8n runtime variables at startup.
Durable queue-mode data
Execution data and binary payloads use PostgreSQL database modes; Redis transports jobs but is not the authoritative execution store.
Guarded deploy path
Production changes pass through explicit gates.
01
Validate config
02
Create recovery point
03
Verify backup
04
Pull exact images
05
Deploy Swarm services
06
Wait for health gates
Security boundary
The origin, secrets and internal services have explicit boundaries.
The public edge is separated from backend services. PostgreSQL, Redis and runner brokers publish no host ports, sensitive application values remain Docker Secrets and Caddy requires an upstream origin token before forwarding real traffic.
Origin guard
Caddy requires a secret upstream header for real traffic and strips it before requests reach n8n.
Private backend
PostgreSQL and Redis stay on internal Docker networks with no host ports published.
Private runner brokers
Task-runner broker traffic remains inside Docker networks; runner/broker host ports are not exposed.
Metrics boundary
Metrics routes are blocked at the application origin instead of being exposed through the public proxy path.
Origin request
Direct-origin traffic must prove it came through the upstream proxy.
Upstream proxy
Overwrite / set token
Caddy
Validate token
n8n
Token stripped upstream
Backup & disaster recovery
A backup is only useful if the restore path is executable.
Production changes create verified recovery points. Offsite material is encrypted before upload, and a disposable PostgreSQL restore drill validates that recovery data can actually be restored before it is needed in an incident.
Verified recovery point
Normal production changes create and verify a recovery point before deployment continues.
Encrypted offsite backup
Recovery material is encrypted before offsite upload and plaintext staging is cleaned after success or failure.
Disposable restore drill
A real PostgreSQL restore path validates that backup material can actually rebuild database state.
Guarded production restore
Full-host recovery refuses conflicting stack state, missing inputs, repository mismatch or mutable runtime images.
Recovery logic
Image rollback alone is not a safe database rollback strategy.
n8n upgrades can include database migrations. Recovery therefore records the repository/runtime state and restores data intentionally instead of assuming an older image can reverse schema changes.
01
Backup
02
Encrypt
03
Restore drill
04
Post-flight checks
Operating control
Operate the platform and the workflow definitions.
Monitoring covers service readiness, queue/database signals, failed executions, disk pressure and backup age. Production workflow definitions can be exported to private Git and compared with live UI changes to surface drift.
Platform health
Service health, n8n readiness, Redis queue/memory, PostgreSQL connections, failures and disk pressure are monitored.
Backup age
A stale recovery point is treated as an operational exception instead of assuming scheduled backup means recoverable backup.
Workflow drift
Production workflows can be exported to private Git and compared with live definitions changed through the UI.
CI invariants
Repository checks cover production invariants, pinned images, secret bridging, runner tooling and live broker/runner smoke topology.
Every 5 min
Health / capacity monitor
Daily · 03:15
Encrypted offsite backup when configured
Daily · 04:00
Workflow drift check
Architecture boundary
Production-ready does not mean pretending a single host is highly available.
PostgreSQL, Redis, Caddy and the Docker host remain single points of failure. Queue mode improves concurrency and responsibility separation; resilience comes from recovery controls, not from calling one machine HA.
What the architecture does provide
Does not claim
What this proves
D2 can engineer the runtime behind automation — not only the workflows on top of it.
This first-party stack demonstrates the infrastructure discipline D2 applies when automation becomes business-critical: architecture, deployment safety, failure handling, observability, recovery and change control.
Architecture
Topology · network · execution model
Reliability
Health gates · timeouts · durable state
Recovery
Verified backup · restore drill · DR
Change control
Pinned runtime · CI · workflow drift
n8n Automation & API Integration
Need n8n workflow engineering and API integration?
D2 designs workflows on top of D2-managed or client-managed runtimes, with scope defined around business events, APIs, failure modes, retries, idempotency and operating ownership.