High Availability & Scaling
Running multiple backend instances, MongoDB replica sets, Redis clustering, and load balancing.
High Availability & Scaling
For production workloads requiring redundancy and horizontal scale, this guide covers running multiple backend instances, configuring MongoDB with replica sets, and load balancing.
Backend Horizontal Scaling
Both the Admin Backend and Main Backend are stateless — all state lives in MongoDB and Redis. You can run multiple instances behind a load balancer with no coordination required.
Docker Compose Scale
docker compose up --scale payment-gateway-admin-backend=3 \
--scale payment-gateway-main-backend=3Ensure the load balancer (Caddy or an external LB) routes traffic across all instances.
Sticky Sessions
The Admin Backend uses cookie-based sessions for WebAuthn flows. Configure your load balancer with sticky sessions (session affinity by cookie) for the Admin Backend only — or ensure the shared Redis cache is configured so any instance can serve a given session.
[!NOTE] API key authentication is stateless and works with any instance. Sticky sessions are only needed for the WebAuthn authentication flow.
MongoDB Replica Set
A single MongoDB node is a single point of failure. For production, configure a 3-node replica set.
Minimal Replica Set (docker-compose)
services:
mongo1:
image: mongo:7
command: ["--replSet", "rs0", "--bind_ip_all"]
ports:
- "27017:27017"
mongo2:
image: mongo:7
command: ["--replSet", "rs0", "--bind_ip_all"]
mongo3:
image: mongo:7
command: ["--replSet", "rs0", "--bind_ip_all"]Initialize the replica set (run once):
docker exec -it mongo1 mongosh --eval "
rs.initiate({
_id: 'rs0',
members: [
{ _id: 0, host: 'mongo1:27017' },
{ _id: 1, host: 'mongo2:27017' },
{ _id: 2, host: 'mongo3:27017' }
]
})
"Update your connection string:
MPG_DATABASE_URI=mongodb://mongo1:27017,mongo2:27017,mongo3:27017/?replicaSet=rs0[!IMPORTANT] Transactions and change streams (used by some worker processes) require a replica set. A standalone MongoDB node will not support these features.
Redis / Garnet High Availability
For high availability of the cache (DEK caching, session data, rate limiting state):
Redis Sentinel (Automatic Failover)
Use Redis Sentinel for automatic primary failover:
services:
redis-primary:
image: redis:7-alpine
command: redis-server --save 60 1
redis-replica:
image: redis:7-alpine
command: redis-server --replicaof redis-primary 6379
redis-sentinel:
image: redis:7-alpine
command: redis-sentinel /etc/redis/sentinel.confConnect using a Sentinel-aware URL:
MPG_GARNET_URL=redis-sentinel://sentinel1:26379,sentinel2:26379/0?master=mymasterGarnet (Microsoft's Redis-Compatible Cache)
The project ships with Garnet as its default Redis-compatible cache. Garnet is a drop-in replacement — the same MPG_GARNET_URL configuration applies.
Load Balancer Configuration
Caddy (included in the payment-gateway-reverseproxy module) can load balance across multiple backend instances:
admin.yourcompany.com {
reverse_proxy admin-backend-1:8080 admin-backend-2:8080 admin-backend-3:8080 {
lb_policy round_robin
health_uri /health
health_interval 10s
}
}For the Admin Backend's WebAuthn flows, add sticky sessions:
admin.yourcompany.com {
reverse_proxy admin-backend-1:8080 admin-backend-2:8080 {
lb_policy cookie {
name ADMIN_SESSION_AFFINITY
}
}
}Worker Process Scaling
The PDF worker (cmd/worker) is a separate process from the main backend. Multiple worker instances can run concurrently — they pull jobs from a shared queue in MongoDB.
# Run 2 worker instances with 5 concurrent jobs each
docker compose up --scale payment-gateway-main-worker=2Set MPG_MAIN_WORKER_CONCURRENCY to control per-instance parallelism.
Connection Pool Tuning
Increase MongoDB connection pool settings for high-traffic deployments:
MPG_DATABASE_MAX_POOL_SIZE=200
MPG_DATABASE_MIN_POOL_SIZE=20
MPG_DATABASE_MAX_CONNECTING=20[!NOTE] Total MongoDB connections =
instances × MAX_POOL_SIZE. Ensure your MongoDB instance (or Atlas tier) supports this limit.
Production HA Checklist
- MongoDB 3-node replica set configured
- Redis/Garnet with Sentinel or cluster mode
- Minimum 2 Admin Backend instances
- Minimum 2 Main Backend instances
- Load balancer with health checks on
/health - Sticky sessions enabled for Admin Backend WebAuthn flows
- Connection pool sizes tuned for instance count
- OpenTelemetry connected for distributed trace correlation
- Automated backups configured (see Backup & Recovery)
Operating the durable IPN queue
These instructions apply to a gateway release containing versioned IPN delivery and the IPN format controls. The main backend API process owns IPN sending. The Admin backend only accepts audited queue commands and reads delivery metadata; no additional IPN service is required. Both backends must use the same release and MongoDB database. The bundled deployment's replica-set and writable-primary checks support the atomic payment/IPN handoff; preserve its existing database volume during upgrades.
Keep MPG_IPN_ALLOW_PRIVATE_TARGETS disabled in production unless an explicitly trusted internal integration requires it. The worker rechecks destination safety and current tenant/site/payment eligibility before each attempt. A readiness probe uses the same outbound safeguards.
Monitor queued/retry-scheduled counts, oldest due time, exhausted/terminal outcomes, failed construction commands, and attempt outcome/HTTP-status logs. Inspect metadata rather than payloads or callback URLs. If work stops progressing, check main-backend health and MongoDB primary availability before requesting a retry. Fix authentication, receiver compatibility, blocked-target or reconciliation errors first.
The automatic v2 budget is 20 attempts within a maximum 48-hour window; the attempt limit can exhaust sooner. V1 keeps three short attempts in the same queue, with a bounded two-minute automatic cycle. Attempt details expire after 30 days, but replay and ordering receipts remain. Keep the system encryption key and database recoverable together, because queued destinations and payloads are encrypted.
IPN upgrade and rollback
Upgrade to the matching main backend, Admin backend and frontend release before permitting v2 traffic; arbitrary mixtures of older and newer producers are not supported. Upgrade compatible published receivers before enabling v2, and keep old checkout callback URLs active. Existing sites are not backfilled to v2; only newly created sites default to it. Site rollback to v1 applies to fresh checkouts, while existing v2 checkout identities stay v2 for later events and queued deliveries remain immutable.
Before reverting to gateway binaries that predate the durable queue, stop new work and drain pending deliveries/construction commands with the compatible release. Older binaries do not process these collections. Do not delete the database, queue or receiver state to force a downgrade. A failed command with unreconstructible historical input requires support investigation, not a fabricated new event.
Legacy support lasts through 30 June 2027. That date does not disable the sender. Removing v1 is a separate announced major release after migration and drain; preserve ordering/effect protection for retained transactions when removing the compatibility adapter.
IPN monitoring and alerts
With the existing OpenTelemetry exporter enabled, monitor:
| Gauge | Fixed labels | Meaning |
|---|---|---|
payment_gateway.ipn.backlog | kind, state | Active delivery or command records. |
payment_gateway.ipn.failures | kind, state | Stored exhausted/terminal deliveries or failed commands, not cumulative failure events. |
payment_gateway.ipn.oldest_due_age | kind | Seconds overdue for the oldest pending or retry-scheduled item. |
payment_gateway.ipn.collection_success | none | 1 for successful collection, 0 on failure. |
payment_gateway.ipn.collection_last_success_unixtime | none | Unix time of the last successful sample; 0 before first success. |
kind is delivery or command. There are no tenant, URL, transaction, customer or secret labels. Samples are collected at most once per minute with a combined five-second database-query deadline. Collection failure omits workload observations and reports health 0; missing observations are not proof of an empty queue. Aggregate with max, not sum, across main replicas that observe the same database.
Suggested starting alerts are failure counts sustained for five minutes, oldest due age above five minutes across two samples, collection health 0, or last successful collection older than three minutes. Tune these thresholds for the deployment. These instructions do not install production alert rules.
Use the Transactions IPN needs attention checkbox or ipnNeedsAttention=true query to find unresolved historical deliveries and failed construction work. Metrics describe stored states: an old failed command may remain counted after its matching delivery recovered, while the attention filter excludes that resolved construction failure. Inspect the transaction's delivery and command history before replay.