Docker-based Deployment

The Helium system is designed with a multi-worker architecture that can be deployed using containers. Each worker type serves a specific purpose and has different scaling requirements. This deployment approach provides:

• Scalability: Independent scaling of different worker types based on load
• Reliability: Fault isolation between different services
• Flexibility: Easy deployment across different environments
• Maintainability: Simplified updates and rollbacks

Prerequisites

Before proceeding with this guide, ensure you have:

• External dependencies configured (see External Dependencies)
• Docker or container runtime installed
• Kubernetes cluster (for Kubernetes deployment)
• Basic understanding of containerization concepts

Container Architecture

Worker Types and Scaling Patterns

The Helium server supports six distinct worker modes, each with specific scaling characteristics:

Scaling Constraints

⚠️ Critical Scaling Limitations

mailer Worker:

• Recommendation: Deploy as single instance only
• Reason: Relies on SMTP server connections and may cause email delivery issues with multiple instances
• Impact: Multiple mailer instances can lead to duplicate emails or SMTP rate limiting

cron_executor Worker:

• Requirement: MUST have exactly one instance
• Reason: Scans the database to check for scheduled tasks in the queue
• Impact: Multiple instances will cause duplicate task execution and potential data corruption

✅ Scalable Workers

API Workers (grpc, subscribe_api, webhook_api):

• Can be horizontally scaled based on traffic demands
• Support standard load balancing techniques
• Share state through external Redis and PostgreSQL

consumer Worker:

• Can run multiple instances for processing message queues
• Automatically distributes work through RabbitMQ

Docker Image

Building the Docker Image

The project includes a multi-stage Dockerfile optimized for production:

# Build the Docker image
docker build -t helium-server:latest .

# Tag for registry
docker tag helium-server:latest your-registry/helium-server:v1.0.0

# Push to registry
docker push your-registry/helium-server:v1.0.0

Image Characteristics

• Base Image: gcr.io/distroless/cc for minimal attack surface
• Size: ~50MB final image
• Architecture: Multi-arch support (amd64, arm64)
• Security: Non-root user, minimal dependencies

Environment Variables

Configure containers using these environment variables:

# Required - Worker mode selection
WORK_MODE=grpc  # grpc, subscribe_api, webhook_api, consumer, mailer, cron_executor

# Required - Database connections
DATABASE_URL=postgres://user:password@postgres-host:5432/helium_db
REDIS_URL=redis://redis-host:6379
MQ_URL=amqp://user:password@rabbitmq-host:5672/

# Optional - Server configuration
LISTEN_ADDR=0.0.0.0:50051  # For API workers
SCAN_INTERVAL=60           # For cron_executor only
RUST_LOG=info             # Logging level

Docker Compose Deployment

For development or simple production setups:

version: "3.8"

services:
  # Main gRPC API (scalable)
  helium-grpc:
    image: helium-server:latest
    ports:
      - "50051:50051"
    environment:
      WORK_MODE: grpc
      DATABASE_URL: postgres://helium:password@postgres:5432/helium_db
      REDIS_URL: redis://redis:6379
      MQ_URL: amqp://helium:password@rabbitmq:5672/
      LISTEN_ADDR: 0.0.0.0:50051
    depends_on:
      - postgres
      - redis
      - rabbitmq
    restart: unless-stopped
    deploy:
      replicas: 2 # Can be scaled horizontally

  # Subscription API (scalable)
  helium-subscribe-api:
    image: helium-server:latest
    ports:
      - "8080:8080"
    environment:
      WORK_MODE: subscribe_api
      DATABASE_URL: postgres://helium:password@postgres:5432/helium_db
      REDIS_URL: redis://redis:6379
      MQ_URL: amqp://helium:password@rabbitmq:5672/
      LISTEN_ADDR: 0.0.0.0:8080
    depends_on:
      - postgres
      - redis
      - rabbitmq
    restart: unless-stopped
    deploy:
      replicas: 2 # Can be scaled horizontally

  # Webhook API (scalable)
  helium-webhook-api:
    image: helium-server:latest
    ports:
      - "8081:8081"
    environment:
      WORK_MODE: webhook_api
      DATABASE_URL: postgres://helium:password@postgres:5432/helium_db
      REDIS_URL: redis://redis:6379
      MQ_URL: amqp://helium:password@rabbitmq:5672/
      LISTEN_ADDR: 0.0.0.0:8081
    depends_on:
      - postgres
      - redis
      - rabbitmq
    restart: unless-stopped
    deploy:
      replicas: 2 # Can be scaled horizontally

  # Background consumer (scalable)
  helium-consumer:
    image: helium-server:latest
    environment:
      WORK_MODE: consumer
      DATABASE_URL: postgres://helium:password@postgres:5432/helium_db
      REDIS_URL: redis://redis:6379
      MQ_URL: amqp://helium:password@rabbitmq:5672/
    depends_on:
      - postgres
      - redis
      - rabbitmq
    restart: unless-stopped
    deploy:
      replicas: 3 # Can run multiple instances

  # Mailer service (single instance recommended)
  helium-mailer:
    image: helium-server:latest
    environment:
      WORK_MODE: mailer
      DATABASE_URL: postgres://helium:password@postgres:5432/helium_db
      REDIS_URL: redis://redis:6379
      MQ_URL: amqp://helium:password@rabbitmq:5672/
    depends_on:
      - postgres
      - redis
      - rabbitmq
    restart: unless-stopped
    deploy:
      replicas: 1 # SINGLE INSTANCE ONLY

  # Cron executor (must be single instance)
  helium-cron:
    image: helium-server:latest
    environment:
      WORK_MODE: cron_executor
      DATABASE_URL: postgres://helium:password@postgres:5432/helium_db
      REDIS_URL: redis://redis:6379
      MQ_URL: amqp://helium:password@rabbitmq:5672/
      SCAN_INTERVAL: 60
    depends_on:
      - postgres
      - redis
      - rabbitmq
    restart: unless-stopped
    deploy:
      replicas: 1 # MUST BE EXACTLY 1

  # External dependencies (for development only)
  postgres:
    image: postgres:15
    environment:
      POSTGRES_USER: helium
      POSTGRES_PASSWORD: password
      POSTGRES_DB: helium_db
    volumes:
      - postgres_data:/var/lib/postgresql/data
    ports:
      - "5432:5432"

  redis:
    image: redis:7
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data

  rabbitmq:
    image: rabbitmq:3-management
    environment:
      RABBITMQ_DEFAULT_USER: helium
      RABBITMQ_DEFAULT_PASS: password
    ports:
      - "5672:5672"
      - "15672:15672"
    volumes:
      - rabbitmq_data:/var/lib/rabbitmq

volumes:
  postgres_data:
  redis_data:
  rabbitmq_data:

Kubernetes Deployment

For production Kubernetes deployments:

Namespace and ConfigMap

apiVersion: v1
kind: Namespace
metadata:
  name: helium-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: helium-config
  namespace: helium-system
data:
  RUST_LOG: "info"
  SCAN_INTERVAL: "60"

Secrets

apiVersion: v1
kind: Secret
metadata:
  name: helium-secrets
  namespace: helium-system
type: Opaque
stringData:
  database-url: "postgres://helium:password@postgres-service:5432/helium_db"
  redis-url: "redis://redis-service:6379"
  rabbitmq-url: "amqp://helium:password@rabbitmq-service:5672/"

gRPC API Deployment (Scalable)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: helium-grpc
  namespace: helium-system
spec:
  replicas: 3 # Can be scaled horizontally
  selector:
    matchLabels:
      app: helium-grpc
  template:
    metadata:
      labels:
        app: helium-grpc
    spec:
      containers:
        - name: helium-server
          image: your-registry/helium-server:v1.0.0
          ports:
            - containerPort: 50051
          env:
            - name: WORK_MODE
              value: "grpc"
            - name: LISTEN_ADDR
              value: "0.0.0.0:50051"
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: helium-secrets
                  key: database-url
            - name: REDIS_URL
              valueFrom:
                secretKeyRef:
                  name: helium-secrets
                  key: redis-url
            - name: MQ_URL
              valueFrom:
                secretKeyRef:
                  name: helium-secrets
                  key: rabbitmq-url
          envFrom:
            - configMapRef:
                name: helium-config
          resources:
            requests:
              memory: "128Mi"
              cpu: "100m"
            limits:
              memory: "512Mi"
              cpu: "500m"
          livenessProbe:
            tcpSocket:
              port: 50051
            initialDelaySeconds: 30
            periodSeconds: 10
          readinessProbe:
            tcpSocket:
              port: 50051
            initialDelaySeconds: 5
            periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: helium-grpc-service
  namespace: helium-system
spec:
  selector:
    app: helium-grpc
  ports:
    - port: 50051
      targetPort: 50051
  type: ClusterIP

Consumer Deployment (Scalable)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: helium-consumer
  namespace: helium-system
spec:
  replicas: 3 # Can run multiple instances
  selector:
    matchLabels:
      app: helium-consumer
  template:
    metadata:
      labels:
        app: helium-consumer
    spec:
      containers:
        - name: helium-server
          image: your-registry/helium-server:v1.0.0
          env:
            - name: WORK_MODE
              value: "consumer"
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: helium-secrets
                  key: database-url
            - name: REDIS_URL
              valueFrom:
                secretKeyRef:
                  name: helium-secrets
                  key: redis-url
            - name: MQ_URL
              valueFrom:
                secretKeyRef:
                  name: helium-secrets
                  key: rabbitmq-url
          envFrom:
            - configMapRef:
                name: helium-config
          resources:
            requests:
              memory: "256Mi"
              cpu: "200m"
            limits:
              memory: "1Gi"
              cpu: "1000m"
          livenessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - "ps aux | grep helium-server | grep -v grep"
            initialDelaySeconds: 30
            periodSeconds: 30

Mailer Deployment (Single Instance)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: helium-mailer
  namespace: helium-system
spec:
  replicas: 1 # SINGLE INSTANCE ONLY
  strategy:
    type: Recreate # Prevent multiple instances during updates
  selector:
    matchLabels:
      app: helium-mailer
  template:
    metadata:
      labels:
        app: helium-mailer
    spec:
      containers:
        - name: helium-server
          image: your-registry/helium-server:v1.0.0
          env:
            - name: WORK_MODE
              value: "mailer"
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: helium-secrets
                  key: database-url
            - name: REDIS_URL
              valueFrom:
                secretKeyRef:
                  name: helium-secrets
                  key: redis-url
            - name: MQ_URL
              valueFrom:
                secretKeyRef:
                  name: helium-secrets
                  key: rabbitmq-url
          envFrom:
            - configMapRef:
                name: helium-config
          resources:
            requests:
              memory: "128Mi"
              cpu: "100m"
            limits:
              memory: "512Mi"
              cpu: "500m"

Cron Executor Deployment (Singleton)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: helium-cron
  namespace: helium-system
spec:
  replicas: 1 # MUST BE EXACTLY 1
  strategy:
    type: Recreate # Ensure no overlap during updates
  selector:
    matchLabels:
      app: helium-cron
  template:
    metadata:
      labels:
        app: helium-cron
    spec:
      containers:
        - name: helium-server
          image: your-registry/helium-server:v1.0.0
          env:
            - name: WORK_MODE
              value: "cron_executor"
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: helium-secrets
                  key: database-url
            - name: REDIS_URL
              valueFrom:
                secretKeyRef:
                  name: helium-secrets
                  key: redis-url
            - name: MQ_URL
              valueFrom:
                secretKeyRef:
                  name: helium-secrets
                  key: rabbitmq-url
            - name: SCAN_INTERVAL
              value: "60"
          envFrom:
            - configMapRef:
                name: helium-config
          resources:
            requests:
              memory: "128Mi"
              cpu: "50m"
            limits:
              memory: "256Mi"
              cpu: "200m"
          livenessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - "ps aux | grep helium-server | grep -v grep"
            initialDelaySeconds: 60
            periodSeconds: 30

Horizontal Pod Autoscaler (HPA)

For scalable workers, configure automatic scaling:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: helium-grpc-hpa
  namespace: helium-system
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: helium-grpc
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80

Load Balancer Configuration

Ingress for API Services

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: helium-ingress
  namespace: helium-system
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  tls:
    - hosts:
        - api.your-domain.com
      secretName: helium-tls
  rules:
    - host: api.your-domain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: helium-grpc-service
                port:
                  number: 50051

Service Mesh Configuration

For advanced deployments with service mesh (Istio):

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: helium-grpc-vs
  namespace: helium-system
spec:
  hosts:
    - api.your-domain.com
  gateways:
    - helium-gateway
  http:
    - match:
        - uri:
            prefix: /
      route:
        - destination:
            host: helium-grpc-service
            port:
              number: 50051
          weight: 100
      fault:
        delay:
          percentage:
            value: 0.1
          fixedDelay: 5s

Database Migration

Database migrations must be run before starting any workers:

Migration Job

apiVersion: batch/v1
kind: Job
metadata:
  name: helium-migration
  namespace: helium-system
spec:
  template:
    spec:
      containers:
        - name: migration
          image: your-registry/helium-server:v1.0.0
          command: ["sqlx", "migrate", "run"]
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: helium-secrets
                  key: database-url
      restartPolicy: Never
  backoffLimit: 3

Init Container for Workers

Add to all worker deployments:

spec:
  template:
    spec:
      initContainers:
        - name: wait-for-migration
          image: postgres:15
          command:
            [
              "sh",
              "-c",
              "until pg_isready -h postgres-service -p 5432; do echo waiting for database; sleep 2; done;",
            ]
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: helium-secrets
                  key: database-url

Monitoring and Observability

Health Checks

Configure appropriate health checks for each worker type:

# For API workers (gRPC, REST)
livenessProbe:
  tcpSocket:
    port: 50051
  initialDelaySeconds: 30
  periodSeconds: 10

# For background workers (consumer, mailer, cron)
livenessProbe:
  exec:
    command:
    - /bin/sh
    - -c
    - "ps aux | grep helium-server | grep -v grep"
  initialDelaySeconds: 30
  periodSeconds: 30

Logging Configuration

env:
  - name: RUST_LOG
    value: "info,helium_server=debug" # Adjust as needed

Metrics Collection

Use Prometheus for metrics collection:

apiVersion: v1
kind: Service
metadata:
  name: helium-metrics
  namespace: helium-system
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"
    prometheus.io/path: "/metrics"
spec:
  selector:
    app: helium-grpc
  ports:
    - port: 8080
      name: metrics

Troubleshooting

Common Issues

Pod Crash Loop:

# Check logs
kubectl logs -n helium-system deployment/helium-grpc

# Check events
kubectl get events -n helium-system --sort-by='.metadata.creationTimestamp'

# Verify environment variables
kubectl exec -n helium-system deployment/helium-grpc -- env | grep -E "(DATABASE_URL|REDIS_URL|MQ_URL)"

Multiple Cron Executors:

# Check for multiple cron instances (should show only 1)
kubectl get pods -n helium-system -l app=helium-cron

# Check cron logs for conflicts
kubectl logs -n helium-system -l app=helium-cron --tail=100

Database Connection Issues:

# Test database connectivity
kubectl run -i --tty --rm debug --image=postgres:15 --restart=Never -- \
  psql postgresql://user:password@postgres-service:5432/helium_db -c "SELECT version();"

# Check migration status
kubectl exec -n helium-system deployment/helium-grpc -- \
  sqlx migrate info --database-url $DATABASE_URL

Performance Tuning

Resource Limits:

• API workers: 200-500m CPU, 256Mi-1Gi RAM per pod
• Consumer workers: 500m-1 CPU, 512Mi-2Gi RAM per pod
• Mailer/Cron: 100-200m CPU, 128-512Mi RAM per pod

Scaling Guidelines:

• Start with 2-3 replicas for API workers
• Scale consumers based on message queue depth
• Monitor CPU/memory usage and adjust limits accordingly

External Dependencies

Refer to the External Dependencies Guide for detailed information about:

• PostgreSQL setup and configuration
• Redis configuration and clustering
• RabbitMQ setup and management
• SMTP server configuration
• OAuth provider setup
• Payment provider integration

Configuration Management

Refer to the Configuration Guide for:

• Environment variable reference
• Configuration file formats
• Runtime configuration updates
• Security best practices

Next Steps

After successful deployment:

1. Configure monitoring and alerting
2. Set up backup procedures for stateful data
3. Implement CI/CD pipelines for automated deployments
4. Configure log aggregation and analysis
5. Plan disaster recovery procedures

For specific configuration details, see the Helium Server Configuration guide.