leocrm/docs/infrastructure.md

Infrastructure Guide

Version: 2.0
Date: 2026-08-20
Applies to: System administrators and DevOps engineers

---

1. Docker-Compose Stack

LeoCRM läuft als Docker-Compose-Stack mit 4 Services:

Service	Image	Beschreibung
postgres	pgvector/pgvector:pg16	PostgreSQL 16 mit pgvector Extension für Vector Search
redis	redis:7-alpine	Redis 7 für Sessions, Caching, Pub/Sub, ARQ Queue
crm_app	Multi-Stage Build	FastAPI Backend (uvicorn), API + WebSocket
crm_worker	Multi-Stage Build	ARQ Background Worker (Cron-Jobs, Queue Processing)

docker-compose.yaml Übersicht

services:
  postgres:
    image: pgvector/pgvector:pg16
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U crm_user -d crm_db"]
      interval: 10s
      timeout: 5s
      retries: 5

  redis:
    image: redis:7-alpine
    volumes:
      - redis_data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5

  crm_app:
    build: .
    command: ["./prestart.sh"]
    depends_on:
      postgres: { condition: service_healthy }
      redis: { condition: service_healthy }
    healthcheck:
      test: ["CMD-SHELL", "bash /app/healthcheck.sh"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 15s

  crm_worker:
    build: .
    command: ["./worker.sh"]
    depends_on:
      postgres: { condition: service_healthy }
      redis: { condition: service_healthy }

Container-Entrypoints

Datei	Service	Funktion
prestart.sh	crm_app	Alembic-Migrationen → DB-Role-Passwörter → Plugin-Schema-Sync → Admin-Seed → uvicorn Start
worker.sh	crm_worker	ARQ Worker Start mit Cron-Jobs
healthcheck.sh	crm_app	HTTP /api/v1/health oder Redis-Ping

Volumes

Volume	Service	Beschreibung
postgres_data	postgres	PostgreSQL Daten (persistent)
redis_data	redis	Redis Snapshot (persistent)
app_storage	crm_app	File Storage (DMS, Uploads)

---

2. Coolify Deployment

LeoCRM ist über Coolify auf einem Hetzner VPS deployiert.

Server-Info

Eigenschaft	Wert
Host	46.225.91.159
Hostname	coolify-01
Provider	Hetzner VPS
Coolify URL	https://server.media-on.de
App URL	https://crm.media-on.de
App UUID	xf7smknlger3hvkrsb910tui
Container-Name	Ändert sich bei jedem Coolify-Deploy (Suffix)

Deploy-Methoden

Frontend-Only Deploy (~20s)

bash /a0/usr/projects/leocrm/scripts/fast-deploy.sh frontend

• Baut Frontend lokal, kopiert dist/ direkt in den laufenden Container
• Kein Coolify-Rebuild, kein Docker-Image-Neubau
• Container wird nicht neu gestartet

Full Deploy (~2min, für Backend-Änderungen)

bash /a0/usr/projects/leocrm/scripts/fast-deploy.sh full

• Triggert Coolify-Rebuild über deploy.py
• Für Python-Code, Requirements, Migrations

Server-Container

Auf dem Hetzner VPS laufen ~25 Docker-Container (Coolify + Services):

• Coolify Proxy (Traefik)
• Coolify Dashboard
• Coolify Database
• LeoCRM Stack (postgres, redis, crm_app, crm_worker)
• Weitere Coolify-managed Services

Container-Übersicht auf dem Server:

ssh -i ~/.ssh/coolify-01-root root@46.225.91.159 'docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"'

---

3. ARQ Worker & Cron-Jobs

Der crm_worker Service läuft ARQ (Async Redis Queue) für Background-Jobs.

Registrierte Cron-Jobs

Job	Schedule	Beschreibung
auto_backup_job	Täglich 03:00	Automatisches Backup (ruft scripts/backup.py auf)
audit_retention_cleanup	Täglich 04:00	Audit-Logs älter als 365 Tage archivieren/löschen
cleanup_expired_trash	Täglich 05:00	Soft-deleted Entitäten älter als 90 Tage endgültig löschen
outbox_cleanup	Stündlich	Outbox-Einträge älter als 30 Tage löschen
cleanup_expired_sessions	Stündlich	Abgelaufene Sessions aus Redis löschen

ARQ Worker Konfiguration

# app/core/worker.py
class WorkerSettings:
    functions = [...]
    cron_jobs = [
        cron(auto_backup_job, hour=3, minute=0),
        cron(audit_retention_cleanup, hour=4, minute=0),
        cron(cleanup_expired_trash, hour=5, minute=0),
        cron(outbox_cleanup, hour={0,6,12,18}, minute=0),
        cron(cleanup_expired_sessions, hour={0,6,12,18}, minute=0),
    ]
    max_jobs = 10
    job_timeout = 300
    queue_name = "arq:queue"

Worker-Stats abfragen

# Queue-Länge
redis-cli ZCARD arq:queue

# Aktive Worker
redis-cli KEYS "arq:heartbeat:*"

# Via API (Admin)
curl -b "leocrm_session=<session>" https://crm.media-on.de/api/v1/system/dashboard | jq '.worker'

---

4. PgBouncer Setup

PgBouncer ist ein leichter Connection-Pooler für PostgreSQL. Er reduziert den Overhead neuer Datenbankverbindungen durch Wiederverwendung bestehender Verbindungen.

Warum PgBouncer?

• Connection pooling — Reduziert PostgreSQL Connection-Overhead
• Resource efficiency — Verwaltet tausende Client-Verbindungen mit minimalen Ressourcen
• Transaction pooling — Optimal für stateless Anwendungen wie FastAPI
• Session pooling — Für stateful Verbindungen
• Statement pooling — Für spezifische Use-Cases

Installation

# Debian/Ubuntu
apt-get update && apt-get install -y pgbouncer

# Verify installation
pgbouncer --version

Konfiguration

Create /etc/pgbouncer/pgbouncer.ini:

[databases]
leocrm = host=localhost port=5432 dbname=leocrm
leocrm_test = host=localhost port=5432 dbname=leocrm_test

[pgbouncer]
listen_addr = 0.0.0.0
listen_port = 6432
unix_socket_dir = /var/run/pgbouncer

# Authentication
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt

# Pool settings
pool_mode = transaction
default_pool_size = 25
max_client_conn = 200
max_db_connections = 50

# Timeouts
server_idle_timeout = 600
server_lifetime = 3600
client_idle_timeout = 1800
query_timeout = 30

# Logging
log_connections = 1
log_disconnections = 1
log_pooler_errors = 1
stats_period = 60

# Security
listen_backlog = 128

User List

Create /etc/pgbouncer/userlist.txt:

"leocrm" "md5<password_hash>"
"postgres" "md5<password_hash>"

Generate the md5 hash:

# Format: md5 + md5(password + username)
echo -n "md5" && echo -n "your_passwordleocrm" | md5sum | cut -d' ' -f1

Running PgBouncer

# Start PgBouncer
pgbouncer -d /etc/pgbouncer/pgbouncer.ini

# Check status
pgbouncer -d /etc/pgbouncer/pgbouncer.ini -R

# Reload configuration
kill -HUP $(cat /var/run/pgbouncer/pgbouncer.pid)

# Stop PgBouncer
kill -INT $(cat /var/run/pgbouncer/pgbouncer.pid)

Docker Compose Integration

Add to docker-compose.yml:

services:
  pgbouncer:
    image: bitnami/pgbouncer:latest
    container_name: leocrm-pgbouncer
    ports:
      - "6432:6432"
    environment:
      - POSTGRESQL_HOST=crm-postgres
      - POSTGRESQL_PORT=5432
      - POSTGRESQL_USERNAME=leocrm
      - POSTGRESQL_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRESQL_DATABASE=crm_db
      - PGBOUNCER_POOL_MODE=transaction
      - PGBOUNCER_DEFAULT_POOL_SIZE=25
      - PGBOUNCER_MAX_CLIENT_CONN=200
    depends_on:
      - crm-postgres
    restart: unless-stopped

Application Configuration

Update the database URL to use PgBouncer:

# Before (direct connection)
DATABASE_URL = "postgresql+asyncpg://leocrm:password@crm-postgres:5432/crm_db"

# After (via PgBouncer)
DATABASE_URL = "postgresql+asyncpg://leocrm:password@leocrm-pgbouncer:6432/crm_db"

Monitoring

# Show pool statistics
echo "SHOW STATS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer

# Show active pools
echo "SHOW POOLS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer

# Show clients
echo "SHOW CLIENTS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer

# Show servers
echo "SHOW SERVERS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer

Troubleshooting

Issue	Cause	Solution
Connection refused	PgBouncer not running	Check pgbouncer -d status
Auth failed	Wrong password in userlist	Regenerate md5 hash
Pool exhausted	Too many connections	Increase default_pool_size
Slow queries	Query timeout	Check query_timeout setting
Connection timeout	PostgreSQL overload	Check PostgreSQL connections

---

5. Audit Log Partitioning

Die audit_log Tabelle kann sehr groß werden. PostgreSQL Table Partitioning hilft durch Aufteilung in kleinere, verwaltbare Stücke.

Warum Partitioning?

• Schnellere Queries — Queries scannen nur relevante Partitionen
• Einfachere Wartung — Alte Partitionen droppen statt DELETE
• Besseres Vacuum — Jede Partition wird unabhängig gevacuumt
• Bessere Performance — Kleinere Indexes pro Partition

Partitioning-Strategie

Wir verwenden monatliches Range-Partitioning auf der created_at Spalte:

-- Jede Partition deckt einen Monat ab
-- Partitions-Name: audit_log_YYYY_MM
-- Beispiel: audit_log_2026_01, audit_log_2026_02, ...

Creating the Partitioned Table

-- Create the partitioned table
CREATE TABLE audit_log_partitioned (
    id UUID NOT NULL DEFAULT gen_random_uuid(),
    tenant_id UUID,
    user_id UUID,
    action VARCHAR(100) NOT NULL,
    entity_type VARCHAR(50),
    entity_id UUID,
    changes JSONB,
    ip_address VARCHAR(45),
    user_agent TEXT,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    PRIMARY KEY (id, created_at)
) PARTITION BY RANGE (created_at);

-- Create monthly partitions
CREATE TABLE audit_log_2026_01 PARTITION OF audit_log_partitioned
    FOR VALUES FROM ('2026-01-01') TO ('2026-02-01');

CREATE TABLE audit_log_2026_02 PARTITION OF audit_log_partitioned
    FOR VALUES FROM ('2026-02-01') TO ('2026-03-01');

CREATE TABLE audit_log_2026_03 PARTITION OF audit_log_partitioned
    FOR VALUES FROM ('2026-03-01') TO ('2026-04-01');

-- Add indexes on each partition
CREATE INDEX idx_audit_log_2026_01_tenant ON audit_log_2026_01 (tenant_id);
CREATE INDEX idx_audit_log_2026_01_action ON audit_log_2026_01 (action);
CREATE INDEX idx_audit_log_2026_01_entity ON audit_log_2026_01 (entity_type, entity_id);
CREATE INDEX idx_audit_log_2026_01_created ON audit_log_2026_01 (created_at DESC);

Automating Partition Creation

Use the setup_audit_partitioning.sql script to automate partition management:

# Run the setup script
psql -h localhost -U leocrm -d crm_db -f scripts/setup_audit_partitioning.sql

Cron Job for Partition Maintenance

# Run on the 1st of each month at 2 AM
0 2 1 * * /usr/bin/psql -h localhost -U leocrm -d crm_db -c "SELECT create_monthly_audit_partition();"

Dropping Old Partitions

-- Drop partitions older than retention period
DROP TABLE IF EXISTS audit_log_2025_01;
DROP TABLE IF EXISTS audit_log_2025_02;

Monitoring Partition Health

-- Check partition sizes
SELECT
    relname AS partition_name,
    pg_size_pretty(pg_total_relation_size(relid)) AS total_size
FROM pg_catalog.pg_statio_user_tables
WHERE relname LIKE 'audit_log_%'
ORDER BY relname;

-- Check row counts per partition
SELECT
    relname AS partition_name,
    n_live_tup AS row_count
FROM pg_catalog.pg_stat_user_tables
WHERE relname LIKE 'audit_log_%'
ORDER BY relname;

---

6. Backup and Recovery

Database Backup

# Full backup
pg_dump -h localhost -U leocrm -d crm_db -F c -f /backups/crm_db_$(date +%Y%m%d).dump

# Backup with compression
pg_dump -h localhost -U leocrm -d crm_db -F c -Z 9 -f /backups/crm_db_$(date +%Y%m%d).dump.gz

Automated Backup

LeoCRM hat einen automatisierten Backup via ARQ Cron-Job:

• Job: auto_backup_job (täglich 03:00 Uhr)
• Script: scripts/backup.py (pg_dump + files)
• Konfiguration: Settings → Backup (backup_enabled, backup_interval, backup_retention_days, backup_destination)
• API: POST /api/v1/system-settings/backup-now (manueller Trigger)
• History: GET /api/v1/system-settings/backup-history (letzte 10 Backups)
• Bei Fehler: System-Message an Communication-System

Database Restore

# Restore full backup
pg_restore -h localhost -U leocrm -d crm_db -c /backups/crm_db_20260701.dump

# Restore with parallel workers (faster)
pg_restore -h localhost -U leocrm -d crm_db -j 4 -c /backups/crm_db_20260701.dump

See scripts/restore.py for the automated restore solution.

---

7. Monitoring and Alerts

Key Metrics

Metric	Target	Alert Threshold
Database connections	< 50	> 80% of max
Query response time	< 100ms	> 500ms
Cache hit ratio	> 95%	< 90%
Partition size	< 10GB	> 50GB
PgBouncer pool usage	< 80%	> 90%
Worker queue length	< 100	> 500
Error rate	< 1%	> 5%

Health Checks

# Check PgBouncer status
echo "SHOW STATS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer | grep -E "total_|avg_"

# Check partition health
psql -h localhost -U leocrm -d crm_db -c "SELECT count(*) FROM audit_log WHERE created_at < NOW() - INTERVAL '3 months';"

# Check database size
psql -h localhost -U leocrm -d crm_db -c "SELECT pg_size_pretty(pg_database_size('crm_db'));"

# Via API (Admin)
curl -b "leocrm_session=<session>" https://crm.media-on.de/api/v1/system/dashboard | jq .

---

8. Production Resource Recommendations

Die docker-compose.yaml hat Development-Defaults. Für Produktion mit 50+ Usern müssen die Limits erhöht werden.

Service	Development	Production (50+ User)	Begründung
PostgreSQL RAM	512m	1-2GB	pgvector HNSW + FTS + JSONB Snapshots + Outbox + AuditLog
PostgreSQL CPU	1.0	2.0	Vector Search + FTS + normale CRM-Queries
PostgreSQL Disk	Named Volume	50-100GB	Embeddings (768 dim × 100k = ~300MB), JSONB Snapshots, Outbox
Redis RAM	128m	256-512m	WS Pub/Sub + Caching + Sessions + ARQ + Rate-Limiting
Redis CPU	0.5	1.0	Pub/Sub + Cache + Queue
App (FastAPI) RAM	Nicht limitiert	512m-1GB	WebSocket Connections + Async Tasks
App CPU	Nicht limitiert	1-2 CPUs	API + WS + LLM-Streaming
Worker (ARQ) RAM	Nicht limitiert	256-512m	Background Jobs (Indexierung, Agent-Runs, Extraction)
Worker CPU	Nicht limitiert	1-2 CPUs	LLM-Calls + Embedding + Text-Extraction

Skalierung bei Bedarf

• Read-Replicas: Bei hohem Lese-Aufkommen (Search, FTS, Vector) können Read-Replicas für PostgreSQL eingerichtet werden.
• Mehr Worker: Bei hohem Background-Job-Aufkommen können zusätzliche ARQ-Worker-Container gestartet werden.
• pgvector auslagern: Bei sehr großen Datasets (>1M Embeddings) kann pgvector auf einen separaten PostgreSQL-Node ausgelagert werden.
• Redis Cluster: Bei sehr hohem Cache-/Pub/Sub-Aufkommen kann Redis Cluster eingesetzt werden.