leocrm/docs/infrastructure.md

# Infrastructure Guide

> **Version:** 2.0  
> **Date:** 2026-08-20  
> **Applies to:** System administrators and DevOps engineers

---

## 1. Docker-Compose Stack

LeoCRM läuft als Docker-Compose-Stack mit 4 Services:

| Service | Image | Beschreibung |
|---------|-------|-------------|
| **postgres** | `pgvector/pgvector:pg16` | PostgreSQL 16 mit pgvector Extension für Vector Search |
| **redis** | `redis:7-alpine` | Redis 7 für Sessions, Caching, Pub/Sub, ARQ Queue |
| **crm_app** | Multi-Stage Build | FastAPI Backend (uvicorn), API + WebSocket |
| **crm_worker** | Multi-Stage Build | ARQ Background Worker (Cron-Jobs, Queue Processing) |

### docker-compose.yaml Übersicht

```yaml
services:
  postgres:
    image: pgvector/pgvector:pg16
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U crm_user -d crm_db"]
      interval: 10s
      timeout: 5s
      retries: 5

  redis:
    image: redis:7-alpine
    volumes:
      - redis_data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5

  crm_app:
    build: .
    command: ["./prestart.sh"]
    depends_on:
      postgres: { condition: service_healthy }
      redis: { condition: service_healthy }
    healthcheck:
      test: ["CMD-SHELL", "bash /app/healthcheck.sh"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 15s

  crm_worker:
    build: .
    command: ["./worker.sh"]
    depends_on:
      postgres: { condition: service_healthy }
      redis: { condition: service_healthy }
```

### Container-Entrypoints

| Datei | Service | Funktion |
|------|---------|----------|
| `prestart.sh` | crm_app | Alembic-Migrationen → DB-Role-Passwörter → Plugin-Schema-Sync → Admin-Seed → uvicorn Start |
| `worker.sh` | crm_worker | ARQ Worker Start mit Cron-Jobs |
| `healthcheck.sh` | crm_app | HTTP `/api/v1/health` oder Redis-Ping |

### Volumes

| Volume | Service | Beschreibung |
|--------|---------|-------------|
| `postgres_data` | postgres | PostgreSQL Daten (persistent) |
| `redis_data` | redis | Redis Snapshot (persistent) |
| `app_storage` | crm_app | File Storage (DMS, Uploads) |

---

## 2. Coolify Deployment

LeoCRM ist über Coolify auf einem Hetzner VPS deployiert.

### Server-Info

| Eigenschaft | Wert |
|-------------|------|
| **Host** | 46.225.91.159 |
| **Hostname** | coolify-01 |
| **Provider** | Hetzner VPS |
| **Coolify URL** | https://server.media-on.de |
| **App URL** | https://crm.media-on.de |
| **App UUID** | xf7smknlger3hvkrsb910tui |
| **Container-Name** | Ändert sich bei jedem Coolify-Deploy (Suffix) |

### Deploy-Methoden

#### Frontend-Only Deploy (~20s)
```bash
bash /a0/usr/projects/leocrm/scripts/fast-deploy.sh frontend
```
- Baut Frontend lokal, kopiert `dist/` direkt in den laufenden Container
- Kein Coolify-Rebuild, kein Docker-Image-Neubau
- Container wird nicht neu gestartet

#### Full Deploy (~2min, für Backend-Änderungen)
```bash
bash /a0/usr/projects/leocrm/scripts/fast-deploy.sh full
```
- Triggert Coolify-Rebuild über `deploy.py`
- Für Python-Code, Requirements, Migrations

### Server-Container

Auf dem Hetzner VPS laufen ~25 Docker-Container (Coolify + Services):
- Coolify Proxy (Traefik)
- Coolify Dashboard
- Coolify Database
- LeoCRM Stack (postgres, redis, crm_app, crm_worker)
- Weitere Coolify-managed Services

Container-Übersicht auf dem Server:
```bash
ssh -i ~/.ssh/coolify-01-root root@46.225.91.159 'docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"'
```

---

## 3. ARQ Worker & Cron-Jobs

Der `crm_worker` Service läuft ARQ (Async Redis Queue) für Background-Jobs.

### Registrierte Cron-Jobs

| Job | Schedule | Beschreibung |
|-----|----------|-------------|
| `auto_backup_job` | Täglich 03:00 | Automatisches Backup (ruft `scripts/backup.py` auf) |
| `audit_retention_cleanup` | Täglich 04:00 | Audit-Logs älter als 365 Tage archivieren/löschen |
| `cleanup_expired_trash` | Täglich 05:00 | Soft-deleted Entitäten älter als 90 Tage endgültig löschen |
| `outbox_cleanup` | Stündlich | Outbox-Einträge älter als 30 Tage löschen |
| `cleanup_expired_sessions` | Stündlich | Abgelaufene Sessions aus Redis löschen |

### ARQ Worker Konfiguration

```python
# app/core/worker.py
class WorkerSettings:
    functions = [...]
    cron_jobs = [
        cron(auto_backup_job, hour=3, minute=0),
        cron(audit_retention_cleanup, hour=4, minute=0),
        cron(cleanup_expired_trash, hour=5, minute=0),
        cron(outbox_cleanup, hour={0,6,12,18}, minute=0),
        cron(cleanup_expired_sessions, hour={0,6,12,18}, minute=0),
    ]
    max_jobs = 10
    job_timeout = 300
    queue_name = "arq:queue"
```

### Worker-Stats abfragen

```bash
# Queue-Länge
redis-cli ZCARD arq:queue

# Aktive Worker
redis-cli KEYS "arq:heartbeat:*"

# Via API (Admin)
curl -b "leocrm_session=<session>" https://crm.media-on.de/api/v1/system/dashboard | jq '.worker'
```

---

## 4. PgBouncer Setup

PgBouncer ist ein leichter Connection-Pooler für PostgreSQL. Er reduziert den Overhead neuer Datenbankverbindungen durch Wiederverwendung bestehender Verbindungen.

### Warum PgBouncer?

- **Connection pooling** — Reduziert PostgreSQL Connection-Overhead
- **Resource efficiency** — Verwaltet tausende Client-Verbindungen mit minimalen Ressourcen
- **Transaction pooling** — Optimal für stateless Anwendungen wie FastAPI
- **Session pooling** — Für stateful Verbindungen
- **Statement pooling** — Für spezifische Use-Cases

### Installation

```bash
# Debian/Ubuntu
apt-get update && apt-get install -y pgbouncer

# Verify installation
pgbouncer --version
```

### Konfiguration

Create `/etc/pgbouncer/pgbouncer.ini`:

```ini
[databases]
leocrm = host=localhost port=5432 dbname=leocrm
leocrm_test = host=localhost port=5432 dbname=leocrm_test

[pgbouncer]
listen_addr = 0.0.0.0
listen_port = 6432
unix_socket_dir = /var/run/pgbouncer

# Authentication
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt

# Pool settings
pool_mode = transaction
default_pool_size = 25
max_client_conn = 200
max_db_connections = 50

# Timeouts
server_idle_timeout = 600
server_lifetime = 3600
client_idle_timeout = 1800
query_timeout = 30

# Logging
log_connections = 1
log_disconnections = 1
log_pooler_errors = 1
stats_period = 60

# Security
listen_backlog = 128
```

### User List

Create `/etc/pgbouncer/userlist.txt`:

```
"leocrm" "md5<password_hash>"
"postgres" "md5<password_hash>"
```

Generate the md5 hash:
```bash
# Format: md5 + md5(password + username)
echo -n "md5" && echo -n "your_passwordleocrm" | md5sum | cut -d' ' -f1
```

### Running PgBouncer

```bash
# Start PgBouncer
pgbouncer -d /etc/pgbouncer/pgbouncer.ini

# Check status
pgbouncer -d /etc/pgbouncer/pgbouncer.ini -R

# Reload configuration
kill -HUP $(cat /var/run/pgbouncer/pgbouncer.pid)

# Stop PgBouncer
kill -INT $(cat /var/run/pgbouncer/pgbouncer.pid)
```

### Docker Compose Integration

Add to `docker-compose.yml`:

```yaml
services:
  pgbouncer:
    image: bitnami/pgbouncer:latest
    container_name: leocrm-pgbouncer
    ports:
      - "6432:6432"
    environment:
      - POSTGRESQL_HOST=crm-postgres
      - POSTGRESQL_PORT=5432
      - POSTGRESQL_USERNAME=leocrm
      - POSTGRESQL_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRESQL_DATABASE=crm_db
      - PGBOUNCER_POOL_MODE=transaction
      - PGBOUNCER_DEFAULT_POOL_SIZE=25
      - PGBOUNCER_MAX_CLIENT_CONN=200
    depends_on:
      - crm-postgres
    restart: unless-stopped
```

### Application Configuration

Update the database URL to use PgBouncer:

```python
# Before (direct connection)
DATABASE_URL = "postgresql+asyncpg://leocrm:password@crm-postgres:5432/crm_db"

# After (via PgBouncer)
DATABASE_URL = "postgresql+asyncpg://leocrm:password@leocrm-pgbouncer:6432/crm_db"
```

### Monitoring

```bash
# Show pool statistics
echo "SHOW STATS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer

# Show active pools
echo "SHOW POOLS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer

# Show clients
echo "SHOW CLIENTS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer

# Show servers
echo "SHOW SERVERS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer
```

### Troubleshooting

| Issue | Cause | Solution |
|-------|-------|----------|
| Connection refused | PgBouncer not running | Check `pgbouncer -d` status |
| Auth failed | Wrong password in userlist | Regenerate md5 hash |
| Pool exhausted | Too many connections | Increase `default_pool_size` |
| Slow queries | Query timeout | Check `query_timeout` setting |
| Connection timeout | PostgreSQL overload | Check PostgreSQL connections |

---

## 5. Audit Log Partitioning

Die `audit_log` Tabelle kann sehr groß werden. PostgreSQL Table Partitioning hilft durch Aufteilung in kleinere, verwaltbare Stücke.

### Warum Partitioning?

- **Schnellere Queries** — Queries scannen nur relevante Partitionen
- **Einfachere Wartung** — Alte Partitionen droppen statt DELETE
- **Besseres Vacuum** — Jede Partition wird unabhängig gevacuumt
- **Bessere Performance** — Kleinere Indexes pro Partition

### Partitioning-Strategie

Wir verwenden **monatliches Range-Partitioning** auf der `created_at` Spalte:

```sql
-- Jede Partition deckt einen Monat ab
-- Partitions-Name: audit_log_YYYY_MM
-- Beispiel: audit_log_2026_01, audit_log_2026_02, ...
```

### Creating the Partitioned Table

```sql
-- Create the partitioned table
CREATE TABLE audit_log_partitioned (
    id UUID NOT NULL DEFAULT gen_random_uuid(),
    tenant_id UUID,
    user_id UUID,
    action VARCHAR(100) NOT NULL,
    entity_type VARCHAR(50),
    entity_id UUID,
    changes JSONB,
    ip_address VARCHAR(45),
    user_agent TEXT,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    PRIMARY KEY (id, created_at)
) PARTITION BY RANGE (created_at);

-- Create monthly partitions
CREATE TABLE audit_log_2026_01 PARTITION OF audit_log_partitioned
    FOR VALUES FROM ('2026-01-01') TO ('2026-02-01');

CREATE TABLE audit_log_2026_02 PARTITION OF audit_log_partitioned
    FOR VALUES FROM ('2026-02-01') TO ('2026-03-01');

CREATE TABLE audit_log_2026_03 PARTITION OF audit_log_partitioned
    FOR VALUES FROM ('2026-03-01') TO ('2026-04-01');

-- Add indexes on each partition
CREATE INDEX idx_audit_log_2026_01_tenant ON audit_log_2026_01 (tenant_id);
CREATE INDEX idx_audit_log_2026_01_action ON audit_log_2026_01 (action);
CREATE INDEX idx_audit_log_2026_01_entity ON audit_log_2026_01 (entity_type, entity_id);
CREATE INDEX idx_audit_log_2026_01_created ON audit_log_2026_01 (created_at DESC);
```

### Automating Partition Creation

Use the `setup_audit_partitioning.sql` script to automate partition management:

```bash
# Run the setup script
psql -h localhost -U leocrm -d crm_db -f scripts/setup_audit_partitioning.sql
```

### Cron Job for Partition Maintenance

```bash
# Run on the 1st of each month at 2 AM
0 2 1 * * /usr/bin/psql -h localhost -U leocrm -d crm_db -c "SELECT create_monthly_audit_partition();"
```

### Dropping Old Partitions

```sql
-- Drop partitions older than retention period
DROP TABLE IF EXISTS audit_log_2025_01;
DROP TABLE IF EXISTS audit_log_2025_02;
```

### Monitoring Partition Health

```sql
-- Check partition sizes
SELECT
    relname AS partition_name,
    pg_size_pretty(pg_total_relation_size(relid)) AS total_size
FROM pg_catalog.pg_statio_user_tables
WHERE relname LIKE 'audit_log_%'
ORDER BY relname;

-- Check row counts per partition
SELECT
    relname AS partition_name,
    n_live_tup AS row_count
FROM pg_catalog.pg_stat_user_tables
WHERE relname LIKE 'audit_log_%'
ORDER BY relname;
```

---

## 6. Backup and Recovery

### Database Backup

```bash
# Full backup
pg_dump -h localhost -U leocrm -d crm_db -F c -f /backups/crm_db_$(date +%Y%m%d).dump

# Backup with compression
pg_dump -h localhost -U leocrm -d crm_db -F c -Z 9 -f /backups/crm_db_$(date +%Y%m%d).dump.gz
```

### Automated Backup

LeoCRM hat einen automatisierten Backup via ARQ Cron-Job:

- **Job:** `auto_backup_job` (täglich 03:00 Uhr)
- **Script:** `scripts/backup.py` (pg_dump + files)
- **Konfiguration:** Settings → Backup (backup_enabled, backup_interval, backup_retention_days, backup_destination)
- **API:** `POST /api/v1/system-settings/backup-now` (manueller Trigger)
- **History:** `GET /api/v1/system-settings/backup-history` (letzte 10 Backups)
- **Bei Fehler:** System-Message an Communication-System

### Database Restore

```bash
# Restore full backup
pg_restore -h localhost -U leocrm -d crm_db -c /backups/crm_db_20260701.dump

# Restore with parallel workers (faster)
pg_restore -h localhost -U leocrm -d crm_db -j 4 -c /backups/crm_db_20260701.dump
```

See `scripts/restore.py` for the automated restore solution.

---

## 7. Monitoring and Alerts

### Key Metrics

| Metric | Target | Alert Threshold |
|--------|--------|----------------|
| Database connections | < 50 | > 80% of max |
| Query response time | < 100ms | > 500ms |
| Cache hit ratio | > 95% | < 90% |
| Partition size | < 10GB | > 50GB |
| PgBouncer pool usage | < 80% | > 90% |
| Worker queue length | < 100 | > 500 |
| Error rate | < 1% | > 5% |

### Health Checks

```bash
# Check PgBouncer status
echo "SHOW STATS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer | grep -E "total_|avg_"

# Check partition health
psql -h localhost -U leocrm -d crm_db -c "SELECT count(*) FROM audit_log WHERE created_at < NOW() - INTERVAL '3 months';"

# Check database size
psql -h localhost -U leocrm -d crm_db -c "SELECT pg_size_pretty(pg_database_size('crm_db'));"

# Via API (Admin)
curl -b "leocrm_session=<session>" https://crm.media-on.de/api/v1/system/dashboard | jq .
```

---

## 8. Production Resource Recommendations

Die `docker-compose.yaml` hat Development-Defaults. Für Produktion mit 50+ Usern müssen die Limits erhöht werden.

| Service | Development | Production (50+ User) | Begründung |
|---|---|---|---|
| **PostgreSQL RAM** | 512m | 1-2GB | pgvector HNSW + FTS + JSONB Snapshots + Outbox + AuditLog |
| **PostgreSQL CPU** | 1.0 | 2.0 | Vector Search + FTS + normale CRM-Queries |
| **PostgreSQL Disk** | Named Volume | 50-100GB | Embeddings (768 dim × 100k = ~300MB), JSONB Snapshots, Outbox |
| **Redis RAM** | 128m | 256-512m | WS Pub/Sub + Caching + Sessions + ARQ + Rate-Limiting |
| **Redis CPU** | 0.5 | 1.0 | Pub/Sub + Cache + Queue |
| **App (FastAPI) RAM** | Nicht limitiert | 512m-1GB | WebSocket Connections + Async Tasks |
| **App CPU** | Nicht limitiert | 1-2 CPUs | API + WS + LLM-Streaming |
| **Worker (ARQ) RAM** | Nicht limitiert | 256-512m | Background Jobs (Indexierung, Agent-Runs, Extraction) |
| **Worker CPU** | Nicht limitiert | 1-2 CPUs | LLM-Calls + Embedding + Text-Extraction |

### Skalierung bei Bedarf

- **Read-Replicas:** Bei hohem Lese-Aufkommen (Search, FTS, Vector) können Read-Replicas für PostgreSQL eingerichtet werden.
- **Mehr Worker:** Bei hohem Background-Job-Aufkommen können zusätzliche ARQ-Worker-Container gestartet werden.
- **pgvector auslagern:** Bei sehr großen Datasets (>1M Embeddings) kann pgvector auf einen separaten PostgreSQL-Node ausgelagert werden.
- **Redis Cluster:** Bei sehr hohem Cache-/Pub/Sub-Aufkommen kann Redis Cluster eingesetzt werden.