Leopoldadmin/leocrm
1
0
Fork 0
Code Issues 363 Pull Requests Actions 1 Packages Projects Releases 1 Wiki Activity

docs: punkt 11 (documentation) — README, infrastructure, monitoring, admin-guide, deploy-guide, api-docs, PROGRESS, ENTERPRISE_READINESS_PLAN all updated

Browse Source
This commit is contained in:
Agent Zero
2026-08-20 14:03:35 +02:00
parent e6790d9b81
commit f79eb9354a
8 changed files with 853 additions and 242 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/infrastructure.md
+237 -116
View File
@@ -1,22 +1,192 @@
# Infrastructure Guide
> **Version:** 1.0
> **Date:** 2026-07-29
> **Version:** 2.0
> **Date:** 2026-08-20
> **Applies to:** System administrators and DevOps engineers
---
## 1. PgBouncer Setup
## 1. Docker-Compose Stack
PgBouncer is a lightweight connection pooler for PostgreSQL. It reduces the overhead of establishing new database connections by reusing existing ones.
LeoCRM läuft als Docker-Compose-Stack mit 4 Services:
### Why PgBouncer?
| Service | Image | Beschreibung |
|---------|-------|-------------|
| **postgres** | `pgvector/pgvector:pg16` | PostgreSQL 16 mit pgvector Extension für Vector Search |
| **redis** | `redis:7-alpine` | Redis 7 für Sessions, Caching, Pub/Sub, ARQ Queue |
| **crm_app** | Multi-Stage Build | FastAPI Backend (uvicorn), API + WebSocket |
| **crm_worker** | Multi-Stage Build | ARQ Background Worker (Cron-Jobs, Queue Processing) |
- **Connection pooling** — Reduces PostgreSQL connection overhead
- **Resource efficiency** — Handles thousands of client connections with minimal resources
- **Transaction pooling** — Best for stateless applications like FastAPI
- **Session pooling** — For stateful connections
- **Statement pooling** — For specific use cases
### docker-compose.yaml Übersicht
```yaml
services:
postgres:
image: pgvector/pgvector:pg16
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U crm_user -d crm_db"]
interval: 10s
timeout: 5s
retries: 5
redis:
image: redis:7-alpine
volumes:
- redis_data:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 10s
timeout: 5s
retries: 5
crm_app:
build: .
command: ["./prestart.sh"]
depends_on:
postgres: { condition: service_healthy }
redis: { condition: service_healthy }
healthcheck:
test: ["CMD-SHELL", "bash /app/healthcheck.sh"]
interval: 30s
timeout: 10s
retries: 3
start_period: 15s
crm_worker:
build: .
command: ["./worker.sh"]
depends_on:
postgres: { condition: service_healthy }
redis: { condition: service_healthy }
```
### Container-Entrypoints
| Datei | Service | Funktion |
|------|---------|----------|
| `prestart.sh` | crm_app | Alembic-Migrationen → DB-Role-Passwörter → Plugin-Schema-Sync → Admin-Seed → uvicorn Start |
| `worker.sh` | crm_worker | ARQ Worker Start mit Cron-Jobs |
| `healthcheck.sh` | crm_app | HTTP `/api/v1/health` oder Redis-Ping |
### Volumes
| Volume | Service | Beschreibung |
|--------|---------|-------------|
| `postgres_data` | postgres | PostgreSQL Daten (persistent) |
| `redis_data` | redis | Redis Snapshot (persistent) |
| `app_storage` | crm_app | File Storage (DMS, Uploads) |
---
## 2. Coolify Deployment
LeoCRM ist über Coolify auf einem Hetzner VPS deployiert.
### Server-Info
| Eigenschaft | Wert |
|-------------|------|
| **Host** | 46.225.91.159 |
| **Hostname** | coolify-01 |
| **Provider** | Hetzner VPS |
| **Coolify URL** | https://server.media-on.de |
| **App URL** | https://crm.media-on.de |
| **App UUID** | xf7smknlger3hvkrsb910tui |
| **Container-Name** | Ändert sich bei jedem Coolify-Deploy (Suffix) |
### Deploy-Methoden
#### Frontend-Only Deploy (~20s)
```bash
bash /a0/usr/projects/leocrm/scripts/fast-deploy.sh frontend
```
- Baut Frontend lokal, kopiert `dist/` direkt in den laufenden Container
- Kein Coolify-Rebuild, kein Docker-Image-Neubau
- Container wird nicht neu gestartet
#### Full Deploy (~2min, für Backend-Änderungen)
```bash
bash /a0/usr/projects/leocrm/scripts/fast-deploy.sh full
```
- Triggert Coolify-Rebuild über `deploy.py`
- Für Python-Code, Requirements, Migrations
### Server-Container
Auf dem Hetzner VPS laufen ~25 Docker-Container (Coolify + Services):
- Coolify Proxy (Traefik)
- Coolify Dashboard
- Coolify Database
- LeoCRM Stack (postgres, redis, crm_app, crm_worker)
- Weitere Coolify-managed Services
Container-Übersicht auf dem Server:
```bash
ssh -i ~/.ssh/coolify-01-root root@46.225.91.159 'docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"'
```
---
## 3. ARQ Worker & Cron-Jobs
Der `crm_worker` Service läuft ARQ (Async Redis Queue) für Background-Jobs.
### Registrierte Cron-Jobs
| Job | Schedule | Beschreibung |
|-----|----------|-------------|
| `auto_backup_job` | Täglich 03:00 | Automatisches Backup (ruft `scripts/backup.py` auf) |
| `audit_retention_cleanup` | Täglich 04:00 | Audit-Logs älter als 365 Tage archivieren/löschen |
| `cleanup_expired_trash` | Täglich 05:00 | Soft-deleted Entitäten älter als 90 Tage endgültig löschen |
| `outbox_cleanup` | Stündlich | Outbox-Einträge älter als 30 Tage löschen |
| `cleanup_expired_sessions` | Stündlich | Abgelaufene Sessions aus Redis löschen |
### ARQ Worker Konfiguration
```python
# app/core/worker.py
class WorkerSettings:
functions = [...]
cron_jobs = [
cron(auto_backup_job, hour=3, minute=0),
cron(audit_retention_cleanup, hour=4, minute=0),
cron(cleanup_expired_trash, hour=5, minute=0),
cron(outbox_cleanup, hour={0,6,12,18}, minute=0),
cron(cleanup_expired_sessions, hour={0,6,12,18}, minute=0),
]
max_jobs = 10
job_timeout = 300
queue_name = "arq:queue"
```
### Worker-Stats abfragen
```bash
# Queue-Länge
redis-cli ZCARD arq:queue
# Aktive Worker
redis-cli KEYS "arq:heartbeat:*"
# Via API (Admin)
curl -b "leocrm_session=<session>" https://crm.media-on.de/api/v1/system/dashboard | jq '.worker'
```
---
## 4. PgBouncer Setup
PgBouncer ist ein leichter Connection-Pooler für PostgreSQL. Er reduziert den Overhead neuer Datenbankverbindungen durch Wiederverwendung bestehender Verbindungen.
### Warum PgBouncer?
- **Connection pooling** — Reduziert PostgreSQL Connection-Overhead
- **Resource efficiency** — Verwaltet tausende Client-Verbindungen mit minimalen Ressourcen
- **Transaction pooling** — Optimal für stateless Anwendungen wie FastAPI
- **Session pooling** — Für stateful Verbindungen
- **Statement pooling** — Für spezifische Use-Cases
### Installation
@@ -28,7 +198,7 @@ apt-get update && apt-get install -y pgbouncer
pgbouncer --version
```
### Configuration
### Konfiguration
Create `/etc/pgbouncer/pgbouncer.ini`:
@@ -43,8 +213,6 @@ listen_port = 6432
unix_socket_dir = /var/run/pgbouncer
# Authentication
# Use md5 for password-based auth
# Use trust for local development
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt
@@ -67,7 +235,6 @@ log_pooler_errors = 1
stats_period = 60
# Security
# Only allow connections from localhost and Docker network
listen_backlog = 128
```
@@ -167,25 +334,25 @@ echo "SHOW SERVERS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer
---
## 2. Audit Log Partitioning
## 5. Audit Log Partitioning
The `audit_log` table can grow very large over time. PostgreSQL table partitioning helps manage this by splitting the table into smaller, more manageable pieces.
Die `audit_log` Tabelle kann sehr groß werden. PostgreSQL Table Partitioning hilft durch Aufteilung in kleinere, verwaltbare Stücke.
### Why Partition?
### Warum Partitioning?
- **Faster queries** — Queries only scan relevant partitions
- **Easier maintenance** — Drop old partitions instead of DELETE
- **Better vacuum** — Each partition is vacuumed independently
- **Improved performance** — Smaller indexes per partition
- **Schnellere Queries** — Queries scannen nur relevante Partitionen
- **Einfachere Wartung** — Alte Partitionen droppen statt DELETE
- **Besseres Vacuum** — Jede Partition wird unabhängig gevacuumt
- **Bessere Performance** — Kleinere Indexes pro Partition
### Partitioning Strategy
### Partitioning-Strategie
We use **monthly range partitioning** on the `created_at` column:
Wir verwenden **monatliches Range-Partitioning** auf der `created_at` Spalte:
```sql
-- Each partition covers one month
-- Partition name: audit_log_YYYY_MM
-- Example: audit_log_2026_01, audit_log_2026_02, ...
-- Jede Partition deckt einen Monat ab
-- Partitions-Name: audit_log_YYYY_MM
-- Beispiel: audit_log_2026_01, audit_log_2026_02, ...
```
### Creating the Partitioned Table
@@ -221,46 +388,6 @@ CREATE INDEX idx_audit_log_2026_01_tenant ON audit_log_2026_01 (tenant_id);
CREATE INDEX idx_audit_log_2026_01_action ON audit_log_2026_01 (action);
CREATE INDEX idx_audit_log_2026_01_entity ON audit_log_2026_01 (entity_type, entity_id);
CREATE INDEX idx_audit_log_2026_01_created ON audit_log_2026_01 (created_at DESC);
CREATE INDEX idx_audit_log_2026_02_tenant ON audit_log_2026_02 (tenant_id);
CREATE INDEX idx_audit_log_2026_02_action ON audit_log_2026_02 (action);
CREATE INDEX idx_audit_log_2026_02_entity ON audit_log_2026_02 (entity_type, entity_id);
CREATE INDEX idx_audit_log_2026_02_created ON audit_log_2026_02 (created_at DESC);
CREATE INDEX idx_audit_log_2026_03_tenant ON audit_log_2026_03 (tenant_id);
CREATE INDEX idx_audit_log_2026_03_action ON audit_log_2026_03 (action);
CREATE INDEX idx_audit_log_2026_03_entity ON audit_log_2026_03 (entity_type, entity_id);
CREATE INDEX idx_audit_log_2026_03_created ON audit_log_2026_03 (created_at DESC);
```
### Migrating Existing Data
```sql
-- Step 1: Create the partitioned table
-- (see script above)
-- Step 2: Insert existing data
INSERT INTO audit_log_partitioned (
id, tenant_id, user_id, action, entity_type,
entity_id, changes, ip_address, user_agent, created_at
)
SELECT id, tenant_id, user_id, action, entity_type,
entity_id, changes, ip_address, user_agent, created_at
FROM audit_log;
-- Step 3: Verify data integrity
SELECT COUNT(*) FROM audit_log_partitioned;
SELECT COUNT(*) FROM audit_log;
-- Step 4: Rename tables
ALTER TABLE audit_log RENAME TO audit_log_old;
ALTER TABLE audit_log_partitioned RENAME TO audit_log;
-- Step 5: Update sequences and indexes
-- (handled by the partitioned table definition)
-- Step 6: Drop old table after verification
-- DROP TABLE audit_log_old;
```
### Automating Partition Creation
@@ -274,54 +401,19 @@ psql -h localhost -U leocrm -d crm_db -f scripts/setup_audit_partitioning.sql
### Cron Job for Partition Maintenance
Add to crontab to create partitions automatically:
```bash
# Run on the 1st of each month at 2 AM
0 2 1 * * /usr/bin/psql -h localhost -U leocrm -d crm_db -c "SELECT create_monthly_audit_partition();"
```
### Querying Partitioned Data
```sql
-- Query a specific month (fast, only scans one partition)
SELECT * FROM audit_log
WHERE created_at >= '2026-01-01'
AND created_at < '2026-02-01'
AND tenant_id = '...';
-- Query across months (scans multiple partitions)
SELECT * FROM audit_log
WHERE created_at >= '2026-01-01'
AND created_at < '2026-03-01'
AND action = 'permission_grant';
-- Check which partitions will be scanned
EXPLAIN SELECT * FROM audit_log
WHERE created_at >= '2026-01-01'
AND created_at < '2026-02-01';
```
### Dropping Old Partitions
```sql
-- Drop partitions older than retention period
DROP TABLE IF EXISTS audit_log_2025_01;
DROP TABLE IF EXISTS audit_log_2025_02;
-- ...
-- Or use a function
SELECT drop_old_audit_partitions(12); -- Keep last 12 months
```
### Performance Considerations
- **Index each partition** — Don't rely on parent table indexes
- **Use `created_at` in WHERE** — Always filter by date for partition pruning
- **Monitor partition count** — Too many partitions can slow planning
- **Archive old partitions** — Consider moving to cheaper storage
- **Vacuum partitions** — Each partition needs independent vacuum
### Monitoring Partition Health
```sql
@@ -340,18 +432,11 @@ SELECT
FROM pg_catalog.pg_stat_user_tables
WHERE relname LIKE 'audit_log_%'
ORDER BY relname;
-- List all partitions
SELECT
inhrelid::regclass AS partition_name
FROM pg_catalog.pg_inherits
WHERE inhparent = 'audit_log'::regclass
ORDER BY partition_name;
```
---
## 3. Backup and Recovery
## 6. Backup and Recovery
### Database Backup
@@ -361,11 +446,19 @@ pg_dump -h localhost -U leocrm -d crm_db -F c -f /backups/crm_db_$(date +%Y%m%d)
# Backup with compression
pg_dump -h localhost -U leocrm -d crm_db -F c -Z 9 -f /backups/crm_db_$(date +%Y%m%d).dump.gz
# Backup specific schema only
pg_dump -h localhost -U leocrm -d crm_db -n public -F c -f /backups/crm_db_schema_$(date +%Y%m%d).dump
```
### Automated Backup
LeoCRM hat einen automatisierten Backup via ARQ Cron-Job:
- **Job:** `auto_backup_job` (täglich 03:00 Uhr)
- **Script:** `scripts/backup.py` (pg_dump + files)
- **Konfiguration:** Settings → Backup (backup_enabled, backup_interval, backup_retention_days, backup_destination)
- **API:** `POST /api/v1/system-settings/backup-now` (manueller Trigger)
- **History:** `GET /api/v1/system-settings/backup-history` (letzte 10 Backups)
- **Bei Fehler:** System-Message an Communication-System
### Database Restore
```bash
@@ -376,13 +469,11 @@ pg_restore -h localhost -U leocrm -d crm_db -c /backups/crm_db_20260701.dump
pg_restore -h localhost -U leocrm -d crm_db -j 4 -c /backups/crm_db_20260701.dump
```
### Automated Backup Script
See `scripts/backup.py` for the automated backup solution.
See `scripts/restore.py` for the automated restore solution.
---
## 4. Monitoring and Alerts
## 7. Monitoring and Alerts
### Key Metrics
@@ -393,6 +484,8 @@ See `scripts/backup.py` for the automated backup solution.
| Cache hit ratio | > 95% | < 90% |
| Partition size | < 10GB | > 50GB |
| PgBouncer pool usage | < 80% | > 90% |
| Worker queue length | < 100 | > 500 |
| Error rate | < 1% | > 5% |
### Health Checks
@@ -405,4 +498,32 @@ psql -h localhost -U leocrm -d crm_db -c "SELECT count(*) FROM audit_log WHERE c
# Check database size
psql -h localhost -U leocrm -d crm_db -c "SELECT pg_size_pretty(pg_database_size('crm_db'));"
# Via API (Admin)
curl -b "leocrm_session=<session>" https://crm.media-on.de/api/v1/system/dashboard | jq .
```
---
## 8. Production Resource Recommendations
Die `docker-compose.yaml` hat Development-Defaults. Für Produktion mit 50+ Usern müssen die Limits erhöht werden.
| Service | Development | Production (50+ User) | Begründung |
|---|---|---|---|
| **PostgreSQL RAM** | 512m | 1-2GB | pgvector HNSW + FTS + JSONB Snapshots + Outbox + AuditLog |
| **PostgreSQL CPU** | 1.0 | 2.0 | Vector Search + FTS + normale CRM-Queries |
| **PostgreSQL Disk** | Named Volume | 50-100GB | Embeddings (768 dim × 100k = ~300MB), JSONB Snapshots, Outbox |
| **Redis RAM** | 128m | 256-512m | WS Pub/Sub + Caching + Sessions + ARQ + Rate-Limiting |
| **Redis CPU** | 0.5 | 1.0 | Pub/Sub + Cache + Queue |
| **App (FastAPI) RAM** | Nicht limitiert | 512m-1GB | WebSocket Connections + Async Tasks |
| **App CPU** | Nicht limitiert | 1-2 CPUs | API + WS + LLM-Streaming |
| **Worker (ARQ) RAM** | Nicht limitiert | 256-512m | Background Jobs (Indexierung, Agent-Runs, Extraction) |
| **Worker CPU** | Nicht limitiert | 1-2 CPUs | LLM-Calls + Embedding + Text-Extraction |
### Skalierung bei Bedarf
- **Read-Replicas:** Bei hohem Lese-Aufkommen (Search, FTS, Vector) können Read-Replicas für PostgreSQL eingerichtet werden.
- **Mehr Worker:** Bei hohem Background-Job-Aufkommen können zusätzliche ARQ-Worker-Container gestartet werden.
- **pgvector auslagern:** Bei sehr großen Datasets (>1M Embeddings) kann pgvector auf einen separaten PostgreSQL-Node ausgelagert werden.
- **Redis Cluster:** Bei sehr hohem Cache-/Pub/Sub-Aufkommen kann Redis Cluster eingesetzt werden.