Leopoldadmin/leocrm
1
0
Fork 0
Code Issues 363 Pull Requests Actions 1 Packages Projects Releases 1 Wiki Activity
Files
fix/block-h
leocrm/docs/admin-guide.md
T

19 KiB
Raw Permalink Blame History

LeoCRM Admin Guide

Operations manual for LeoCRM administrators: deployment, backup, restore, environment configuration, monitoring, and troubleshooting.

Table of Contents

  1. Deployment
  2. Environment Configuration
  3. Environment Profiles
  4. System Dashboard
  5. Backup
  6. Restore
  7. Audit Log
  8. Trash Cleanup
  9. Monitoring
  10. Incident Response
  11. Troubleshooting

Deployment

Prerequisites

  • Docker 24+ and Docker Compose v2
  • PostgreSQL 16+ (or use the included Docker container with pgvector)
  • Redis 7+ (or use the included Docker container)
  • A Coolify instance (for managed deployment) or a VPS with Docker

Docker Compose Deployment (Production)

  1. Clone the repository:

    git clone <repo-url> leocrm
cd leocrm

  2. Copy and configure environment:

    cp .env.example .env
# Edit .env — set DATABASE_URL, REDIS_URL, SECRET_KEY, CORS_ORIGINS
nano .env

  3. Start services:

    docker compose up -d

  4. Run database migrations:

    docker compose exec api alembic upgrade head

  5. Create the first admin user:

    docker compose exec api python -c \
  "from app.services.auth_service import bootstrap_first_user; import asyncio; asyncio.run(bootstrap_first_user('admin@example.com', 'SecurePassword123!', 'Admin'))"

  6. Verify health:

    curl http://localhost:8000/api/v1/health

Coolify Deployment

See deploy-guide.md for detailed Coolify deployment instructions.

Manual Deployment (without Docker)

  1. Install Python 3.12+ and PostgreSQL 16+
  2. Create a virtual environment: python3 -m venv .venv && source .venv/bin/activate
  3. Install dependencies: pip install -r requirements.txt
  4. Configure .env (see Environment Configuration)
  5. Run migrations: alembic upgrade head
  6. Start the server: uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 2
  7. Start the ARQ worker: arq app.core.jobs.WorkerSettings

Environment Configuration

All configuration is managed via environment variables (loaded from .env).

Required Variables

Variable Description Example
DATABASE_URL PostgreSQL async connection URL postgresql+asyncpg://user:pass@host:5432/dbname
REDIS_URL Redis connection URL redis://localhost:6379/0
SECRET_KEY Secret key for signing (≥32 chars) Use python3 -c "import secrets; print(secrets.token_urlsafe(48))"
CORS_ORIGINS Comma-separated allowed origins (no wildcards) https://crm.example.com

Database Configuration

Variable Default Description
DB_POOL_SIZE 10 Connection pool size
DB_MAX_OVERFLOW 20 Max overflow connections
DB_ECHO false Echo SQL statements (debug only)

Redis Configuration

Variable Default Description
REDIS_URL redis://localhost:6379/0 Redis connection URL
SESSION_TTL_SECONDS 28800 Session lifetime (8 hours)

SMTP / Email Configuration

Variable Default Description
SMTP_HOST localhost SMTP server hostname
SMTP_PORT 587 SMTP server port
SMTP_USERNAME (none) SMTP username (if auth required)
SMTP_PASSWORD (none) SMTP password (if auth required)
SMTP_FROM_EMAIL noreply@leocrm.local From email address
SMTP_USE_TLS true Use TLS for SMTP connection

Storage Configuration

Variable Default Description
STORAGE_PATH /tmp File storage path (DMS, uploads)

Security Configuration

Variable Default Description
BCRYPT_ROUNDS 12 Password hashing cost factor
SESSION_COOKIE_SECURE false Set true in production (HTTPS only)
SESSION_COOKIE_SAMESITE strict SameSite cookie policy
SESSION_COOKIE_HTTPONLY true HttpOnly cookie flag
PASSWORD_RESET_EXPIRY_HOURS 1 Password reset token lifetime

Rate Limiting

Variable Default Description
RATE_LIMIT_LOGIN_MAX 5 Max login attempts per window
RATE_LIMIT_LOGIN_WINDOW 900 Login window (seconds, 15 min)
RATE_LIMIT_GENERAL_MAX 60 General API rate limit
RATE_LIMIT_GENERAL_WINDOW 60 General window (seconds, 1 min)

Environment Profiles

LeoCRM supports three environment profiles via the ENVIRONMENT variable.

Development (ENVIRONMENT=development)

  • Database: Local PostgreSQL or Docker container
  • Redis: Local instance
  • Logging: DEBUG level (verbose)
  • CORS: http://localhost:5173,http://localhost:3000
  • Session cookie secure: false
  • DB Echo: false (set true for SQL debugging)
  • Auto-reload: uvicorn app.main:app --reload --port 8000

Testing (ENVIRONMENT=testing)

  • Database: leocrm_test database (separate from dev/prod)
  • Redis: Local instance (flushed between tests)
  • Logging: WARNING level (minimal output)
  • Test fixtures: Auto-seeded tenants, users, and test data
  • Run tests: pytest -v --tb=short

Production (ENVIRONMENT=production)

  • Database: Managed PostgreSQL (e.g., Coolify managed DB)
  • Redis: Separate Redis container or managed service
  • Logging: INFO level (structured JSON via structlog)
  • CORS: Production domain only (e.g., https://crm.example.com)
  • Session cookie secure: true (HTTPS only)
  • DB Echo: false
  • Workers: uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 2
  • ARQ Worker: arq app.core.jobs.WorkerSettings
  • Secret key: Must be a secure random string (≥32 chars)

System Dashboard

Das System Dashboard ist die zentrale Monitoring-Oberfläche für Administratoren.

Zugriff

  • Frontend: /system-dashboard (Admin-only, Sidebar-Eintrag nur für Admins sichtbar)
  • API: GET /api/v1/system/dashboard (Admin-only)
  • Alerts: GET /api/v1/system/alerts (Admin-only)

Dashboard-Inhalte

Bereich Metriken
System Health Overall status (healthy/degraded/down)
Database Connections, Table Count, DB Size
Redis Connected Clients, Used Memory, Peak Memory, Uptime
Worker Queue Length, Active Workers, Status
API Stats Total Requests, Error Count, Error Rate, Avg Response Time
Plugins Total Discovered, Active Plugins
Storage Disk Usage, File Count, Status
Alert Feed System Messages aus Communication-System

Alert-Dispatch

Bei Problemen (DB down, Redis down, Worker down, High Error Rate) sendet das System Dashboard automatisch eine System-Message an das Communication-System. Diese erscheint im Alert-Feed und in den Benachrichtigungen der Admins.

# Dashboard abfragen
curl -b "leocrm_session=<session>" https://crm.media-on.de/api/v1/system/dashboard | jq .

# Aktive Alerts abfragen
curl -b "leocrm_session=<session>" https://crm.media-on.de/api/v1/system/alerts | jq .

Siehe auch monitoring.md für Details zu Health Endpoints und Metrics.

Backup

Backup-Konfiguration (Settings → Backup)

Die Backup-Konfiguration erfolgt über die System-Settings API oder das Settings-UI.

Einstellung Default Beschreibung
backup_enabled false Backup Automation aktivieren
backup_interval 24h Backup-Intervall (Cron-Schedule)
backup_retention_days 7 Aufbewahrung in Tagen
backup_destination local Backup-Ziel (local, s3, nextcloud)

API-Endpunkte

Method Path Beschreibung
GET /api/v1/system-settings/backup-config Backup-Konfiguration abfragen
PUT /api/v1/system-settings/backup-config Backup-Konfiguration aktualisieren
POST /api/v1/system-settings/backup-now Sofortiges Backup auslösen (ARQ-Job)
GET /api/v1/system-settings/backup-history Letzte 10 Backup-Ergebnisse (Audit Log) 
# Backup-Konfiguration abfragen
curl -b "leocrm_session=<session>" https://crm.media-on.de/api/v1/system-settings/backup-config | jq .

# Backup aktivieren und konfigurieren
curl -X PUT -b "leocrm_session=<session>" \
  -H "Content-Type: application/json" \
  -d '{"backup_enabled": true, "backup_interval": "24h", "backup_retention_days": 7, "backup_destination": "local"}' \
  https://crm.media-on.de/api/v1/system-settings/backup-config

# Sofortiges Backup auslösen
curl -X POST -b "leocrm_session=<session>" https://crm.media-on.de/api/v1/system-settings/backup-now

# Backup-Historie abfragen
curl -b "leocrm_session=<session>" https://crm.media-on.de/api/v1/system-settings/backup-history | jq .

Automated Backup (ARQ Cron-Job)

  • Job: auto_backup_job (täglich 03:00 Uhr, einstellbar)
  • Script: scripts/backup.py (pg_dump + files)
  • Bei Fehler: System-Message an Communication-System + Audit Log Eintrag
  • Bei Erfolg: Audit Log Eintrag (backup_success)

Manual Database Backup

# Full database dump (recommended daily)
pg_dump -U leocrm -h localhost leocrm > backup_$(date +%Y%m%d).sql

# Compressed backup
pg_dump -U leocrm -h localhost leocrm | gzip > backup_$(date +%Y%m%d).sql.gz

File Storage Backup

# Backup the storage directory
tar -czf storage_$(date +%Y%m%d).tar.gz /data/storage/

Redis Backup

# Save Redis snapshot
redis-cli SAVE
cp /var/lib/redis/dump.rdb /backups/redis_$(date +%Y%m%d).rdb

Restore

Database Restore

# Restore from SQL dump
psql -U leocrm -h localhost leocrm < backup_20260101.sql

# Restore from compressed backup
gunzip -c backup_20260101.sql.gz | psql -U leocrm -h localhost leocrm

File Storage Restore

# Restore storage directory
tar -xzf storage_20260101.tar.gz -C /data/

Redis Restore

# Stop Redis, replace dump file, start Redis
systemctl stop redis
cp /backups/redis_20260101.rdb /var/lib/redis/dump.rdb
systemctl start redis

See scripts/restore.py for the automated restore solution and scripts/restore_test.sh for restore testing.

Audit Log

Audit Log Export (CSV/JSON)

Audit-Logs können als CSV oder JSON exportiert werden. Der Export erfolgt als Streaming-Response (max. 10.000 Einträge).

# CSV Export
curl -b "leocrm_session=<session>" \
  "https://crm.media-on.de/api/v1/audit-log/export?format=csv" \
  -o audit_log_export.csv

# JSON Export
curl -b "leocrm_session=<session>" \
  "https://crm.media-on.de/api/v1/audit-log/export?format=json" \
  -o audit_log_export.json

# Gefilterter Export (nach Entity-Type und Datum)
curl -b "leocrm_session=<session>" \
  "https://crm.media-on.de/api/v1/audit-log/export?format=csv&entity_type=contact&date_from=2026-01-01&date_to=2026-12-31" \
  -o audit_log_contacts_2026.csv

Audit Log Retention (365 Tage)

Audit-Logs werden standardmäßig nach 365 Tagen archiviert/gelöscht. Die Retention ist einstellbar.

  • Default: 365 Tage
  • ARQ-Cron-Job: audit_retention_cleanup (täglich 04:00 Uhr)
  • API: DELETE /api/v1/audit-log/retention?retention_days=365
# Manuelle Retention-Bereinigung (Admin-only)
curl -X DELETE -b "leocrm_session=<session>" \
  "https://crm.media-on.de/api/v1/audit-log/retention?retention_days=365"
# → {"deleted": 1234, "retention_days": 365, "cutoff": "2025-08-20T00:00:00"}

Audit Log Abfrage

# Audit-Logs abfragen (mit Filter und Pagination)
curl -b "leocrm_session=<session>" \
  "https://crm.media-on.de/api/v1/audit-log?entity_type=contact&action=create&page=1&page_size=50" | jq .

Filter-Parameter:

  • entity_type — Filter nach Entity-Typ
  • user_id — Filter nach User-ID
  • action — Filter nach Action (create/update/delete/login)
  • date_from — ISO Datum-Start (inklusive)
  • date_to — ISO Datum-Ende (inklusive)
  • page / page_size — Pagination (max. 200 pro Seite)

Tamper-Proof

  • DELETE auf AuditLog nur mit ?gdpr=true + Admin-Berechtigung
  • Audit-Logs können nicht modifiziert werden (nur erstellen und lesen)

Trash Cleanup

Soft-deleted Entitäten werden nach Ablauf der Retention-Periode endgültig gelöscht.

  • Default: 90 Tage (trash_retention_days)
  • ARQ-Cron-Job: cleanup_expired_trash (täglich 05:00 Uhr)
  • Kriterium: deleted_at < now() - retention_days
  • Bei Löschung: Audit-Log Eintrag

Konfiguration

Die Retention-Periode ist über die System-Settings einstellbar:

# Trash-Retention konfigurieren
curl -X PUT -b "leocrm_session=<session>" \
  -H "Content-Type: application/json" \
  -d '{"trash_retention_days": 90}' \
  https://crm.media-on.de/api/v1/system-settings

Hard-Delete (GDPR)

Einzelne Entitäten können sofort endgültig gelöscht werden (Admin-only):

# Hard-Delete mit GDPR-Flag
curl -X DELETE -b "leocrm_session=<session>" \
  "https://crm.media-on.de/api/v1/contacts/{id}?gdpr=true"

Monitoring

Health Endpoints

# Liveness
curl http://localhost:8000/health/live
# → {"status":"alive"}

# Readiness
curl http://localhost:8000/health/ready
# → {"status":"ready","checks":{"database":"ok","redis":"ok","storage":"ok"}}

# Full health
curl http://localhost:8000/api/v1/health
# → {"status":"healthy","version":"1.0.0","checks":{...}}

Returns JSON with overall status and individual checks:

  • database — PostgreSQL connectivity
  • redis — Redis connectivity
  • storage — Storage directory writability
  • worker — ARQ worker queue status

Status values: healthy (all checks up) or degraded (one or more checks down).

Prometheus Metrics

# Requires admin authentication
curl -H "Cookie: leocrm_session=<session_id>" http://localhost:8000/api/v1/metrics

Available metrics:

  • leocrm_http_requests_total — Total HTTP requests (by method, path, status)
  • leocrm_http_request_duration_seconds — Request duration histogram
  • leocrm_db_pool_connections — Database connection pool size
  • leocrm_arq_jobs_total — Total ARQ background jobs (by function, status)

Structured Logging

LeoCRM uses structlog for structured JSON logging. Logs include:

  • timestamp — ISO timestamp
  • level — Log level (INFO, ERROR, etc.)
  • event — Event name (e.g., api_request)
  • method — HTTP method
  • path — Request path
  • status — HTTP status code
  • duration_ms — Request duration in milliseconds
  • tenant_id — Tenant identifier (when available)

Error logs additionally include:

  • error — Error message
  • traceback — Full stacktrace

Performance Testing

# Seed 200k contacts for performance testing
python scripts/seed_perf_data.py --count 200000

# Verify database indexes
python scripts/check_indexes.py

Siehe monitoring.md für vollständige Monitoring-Dokumentation.

Incident Response

Siehe docs/incident-response-runbook.md für detaillierte Notfall-Prozeduren.

Schnell-Referenz

Incident Erste Maßnahme Eskalation
Server-Ausfall Coolify Restart → Health-Check → System-Message Hetzner Support
DB-Crash PostgreSQL Restart → Migration-Check → Backup-Restore Coolify DB Restart
Redis-Crash Redis Restart → Session-Check Coolify Service Restart
Security-Breach Logs prüfen → Password-Reset → Audit-Log Export Incident Response Team
Backup Failed Backup-Log prüfen → Manueller Retry → Storage prüfen Admin
Worker Down Worker-Container Restart → Queue prüfen Coolify Service Restart

Incident Response Schritte

  1. Erkennen — Alert im System Dashboard oder externes Monitoring
  2. Eingrenzen — Health-Checks, Logs, Metrics prüfen
  3. Beheben — Restart, Restore, Konfiguration anpassen
  4. Verifizieren — Health-Check grün, System Dashboard ok
  5. Dokumentieren — Audit Log Eintrag, Post-Mortem bei Critical

Troubleshooting

Database Connection Issues

Symptom: Health check reports database: down

  1. Verify PostgreSQL is running: systemctl status postgresql
  2. Check connection string in .env: DATABASE_URL=postgresql+asyncpg://...
  3. Test connection: psql -U leocrm -h localhost leocrm
  4. Check pool settings: DB_POOL_SIZE and DB_MAX_OVERFLOW

Redis Connection Issues

Symptom: Health check reports redis: down, sessions not persisting

  1. Verify Redis is running: systemctl status redis or redis-cli ping
  2. Check REDIS_URL in .env
  3. Check Redis logs for errors

Storage Issues

Symptom: Health check reports storage: down, file uploads failing

  1. Verify storage path exists: ls -la $STORAGE_PATH
  2. Check write permissions: touch $STORAGE_PATH/test && rm $STORAGE_PATH/test
  3. Update STORAGE_PATH in .env if needed

Worker Issues

Symptom: Background jobs not processing, health check reports worker: down

  1. Verify ARQ worker is running: ps aux | grep arq
  2. Start worker: arq app.core.jobs.WorkerSettings
  3. Check Redis queue: redis-cli ZCARD arq:queue
  4. Check System Dashboard: GET /api/v1/system/dashboard → .worker

Performance Issues

Symptom: Slow API responses (>500ms)

  1. Run python scripts/check_indexes.py — verify all indexes exist
  2. Check database pool size: increase DB_POOL_SIZE if needed
  3. Seed test data: python scripts/seed_perf_data.py --count 10000
  4. Check Prometheus metrics at /api/v1/metrics for slow endpoints
  5. Enable SQL echo (temporarily): set DB_ECHO=true in .env

Authentication Issues

Symptom: Login fails, 401 errors

  1. Verify user exists and is active
  2. Check session cookie name: SESSION_COOKIE_NAME in .env
  3. Verify SESSION_COOKIE_SECURE is false in development (no HTTPS)
  4. Check Redis for session data: redis-cli KEYS session:*

CORS Errors

Symptom: Browser console shows CORS errors

  1. Check CORS_ORIGINS in .env — must include frontend URL
  2. No wildcards allowed — explicit origins only
  3. Restart server after changing .env

Migration Issues

Symptom: alembic upgrade head fails

  1. Check database connectivity
  2. Verify Alembic config: alembic.ini
  3. Review migration files in alembic/versions/
  4. Check current revision: alembic current
  5. Reset (DESTRUCTIVE): alembic downgrade base && alembic upgrade head

Backup Issues

Symptom: Backup job fails, no backup history

  1. Check System Dashboard for backup alerts
  2. Check Audit Log for backup_failed entries: GET /api/v1/audit-log?action=backup_failed
  3. Verify storage path has enough disk space
  4. Try manual backup: POST /api/v1/system-settings/backup-now
  5. Check scripts/backup.py logs
Reference in New Issue View Git Blame Copy Permalink