leocrm/docs/admin-guide.md

# LeoCRM Admin Guide

> Operations manual for LeoCRM administrators: deployment, backup, restore, environment configuration, monitoring, and troubleshooting.

## Table of Contents

1. [Deployment](#deployment)
2. [Environment Configuration](#environment-configuration)
3. [Environment Profiles](#environment-profiles)
4. [System Dashboard](#system-dashboard)
5. [Backup](#backup)
6. [Restore](#restore)
7. [Audit Log](#audit-log)
8. [Trash Cleanup](#trash-cleanup)
9. [Monitoring](#monitoring)
10. [Incident Response](#incident-response)
11. [Troubleshooting](#troubleshooting)

---

## Deployment

### Prerequisites

- Docker 24+ and Docker Compose v2
- PostgreSQL 16+ (or use the included Docker container with pgvector)
- Redis 7+ (or use the included Docker container)
- A Coolify instance (for managed deployment) or a VPS with Docker

### Docker Compose Deployment (Production)

1. **Clone the repository:**

   ```bash
   git clone <repo-url> leocrm
   cd leocrm
   ```

2. **Copy and configure environment:**

   ```bash
   cp .env.example .env
   # Edit .env — set DATABASE_URL, REDIS_URL, SECRET_KEY, CORS_ORIGINS
   nano .env
   ```

3. **Start services:**

   ```bash
   docker compose up -d
   ```

4. **Run database migrations:**

   ```bash
   docker compose exec api alembic upgrade head
   ```

5. **Create the first admin user:**

   ```bash
   docker compose exec api python -c \
     "from app.services.auth_service import bootstrap_first_user; import asyncio; asyncio.run(bootstrap_first_user('admin@example.com', 'SecurePassword123!', 'Admin'))"
   ```

6. **Verify health:**

   ```bash
   curl http://localhost:8000/api/v1/health
   ```

### Coolify Deployment

See [deploy-guide.md](deploy-guide.md) for detailed Coolify deployment instructions.

### Manual Deployment (without Docker)

1. Install Python 3.12+ and PostgreSQL 16+
2. Create a virtual environment: `python3 -m venv .venv && source .venv/bin/activate`
3. Install dependencies: `pip install -r requirements.txt`
4. Configure `.env` (see [Environment Configuration](#environment-configuration))
5. Run migrations: `alembic upgrade head`
6. Start the server: `uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 2`
7. Start the ARQ worker: `arq app.core.jobs.WorkerSettings`

---

## Environment Configuration

All configuration is managed via environment variables (loaded from `.env`).

### Required Variables

| Variable | Description | Example |
|---|---|---|
| `DATABASE_URL` | PostgreSQL async connection URL | `postgresql+asyncpg://user:pass@host:5432/dbname` |
| `REDIS_URL` | Redis connection URL | `redis://localhost:6379/0` |
| `SECRET_KEY` | Secret key for signing (≥32 chars) | Use `python3 -c "import secrets; print(secrets.token_urlsafe(48))"` |
| `CORS_ORIGINS` | Comma-separated allowed origins (no wildcards) | `https://crm.example.com` |

### Database Configuration

| Variable | Default | Description |
|---|---|---|
| `DB_POOL_SIZE` | `10` | Connection pool size |
| `DB_MAX_OVERFLOW` | `20` | Max overflow connections |
| `DB_ECHO` | `false` | Echo SQL statements (debug only) |

### Redis Configuration

| Variable | Default | Description |
|---|---|---|
| `REDIS_URL` | `redis://localhost:6379/0` | Redis connection URL |
| `SESSION_TTL_SECONDS` | `28800` | Session lifetime (8 hours) |

### SMTP / Email Configuration

| Variable | Default | Description |
|---|---|---|
| `SMTP_HOST` | `localhost` | SMTP server hostname |
| `SMTP_PORT` | `587` | SMTP server port |
| `SMTP_USERNAME` | _(none)_ | SMTP username (if auth required) |
| `SMTP_PASSWORD` | _(none)_ | SMTP password (if auth required) |
| `SMTP_FROM_EMAIL` | `noreply@leocrm.local` | From email address |
| `SMTP_USE_TLS` | `true` | Use TLS for SMTP connection |

### Storage Configuration

| Variable | Default | Description |
|---|---|---|
| `STORAGE_PATH` | `/tmp` | File storage path (DMS, uploads) |

### Security Configuration

| Variable | Default | Description |
|---|---|---|
| `BCRYPT_ROUNDS` | `12` | Password hashing cost factor |
| `SESSION_COOKIE_SECURE` | `false` | Set `true` in production (HTTPS only) |
| `SESSION_COOKIE_SAMESITE` | `strict` | SameSite cookie policy |
| `SESSION_COOKIE_HTTPONLY` | `true` | HttpOnly cookie flag |
| `PASSWORD_RESET_EXPIRY_HOURS` | `1` | Password reset token lifetime |

### Rate Limiting

| Variable | Default | Description |
|---|---|---|
| `RATE_LIMIT_LOGIN_MAX` | `5` | Max login attempts per window |
| `RATE_LIMIT_LOGIN_WINDOW` | `900` | Login window (seconds, 15 min) |
| `RATE_LIMIT_GENERAL_MAX` | `60` | General API rate limit |
| `RATE_LIMIT_GENERAL_WINDOW` | `60` | General window (seconds, 1 min) |

---

## Environment Profiles

LeoCRM supports three environment profiles via the `ENVIRONMENT` variable.

### Development (`ENVIRONMENT=development`)

- **Database**: Local PostgreSQL or Docker container
- **Redis**: Local instance
- **Logging**: DEBUG level (verbose)
- **CORS**: `http://localhost:5173,http://localhost:3000`
- **Session cookie secure**: `false`
- **DB Echo**: `false` (set `true` for SQL debugging)
- **Auto-reload**: `uvicorn app.main:app --reload --port 8000`

### Testing (`ENVIRONMENT=testing`)

- **Database**: `leocrm_test` database (separate from dev/prod)
- **Redis**: Local instance (flushed between tests)
- **Logging**: WARNING level (minimal output)
- **Test fixtures**: Auto-seeded tenants, users, and test data
- **Run tests**: `pytest -v --tb=short`

### Production (`ENVIRONMENT=production`)

- **Database**: Managed PostgreSQL (e.g., Coolify managed DB)
- **Redis**: Separate Redis container or managed service
- **Logging**: INFO level (structured JSON via structlog)
- **CORS**: Production domain only (e.g., `https://crm.example.com`)
- **Session cookie secure**: `true` (HTTPS only)
- **DB Echo**: `false`
- **Workers**: `uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 2`
- **ARQ Worker**: `arq app.core.jobs.WorkerSettings`
- **Secret key**: Must be a secure random string (≥32 chars)

---

## System Dashboard

Das System Dashboard ist die zentrale Monitoring-Oberfläche für Administratoren.

### Zugriff

- **Frontend:** `/system-dashboard` (Admin-only, Sidebar-Eintrag nur für Admins sichtbar)
- **API:** `GET /api/v1/system/dashboard` (Admin-only)
- **Alerts:** `GET /api/v1/system/alerts` (Admin-only)

### Dashboard-Inhalte

| Bereich | Metriken |
|---------|----------|
| **System Health** | Overall status (healthy/degraded/down) |
| **Database** | Connections, Table Count, DB Size |
| **Redis** | Connected Clients, Used Memory, Peak Memory, Uptime |
| **Worker** | Queue Length, Active Workers, Status |
| **API Stats** | Total Requests, Error Count, Error Rate, Avg Response Time |
| **Plugins** | Total Discovered, Active Plugins |
| **Storage** | Disk Usage, File Count, Status |
| **Alert Feed** | System Messages aus Communication-System |

### Alert-Dispatch

Bei Problemen (DB down, Redis down, Worker down, High Error Rate) sendet das System Dashboard automatisch eine System-Message an das Communication-System. Diese erscheint im Alert-Feed und in den Benachrichtigungen der Admins.

```bash
# Dashboard abfragen
curl -b "leocrm_session=<session>" https://crm.media-on.de/api/v1/system/dashboard | jq .

# Aktive Alerts abfragen
curl -b "leocrm_session=<session>" https://crm.media-on.de/api/v1/system/alerts | jq .
```

Siehe auch [monitoring.md](monitoring.md) für Details zu Health Endpoints und Metrics.

---

## Backup

### Backup-Konfiguration (Settings → Backup)

Die Backup-Konfiguration erfolgt über die System-Settings API oder das Settings-UI.

| Einstellung | Default | Beschreibung |
|-------------|---------|-------------|
| `backup_enabled` | `false` | Backup Automation aktivieren |
| `backup_interval` | `24h` | Backup-Intervall (Cron-Schedule) |
| `backup_retention_days` | `7` | Aufbewahrung in Tagen |
| `backup_destination` | `local` | Backup-Ziel (local, s3, nextcloud) |

#### API-Endpunkte

| Method | Path | Beschreibung |
|--------|------|-------------|
| GET | `/api/v1/system-settings/backup-config` | Backup-Konfiguration abfragen |
| PUT | `/api/v1/system-settings/backup-config` | Backup-Konfiguration aktualisieren |
| POST | `/api/v1/system-settings/backup-now` | Sofortiges Backup auslösen (ARQ-Job) |
| GET | `/api/v1/system-settings/backup-history` | Letzte 10 Backup-Ergebnisse (Audit Log) |

```bash
# Backup-Konfiguration abfragen
curl -b "leocrm_session=<session>" https://crm.media-on.de/api/v1/system-settings/backup-config | jq .

# Backup aktivieren und konfigurieren
curl -X PUT -b "leocrm_session=<session>" \
  -H "Content-Type: application/json" \
  -d '{"backup_enabled": true, "backup_interval": "24h", "backup_retention_days": 7, "backup_destination": "local"}' \
  https://crm.media-on.de/api/v1/system-settings/backup-config

# Sofortiges Backup auslösen
curl -X POST -b "leocrm_session=<session>" https://crm.media-on.de/api/v1/system-settings/backup-now

# Backup-Historie abfragen
curl -b "leocrm_session=<session>" https://crm.media-on.de/api/v1/system-settings/backup-history | jq .
```

### Automated Backup (ARQ Cron-Job)

- **Job:** `auto_backup_job` (täglich 03:00 Uhr, einstellbar)
- **Script:** `scripts/backup.py` (pg_dump + files)
- **Bei Fehler:** System-Message an Communication-System + Audit Log Eintrag
- **Bei Erfolg:** Audit Log Eintrag (`backup_success`)

### Manual Database Backup

```bash
# Full database dump (recommended daily)
pg_dump -U leocrm -h localhost leocrm > backup_$(date +%Y%m%d).sql

# Compressed backup
pg_dump -U leocrm -h localhost leocrm | gzip > backup_$(date +%Y%m%d).sql.gz
```

### File Storage Backup

```bash
# Backup the storage directory
tar -czf storage_$(date +%Y%m%d).tar.gz /data/storage/
```

### Redis Backup

```bash
# Save Redis snapshot
redis-cli SAVE
cp /var/lib/redis/dump.rdb /backups/redis_$(date +%Y%m%d).rdb
```

---

## Restore

### Database Restore

```bash
# Restore from SQL dump
psql -U leocrm -h localhost leocrm < backup_20260101.sql

# Restore from compressed backup
gunzip -c backup_20260101.sql.gz | psql -U leocrm -h localhost leocrm
```

### File Storage Restore

```bash
# Restore storage directory
tar -xzf storage_20260101.tar.gz -C /data/
```

### Redis Restore

```bash
# Stop Redis, replace dump file, start Redis
systemctl stop redis
cp /backups/redis_20260101.rdb /var/lib/redis/dump.rdb
systemctl start redis
```

See `scripts/restore.py` for the automated restore solution and `scripts/restore_test.sh` for restore testing.

---

## Audit Log

### Audit Log Export (CSV/JSON)

Audit-Logs können als CSV oder JSON exportiert werden. Der Export erfolgt als Streaming-Response (max. 10.000 Einträge).

```bash
# CSV Export
curl -b "leocrm_session=<session>" \
  "https://crm.media-on.de/api/v1/audit-log/export?format=csv" \
  -o audit_log_export.csv

# JSON Export
curl -b "leocrm_session=<session>" \
  "https://crm.media-on.de/api/v1/audit-log/export?format=json" \
  -o audit_log_export.json

# Gefilterter Export (nach Entity-Type und Datum)
curl -b "leocrm_session=<session>" \
  "https://crm.media-on.de/api/v1/audit-log/export?format=csv&entity_type=contact&date_from=2026-01-01&date_to=2026-12-31" \
  -o audit_log_contacts_2026.csv
```

### Audit Log Retention (365 Tage)

Audit-Logs werden standardmäßig nach 365 Tagen archiviert/gelöscht. Die Retention ist einstellbar.

- **Default:** 365 Tage
- **ARQ-Cron-Job:** `audit_retention_cleanup` (täglich 04:00 Uhr)
- **API:** `DELETE /api/v1/audit-log/retention?retention_days=365`

```bash
# Manuelle Retention-Bereinigung (Admin-only)
curl -X DELETE -b "leocrm_session=<session>" \
  "https://crm.media-on.de/api/v1/audit-log/retention?retention_days=365"
# → {"deleted": 1234, "retention_days": 365, "cutoff": "2025-08-20T00:00:00"}
```

### Audit Log Abfrage

```bash
# Audit-Logs abfragen (mit Filter und Pagination)
curl -b "leocrm_session=<session>" \
  "https://crm.media-on.de/api/v1/audit-log?entity_type=contact&action=create&page=1&page_size=50" | jq .
```

**Filter-Parameter:**
- `entity_type` — Filter nach Entity-Typ
- `user_id` — Filter nach User-ID
- `action` — Filter nach Action (create/update/delete/login)
- `date_from` — ISO Datum-Start (inklusive)
- `date_to` — ISO Datum-Ende (inklusive)
- `page` / `page_size` — Pagination (max. 200 pro Seite)

### Tamper-Proof

- DELETE auf AuditLog nur mit `?gdpr=true` + Admin-Berechtigung
- Audit-Logs können nicht modifiziert werden (nur erstellen und lesen)

---

## Trash Cleanup

Soft-deleted Entitäten werden nach Ablauf der Retention-Periode endgültig gelöscht.

- **Default:** 90 Tage (`trash_retention_days`)
- **ARQ-Cron-Job:** `cleanup_expired_trash` (täglich 05:00 Uhr)
- **Kriterium:** `deleted_at < now() - retention_days`
- **Bei Löschung:** Audit-Log Eintrag

### Konfiguration

Die Retention-Periode ist über die System-Settings einstellbar:

```bash
# Trash-Retention konfigurieren
curl -X PUT -b "leocrm_session=<session>" \
  -H "Content-Type: application/json" \
  -d '{"trash_retention_days": 90}' \
  https://crm.media-on.de/api/v1/system-settings
```

### Hard-Delete (GDPR)

Einzelne Entitäten können sofort endgültig gelöscht werden (Admin-only):

```bash
# Hard-Delete mit GDPR-Flag
curl -X DELETE -b "leocrm_session=<session>" \
  "https://crm.media-on.de/api/v1/contacts/{id}?gdpr=true"
```

---

## Monitoring

### Health Endpoints

```bash
# Liveness
curl http://localhost:8000/health/live
# → {"status":"alive"}

# Readiness
curl http://localhost:8000/health/ready
# → {"status":"ready","checks":{"database":"ok","redis":"ok","storage":"ok"}}

# Full health
curl http://localhost:8000/api/v1/health
# → {"status":"healthy","version":"1.0.0","checks":{...}}
```

Returns JSON with overall status and individual checks:
- `database` — PostgreSQL connectivity
- `redis` — Redis connectivity
- `storage` — Storage directory writability
- `worker` — ARQ worker queue status

Status values: `healthy` (all checks up) or `degraded` (one or more checks down).

### Prometheus Metrics

```bash
# Requires admin authentication
curl -H "Cookie: leocrm_session=<session_id>" http://localhost:8000/api/v1/metrics
```

Available metrics:
- `leocrm_http_requests_total` — Total HTTP requests (by method, path, status)
- `leocrm_http_request_duration_seconds` — Request duration histogram
- `leocrm_db_pool_connections` — Database connection pool size
- `leocrm_arq_jobs_total` — Total ARQ background jobs (by function, status)

### Structured Logging

LeoCRM uses `structlog` for structured JSON logging. Logs include:
- `timestamp` — ISO timestamp
- `level` — Log level (INFO, ERROR, etc.)
- `event` — Event name (e.g., `api_request`)
- `method` — HTTP method
- `path` — Request path
- `status` — HTTP status code
- `duration_ms` — Request duration in milliseconds
- `tenant_id` — Tenant identifier (when available)

Error logs additionally include:
- `error` — Error message
- `traceback` — Full stacktrace

### Performance Testing

```bash
# Seed 200k contacts for performance testing
python scripts/seed_perf_data.py --count 200000

# Verify database indexes
python scripts/check_indexes.py
```

Siehe [monitoring.md](monitoring.md) für vollständige Monitoring-Dokumentation.

---

## Incident Response

Siehe `docs/incident-response-runbook.md` für detaillierte Notfall-Prozeduren.

### Schnell-Referenz

| Incident | Erste Maßnahme | Eskalation |
|----------|---------------|-----------|
| **Server-Ausfall** | Coolify Restart → Health-Check → System-Message | Hetzner Support |
| **DB-Crash** | PostgreSQL Restart → Migration-Check → Backup-Restore | Coolify DB Restart |
| **Redis-Crash** | Redis Restart → Session-Check | Coolify Service Restart |
| **Security-Breach** | Logs prüfen → Password-Reset → Audit-Log Export | Incident Response Team |
| **Backup Failed** | Backup-Log prüfen → Manueller Retry → Storage prüfen | Admin |
| **Worker Down** | Worker-Container Restart → Queue prüfen | Coolify Service Restart |

### Incident Response Schritte

1. **Erkennen** — Alert im System Dashboard oder externes Monitoring
2. **Eingrenzen** — Health-Checks, Logs, Metrics prüfen
3. **Beheben** — Restart, Restore, Konfiguration anpassen
4. **Verifizieren** — Health-Check grün, System Dashboard ok
5. **Dokumentieren** — Audit Log Eintrag, Post-Mortem bei Critical

---

## Troubleshooting

### Database Connection Issues

**Symptom**: Health check reports `database: down`

1. Verify PostgreSQL is running: `systemctl status postgresql`
2. Check connection string in `.env`: `DATABASE_URL=postgresql+asyncpg://...`
3. Test connection: `psql -U leocrm -h localhost leocrm`
4. Check pool settings: `DB_POOL_SIZE` and `DB_MAX_OVERFLOW`

### Redis Connection Issues

**Symptom**: Health check reports `redis: down`, sessions not persisting

1. Verify Redis is running: `systemctl status redis` or `redis-cli ping`
2. Check `REDIS_URL` in `.env`
3. Check Redis logs for errors

### Storage Issues

**Symptom**: Health check reports `storage: down`, file uploads failing

1. Verify storage path exists: `ls -la $STORAGE_PATH`
2. Check write permissions: `touch $STORAGE_PATH/test && rm $STORAGE_PATH/test`
3. Update `STORAGE_PATH` in `.env` if needed

### Worker Issues

**Symptom**: Background jobs not processing, health check reports `worker: down`

1. Verify ARQ worker is running: `ps aux | grep arq`
2. Start worker: `arq app.core.jobs.WorkerSettings`
3. Check Redis queue: `redis-cli ZCARD arq:queue`
4. Check System Dashboard: `GET /api/v1/system/dashboard` → `.worker`

### Performance Issues

**Symptom**: Slow API responses (>500ms)

1. Run `python scripts/check_indexes.py` — verify all indexes exist
2. Check database pool size: increase `DB_POOL_SIZE` if needed
3. Seed test data: `python scripts/seed_perf_data.py --count 10000`
4. Check Prometheus metrics at `/api/v1/metrics` for slow endpoints
5. Enable SQL echo (temporarily): set `DB_ECHO=true` in `.env`

### Authentication Issues

**Symptom**: Login fails, 401 errors

1. Verify user exists and is active
2. Check session cookie name: `SESSION_COOKIE_NAME` in `.env`
3. Verify `SESSION_COOKIE_SECURE` is `false` in development (no HTTPS)
4. Check Redis for session data: `redis-cli KEYS session:*`

### CORS Errors

**Symptom**: Browser console shows CORS errors

1. Check `CORS_ORIGINS` in `.env` — must include frontend URL
2. No wildcards allowed — explicit origins only
3. Restart server after changing `.env`

### Migration Issues

**Symptom**: `alembic upgrade head` fails

1. Check database connectivity
2. Verify Alembic config: `alembic.ini`
3. Review migration files in `alembic/versions/`
4. Check current revision: `alembic current`
5. Reset (DESTRUCTIVE): `alembic downgrade base && alembic upgrade head`

### Backup Issues

**Symptom**: Backup job fails, no backup history

1. Check System Dashboard for backup alerts
2. Check Audit Log for `backup_failed` entries: `GET /api/v1/audit-log?action=backup_failed`
3. Verify storage path has enough disk space
4. Try manual backup: `POST /api/v1/system-settings/backup-now`
5. Check `scripts/backup.py` logs