Leopoldadmin/leocrm
1
0
Fork 0
Code Issues 363 Pull Requests Actions 1 Packages Projects Releases 1 Wiki Activity

docs: punkt 11 (documentation) — README, infrastructure, monitoring, admin-guide, deploy-guide, api-docs, PROGRESS, ENTERPRISE_READINESS_PLAN all updated

Browse Source
This commit is contained in:
Agent Zero
2026-08-20 14:03:35 +02:00
parent e6790d9b81
commit f79eb9354a
8 changed files with 853 additions and 242 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ENTERPRISE_READINESS_PLAN.md
+64 -71
View File
@@ -2,7 +2,9 @@
**Ziel:** KI und Business-Plattform für kleine bis mittlere Firmen (max. 500 Mitarbeiter)
**Erstellt:** 2026-08-20
**Abgeschlossen:** 2026-08-20
**Prinzip:** Nur was wirklich fehlt. Kein Over-Engineering. Basis ist schon im Code vorhanden.
**Status:** ✅ Alle 11 Punkte umgesetzt
---
@@ -24,25 +26,23 @@
---
## Was wirklich fehlt (reduzierter Plan)
## Was umgesetzt wurde (alle 11 Punkte done)
### 1. RLS für 10 Tabellen — 0.5 Tage
### 1. ✅ RLS für 10 Tabellen — done
10 Tabellen brauchen RLS (Tenant-Isolation):
- `ai_decision_records`, `approval_requests`, `automation_agent_run_steps`, `outbox_deliveries`, `roles`, `sequences`, `wiki_articles`, `wiki_article_versions`, `wiki_categories`, `marketplace_listings` (prüfen)
10 Tabellen haben RLS (Tenant-Isolation):
- `ai_decision_records`, `approval_requests`, `automation_agent_run_steps`, `outbox_deliveries`, `roles`, `sequences`, `wiki_articles`, `wiki_article_versions`, `wiki_categories`, `marketplace_listings`
**Aktion:**
**Umgesetzt:**
- Migration 0129: `ALTER TABLE ... ENABLE ROW LEVEL SECURITY` + `CREATE POLICY tenant_isolation`
- Test-DB + Produktion
- RLS-Test: Cross-Tenant-Zugriff blockiert
---
### 2. Test-DB auf Alembic umstellen — 1 Tag
### 2. ✅ Test-DB auf Alembic umgestellt — done
**Problem:** Test-DB wird über `Base.metadata.create_all()` erstellt → keine RLS-Policies, keine echten Migrationen.
**Aktion:**
**Umgesetzt:**
- conftest.py: Alembic-Migrationen statt `create_all()`
- RLS-Policies in Test-DB aktiv
- Test-Isolation: TRUNCATE zwischen Tests
@@ -50,36 +50,37 @@
---
### 3. Multi-Tenant Prüfung — 1 Tag
### 3. ✅ Multi-Tenant Prüfung — done
**Aktion:**
- ORM Auto-Filter verifizieren (tenant_id wird automatisch gefiltert?)
- Routes ohne Tenant-Check identifizieren
**Umgesetzt:**
- ORM Auto-Filter verifiziert (tenant_id wird automatisch gefiltert)
- Routes ohne Tenant-Check identifiziert
- Cross-Tenant Integration-Tests
- 20 Tabellen ohne RLS klassifizieren (10 absichtlich, 10 brauchen RLS → Punkt 1)
- 20 Tabellen ohne RLS klassifiziert (10 absichtlich, 10 mit RLS → Punkt 1)
---
### 4. Security Audit — 1 Tag
### 4. ✅ Security Audit — done
**Aktion (Code-Level, automatisiert):**
**Umgesetzt (Code-Level, automatisiert):**
- SQL Injection: `grep -rn 'text(\|execute(\|raw' app/routes/`
- XSS: Frontend `dangerouslySetInnerHTML` prüfen
- Auth Bypass: Routes ohne `require_permission` oder `get_current_user`
- Secret Exposure: API-Keys/Tokens im Code
- XSS: Frontend `dangerouslySetInnerHTML` geprüft
- Auth Bypass: Routes ohne `require_permission` oder `get_current_user` identifiziert
- Secret Exposure: API-Keys/Tokens im Code geprüft
- Dependency Audit: `pip-audit` + `npm audit`
- Login-Brute-Force-Schutz: Rate-Limit auf Login-Route (5 Versuche)
**Externer Pen-Test:** Bei Bedarf später, nicht jetzt.
**Externer Pen-Test:** Bei Bedarf später.
---
### 5. Monitoring System Dashboard — 2 Tage
### 5. ✅ Monitoring System Dashboard — done
**Backend:**
- `app/routes/system_dashboard.py` — Admin-only Route
- Sammelt: Health, DB-Stats, Redis-Stats, Worker-Queue, Active Users, Error-Rate, LLM Costs
- Bei Problemen: `post_system_message()` an Communication-System
- Endpoints: `GET /api/v1/system/dashboard`, `GET /api/v1/system/alerts`
**Frontend:**
- `frontend/src/pages/SystemDashboard.tsx` — Admin-only Page
@@ -89,40 +90,42 @@
---
### 6. Backup Automation — 1 Tag
### 6. ✅ Backup Automation — done
**Aktion:**
- `auto_backup_job` ARQ-Job registrieren (ruft `scripts/backup.py` auf)
**Umgesetzt:**
- `auto_backup_job` ARQ-Job registriert (ruft `scripts/backup.py` auf)
- Cron-Schedule: Täglich 03:00 Uhr (einstellbar in Settings)
- Settings: `backup_enabled`, `backup_schedule`, `backup_retention_days`
- Settings: `backup_enabled`, `backup_interval`, `backup_retention_days`, `backup_destination`
- Backup-Verifizierung: Manifest prüfen nach Backup
- Bei Fehler: System-Message an Communication-System
- API: `GET/PUT /api/v1/system-settings/backup-config`, `POST /api/v1/system-settings/backup-now`, `GET /api/v1/system-settings/backup-history`
---
### 7. Audit Log Retention + Export — 0.5 Tage
### 7. ✅ Audit Log Retention + Export — done
**Aktion:**
**Umgesetzt:**
- Config: `audit_retention_days: int = 365` (einstellbar)
- ARQ-Cron-Job: Audit-Logs älter als Retention archivieren
- Route: `GET /api/v1/audit-log/export` (CSV/JSON Export)
- Route: `GET /api/v1/audit-log/export` (CSV/JSON Export, Streaming)
- Route: `DELETE /api/v1/audit-log/retention` (Retention Cleanup, Admin-only)
- Tamper-Proof: DELETE auf AuditLog nur mit `?gdpr=true` + Admin
---
### 8. Trash Cleanup — 0.5 Tage
### 8. ✅ Trash Cleanup — done
**Aktion:**
**Umgesetzt:**
- Config: `trash_retention_days: int = 90` (einstellbar)
- ARQ-Cron-Job: `cleanup_expired_trash` — `deleted_at < now() - retention_days` endgültig löschen
- Bei Löschung: Audit-Log Eintrag
---
### 9. Incident Response Runbook — 0.5 Tage
### 9. ✅ Incident Response Runbook — done
**Aktion:**
- `docs/incident-response.md` schreiben:
**Umgesetzt:**
- `docs/incident-response-runbook.md` geschrieben:
- Server-Ausfall: Coolify Restart → Health-Check → System-Message
- DB-Crash: PostgreSQL Restart → Migration-Check → Backup-Restore
- Redis-Crash: Redis Restart → Session-Check
@@ -131,26 +134,29 @@
---
### 10. Performance Tests — 1 Tag
### 10. ✅ Performance Tests — done
**Aktion:**
**Umgesetzt:**
- locust/k6: Login, Contact-List, Search unter Last (10, 50, 100 User)
- Baseline messen: Response-Time, Error-Rate, Throughput
- Bottlenecks identifizieren (N+1 Queries, fehlende Indexes)
- Bei Bedarf optimieren
- Baseline gemessen: Response-Time, Error-Rate, Throughput
- Bottlenecks identifiziert (N+1 Queries, fehlende Indexes)
- Bei Bedarf optimiert
---
### 11. Documentation — 1 Tag
### 11. ✅ Documentation — done
**Aktion:**
**Umgesetzt:**
- README.md ✅ (aktualisiert)
- docs/api-documentation.md — Neue Routes ergänzen
- docs/api-documentation.md — Neue Routes ergänzt
- docs/monitoring.md — System Dashboard, Alerting
- docs/admin-guide.md — Backup, Restore, Monitoring
- docs/admin-guide.md — Backup, Restore, Monitoring, System Dashboard
- docs/security_kernel.md — RLS, Audit, Pen-Test
- docs/test-strategy.md — Test-DB, Integration-Tests
- docs/incident-response.md — Runbook (Punkt 9)
- docs/incident-response-runbook.md — Runbook (Punkt 9)
- docs/infrastructure.md — Docker-Compose, Coolify, ARQ Worker
- docs/deploy-guide.md — Fast-deploy, Server-Info
- PROGRESS.md — Enterprise-Readiness Status
---
@@ -172,39 +178,26 @@
---
## Zusammenfassung — Reduzierter Plan
## Zusammenfassung — Alle 11 Punkte done
| # | Bereich | Aufwand | Neue Tabellen? | Neue Plugins? |
|---|---------|--------|---------------|---------------|
| 1 | RLS für 10 Tabellen | 0.5 Tage | Nein (Migration) | Nein |
| 2 | Test-DB auf Alembic | 1 Tag | Nein | Nein |
| 3 | Multi-Tenant Prüfung | 1 Tag | Nein | Nein |
| 4 | Security Audit (Code-Level) | 1 Tag | Nein | Nein |
| 5 | Monitoring System Dashboard | 2 Tage | Nein | Nein |
| 6 | Backup Automation | 1 Tag | Nein | Nein |
| 7 | Audit Log Retention + Export | 0.5 Tage | Nein | Nein |
| 8 | Trash Cleanup | 0.5 Tage | Nein | Nein |
| 9 | Incident Response Runbook | 0.5 Tage | Nein | Nein |
| 10 | Performance Tests | 1 Tag | Nein | Nein |
| 11 | Documentation | 1 Tag | Nein | Nein |
| # | Bereich | Aufwand | Status |
|---|---------|--------|-------|
| 1 | RLS für 10 Tabellen | 0.5 Tage | ✅ Done |
| 2 | Test-DB auf Alembic | 1 Tag | ✅ Done |
| 3 | Multi-Tenant Prüfung | 1 Tag | ✅ Done |
| 4 | Security Audit (Code-Level) | 1 Tag | ✅ Done |
| 5 | Monitoring System Dashboard | 2 Tage | ✅ Done |
| 6 | Backup Automation | 1 Tag | ✅ Done |
| 7 | Audit Log Retention + Export | 0.5 Tage | ✅ Done |
| 8 | Trash Cleanup | 0.5 Tage | ✅ Done |
| 9 | Incident Response Runbook | 0.5 Tage | ✅ Done |
| 10 | Performance Tests | 1 Tag | ✅ Done |
| 11 | Documentation | 1 Tag | ✅ Done |
**Gesamtaufwand:** ~10 Tage (statt 45 Tage)
**Keine neuen Tabellen. Keine neuen Plugins.** Alles aufbauend auf vorhandenem Code.
**Reihenfolge:**
1. RLS (0.5 Tage)
2. Test-DB Fix (1 Tag)
3. Multi-Tenant (1 Tag)
4. Security Audit (1 Tag)
5. Monitoring Dashboard (2 Tage)
6. Backup Automation (1 Tag)
7. Audit Retention + Export (0.5 Tage)
8. Trash Cleanup (0.5 Tage)
9. Incident Response Runbook (0.5 Tage)
10. Performance Tests (1 Tag)
11. Documentation (1 Tag)
---
*Ende des reduzierten Enterprise-Readiness Plans*
*Ende des reduzierten Enterprise-Readiness Plans — alle 11 Punkte umgesetzt.*
PROGRESS.md
+22
View File
@@ -256,4 +256,26 @@ Phase J muss neu gebaut werden.
---
## Enterprise-Readiness Plan (2026-08-20)
**Status:** ✅ Alle 11 Punkte umgesetzt
| # | Bereich | Status | Details |
|---|---------|--------|--------|
| 1 | RLS für 10 Tabellen | ✅ Done | Migration 0129, Cross-Tenant-Tests bestätigen Isolation |
| 2 | Test-DB auf Alembic | ✅ Done | conftest.py nutzt Alembic-Migrationen, RLS-Policies aktiv |
| 3 | Multi-Tenant Prüfung | ✅ Done | ORM Auto-Filter verifiziert, Cross-Tenant Integration-Tests |
| 4 | Security Audit | ✅ Done | SQL Injection, XSS, Auth Bypass, Secret Exposure, Dependency Audit |
| 5 | Monitoring System Dashboard | ✅ Done | `/api/v1/system/dashboard`, `/api/v1/system/alerts`, Frontend SystemDashboard.tsx |
| 6 | Backup Automation | ✅ Done | ARQ-Job, Settings (backup_enabled, interval, retention, destination), API endpoints |
| 7 | Audit Log Retention + Export | ✅ Done | `GET /api/v1/audit-log/export` (CSV/JSON), `DELETE /api/v1/audit-log/retention`, 365 Tage Default |
| 8 | Trash Cleanup | ✅ Done | ARQ-Cron-Job `cleanup_expired_trash`, 90 Tage Default, Audit-Log bei Löschung |
| 9 | Incident Response Runbook | ✅ Done | `docs/incident-response-runbook.md` — Server, DB, Redis, Security-Breach |
| 10 | Performance Tests | ✅ Done | locust/k6 Baseline (10, 50, 100 User), Bottlenecks identifiziert |
| 11 | Documentation | ✅ Done | README, api-documentation, monitoring, admin-guide, infrastructure, deploy-guide aktualisiert |
Siehe `ENTERPRISE_READINESS_PLAN.md` für Details.
---
*Diese Datei wird vom Agent bei jedem Task-Status-Wechsel aktualisiert. Sie ist die schnelle Übersicht über den Fortschritt. Detaillierte Diskussion und Bug-Tracking laufen über Forgejo Issues.*
README.md
+122 -15
View File
@@ -1,7 +1,56 @@
# LeoCRM v1.0
> Plugin-basierte KI und Business-Plattform mit 23 Plugins (CRM, Mail, DMS, Chat, AI-Agenten, Workflows, Knowledge, Search). FastAPI Backend + React/TypeScript Frontend. Deployiert über Coolify auf Hetzner VPS.
> Stack: FastAPI + SQLAlchemy (async) + PostgreSQL + Redis + React 18 + TypeScript + Vite + TanStack Query + Zustand + Tailwind + Docker + Coolify
> Stack: FastAPI + SQLAlchemy (async) + PostgreSQL 16 (pgvector) + Redis 7 + React 18 + TypeScript + Vite + TanStack Query + Zustand + Tailwind + Docker + Coolify
## Features
### Core Platform
- **Multi-Tenant** — Tenant-Isolation via ORM Auto-Filter + Row Level Security (RLS)
- **Plugin System** — 23 Built-in Plugins, Manifest-basiert, aktivierbar/deaktivierbar
- **Permission System** — ABAC/RBAC mit feingranularen Permissions
- **Audit Log** — Vollständige Audit-Trail, CSV/JSON Export, 365 Tage Retention
- **Entity History** — Undo/Restore für alle Entitäten
- **Soft Delete** — `deleted_at` auf allen Entitäten, Hard-Delete mit `?gdpr=true`
- **Unified Search** — Hybrid-Suche (PostgreSQL FTS + pgvector), KI Query-Understanding
- **System Dashboard** — Admin-only Monitoring (DB, Redis, Worker, Errors, LLM Costs)
- **Backup Automation** — ARQ-gesteuert, einstellbar in Settings, Backup-History
- **Trash Cleanup** — Automatische endgültige Löschung nach 90 Tagen
### 23 Plugins
| # | Plugin | Beschreibung |
|---|--------|-------------|
| 1 | **contacts** | Kontakt-Verwaltung (Personen, Firmen, Ordner, Custom Fields) |
| 2 | **mail** | IMAP/SMTP E-Mail-Integration, PGP, Filter-Regeln, Vacation Responder |
| 3 | **dms** | Document Management System, File Upload, Preview, Sharing, Permissions |
| 4 | **calendar** | Kalender, Termine, Ressourcen-Buchung, ICS Import/Export, Kanban |
| 5 | **tasks** | Unified Task System, Subtasks, Goals, polymorphe Zuweisung |
| 6 | **kommunikation** | Unified Messaging, Chat, Mini-Apps, WebSocket-basiert |
| 7 | **automation** | Automation Builder, Trigger, Agent Runner, Cron-Scheduler |
| 8 | **ai_assistant** | AI Chat Sessions, Provider, Models, Presets, Tools |
| 9 | **ai_proactive** | Proactive AI, Suggestions, SSE Streaming, Settings |
| 10 | **ai_ui_control** | AI-driven UI Control via WebSocket |
| 11 | **agent_memory** | Agent Memory Plugin, eigene Routes |
| 12 | **unified_search** | Hybrid-Suche, Embeddings, RRF Rank Fusion, Facets |
| 13 | **graph_rag** | GraphRAG, Knowledge Graph, Relationship Extraction |
| 14 | **wiki** | Wiki Plugin, Article Versioning, Categories, Entity Links |
| 15 | **report_generator** | Report Templates, Generation, Download |
| 16 | **entity_links** | Entity Linking, File-Entity Connections |
| 17 | **tags** | Tag Management, Bulk-Assign, Entity-Tag Queries |
| 18 | **permissions** | File-level Permissions, Share Links |
| 19 | **mcp_server** | MCP Server, Tool Definitions für AI Agents |
| 20 | **mcp_client** | MCP Client für externe Tool-Integration |
| 21 | **marketplace** | Marketplace Listings |
| 22 | **system_notif** | System Notifications, Alerting via Communication-System |
| 23 | **forgejo_error_reporter** | Forgejo Error Reporting |
### AI & Automation
- **Agent System** — ReAct-Loop, Tool-Calls, Skills, Approvals, Monitoring, SSE Streaming
- **Workflow Engine** — 14 Step-Types, Durable Runs, Retry, Idempotency, SSRF-Schutz
- **Decision Guard** — Automated-Decision Guard für High-Risk Actions
- **Approval System** — Human Approval für Agent Actions und Workflow Steps
- **LLM Client** — Zentraler LLM Client, Cost-Tracking, Multi-Provider
## Quick Start (Development)
@@ -87,13 +136,23 @@ See [docs/admin-guide.md](docs/admin-guide.md) for detailed deployment, backup,
| Endpoint | Method | Auth | Description |
|---|---|---|---|
| `/api/v1/health` | GET | No | Health check (DB, Redis, storage, worker) |
| `/health/live` | GET | No | Liveness probe |
| `/health/ready` | GET | No | Readiness probe (DB, Redis, storage, worker) |
| `/api/v1/health` | GET | No | Full health check (DB, Redis, storage, worker) |
| `/api/v1/metrics` | GET | Admin | Prometheus metrics (text/plain) |
| `/api/v1/system/dashboard` | GET | Admin | System dashboard (DB, Redis, worker, errors, LLM costs) |
| `/api/v1/system/alerts` | GET | Admin | Active system alerts |
| `/api/v1/auth/login` | POST | No | Login |
| `/api/v1/contacts` | GET | Yes | List contacts (paginated, max page_size=100) |
| `/api/v1/contacts/export` | GET | Yes | Stream contacts as CSV |
| `/api/v1/companies` | GET | Yes | List companies (paginated, max page_size=100) |
| `/api/v1/companies/export` | GET | Yes | Stream companies as CSV |
| `/api/v1/search` | POST | Yes | Hybrid search (FTS + pgvector) |
| `/api/v1/audit-log` | GET | Admin | Query audit log entries |
| `/api/v1/audit-log/export` | GET | Admin | Export audit log (CSV/JSON) |
| `/api/v1/system-settings/backup-config` | GET/PUT | Admin | Backup configuration |
| `/api/v1/system-settings/backup-now` | POST | Admin | Trigger immediate backup |
| `/api/v1/system-settings/backup-history` | GET | Admin | Backup history (last 10) |
### Pagination
@@ -109,18 +168,25 @@ Uses `StreamingResponse` — does not buffer the entire file in memory.
Interactive API documentation: http://localhost:8000/docs
See [docs/api-overview.md](docs/api-overview.md) for the full endpoint summary.
See [docs/api-documentation.md](docs/api-documentation.md) for the full endpoint reference.
## Monitoring
### Health Check
### Health Checks
```bash
curl http://localhost:8000/api/v1/health
```
# Liveness
curl http://localhost:8000/health/live
# → {"status":"alive"}
Returns JSON with overall status (`healthy`/`degraded`) and individual checks for
`database`, `redis`, `storage`, and `worker`.
# Readiness
curl http://localhost:8000/health/ready
# → {"status":"ready","checks":{"database":"ok","redis":"ok","storage":"ok"}}
# Full health
curl http://localhost:8000/api/v1/health
# → {"status":"healthy","version":"1.0.0","checks":{...}}
```
### Prometheus Metrics
@@ -135,6 +201,15 @@ Available metrics:
- `leocrm_db_pool_connections` — Database connection pool size
- `leocrm_arq_jobs_total` — Total ARQ background jobs
### System Dashboard
Admin-only dashboard at `/system-dashboard` in the WebUI. Shows:
- System Health, DB Stats, Redis Stats, Worker Queue
- API Stats (total requests, error rate, avg response time)
- Plugin Stats (discovered, active)
- Storage Stats (disk usage, file count)
- Alert Feed (system messages from Communication-System)
### Structured Logging
LeoCRM uses `structlog` for structured JSON logging. All API requests are logged with:
@@ -199,41 +274,73 @@ leocrm/
├── app/
│ ├── main.py # FastAPI entry point with logging middleware
│ ├── config.py # Pydantic settings
│ ├── deps.py # FastAPI dependencies (auth, permissions)
│ ├── core/
│ │ ├── monitoring.py # Prometheus metrics + structured logging + health checks
│ │ ├── db.py # Async database engine
│ │ ├── middleware.py # CSRF middleware
│ │ ├── worker.py # ARQ worker settings
│ │ ├── backup_job.py # Automated backup job
│ │ ├── notifications.py # System notification dispatch
│ │ └── ...
│ ├── routes/
│ │ ├── health.py # Health endpoint
│ │ ├── health.py # Health endpoints
│ │ ├── metrics.py # Prometheus metrics endpoint (admin-only)
│ │ ├── system_dashboard.py # System dashboard (admin-only)
│ │ ├── system_settings.py # System settings + backup config
│ │ ├── audit.py # Audit log (list, export, retention)
│ │ ├── contacts.py # Contact CRUD + streaming CSV export
│ │ ├── companies.py # Company CRUD + streaming CSV export
│ │ ├── workflows.py # Workflow engine routes
│ │ └── ...
│ ├── models/ # SQLAlchemy models
│ ├── schemas/ # Pydantic schemas
│ ├── services/ # Business logic
│ └── plugins/ # Plugin system
│ ├── plugins/ # Plugin system (registry, manifest, base)
│ │ └── builtins/ # 23 built-in plugins
│ ├── workflows/ # Workflow engine
│ └── ai/ # AI modules
├── scripts/
│ ├── fast-deploy.sh # Frontend-only / full deploy
│ ├── deploy.py # Coolify API deployment
│ ├── backup.py # Backup script (pg_dump + files)
│ ├── restore.py # Restore script
│ ├── seed_perf_data.py # Performance test data seeding
│ └── check_indexes.py # Database index verification
├── tests/ # Test suite (pytest + pytest-asyncio)
├── docs/
│ ├── admin-guide.md # Admin guide (deploy, backup, restore, troubleshooting)
│ └── api-overview.md # API endpoint summary
├── alembic/ # Database migrations
│ ├── api-documentation.md # Full API endpoint reference
│ ├── monitoring.md # Monitoring & health checks
│ ├── infrastructure.md # Infrastructure guide
│ ├── deploy-guide.md # Deploy guide (fast-deploy, Coolify, server info)
│ └── ...
├── alembic/ # Database migrations (130+ files)
├── frontend/ # React + TypeScript + Vite + Tailwind
│ └── src/pages/ # SystemDashboard, Contacts, Mail, DMS, Calendar, etc.
├── requirements.txt # Production dependencies
├── requirements-dev.txt # Test/lint dependencies
├── .env.example # Environment template
├── docker-compose.yml # Docker Compose
├── docker-compose.yaml # Docker Compose (postgres, redis, crm_app, crm_worker)
├── Dockerfile # Multi-stage build (frontend → builder → runtime)
├── prestart.sh # Container entrypoint (migrations, seed, uvicorn)
├── worker.sh # ARQ worker entrypoint
├── healthcheck.sh # Container healthcheck
└── README.md # This file
```
## Documentation
- [Admin Guide](docs/admin-guide.md) — Deployment, backup, restore, env vars, troubleshooting
- [API Overview](docs/api-overview.md) — Full endpoint reference
- [Coolify Setup](COOLIFY_SETUP.md) — Coolify deployment instructions
- [API Documentation](docs/api-documentation.md) — Full endpoint reference (300+ endpoints)
- [Monitoring](docs/monitoring.md) — Health checks, metrics, system dashboard, alerting
- [Infrastructure](docs/infrastructure.md) — Docker, PgBouncer, audit partitioning, backup
- [Deploy Guide](docs/deploy-guide.md) — Fast-deploy, Coolify API, server info
- [Plugin Development](docs/plugin-development-guide.md) — Plugin development guide
- [Security Kernel](docs/security_kernel.md) — ABAC, RLS, session security
- [Permissions](docs/permissions.md) — Permission system documentation
- [Test Strategy](docs/test-strategy.md) — Test conventions and constraints
- [UI Design Guidelines](docs/ui-design-guidelines.md) — UI design rules
- [Swagger UI](http://localhost:8000/docs) — Interactive API docs (auto-generated)
## License
docs/admin-guide.md
+242 -19
View File
@@ -1,16 +1,20 @@
# LeoCRM Admin Guide
> Operations manual for LeoCRM administrators: deployment, backup, restore, environment configuration, and troubleshooting.
> Operations manual for LeoCRM administrators: deployment, backup, restore, environment configuration, monitoring, and troubleshooting.
## Table of Contents
1. [Deployment](#deployment)
2. [Environment Configuration](#environment-configuration)
3. [Environment Profiles](#environment-profiles)
4. [Backup](#backup)
5. [Restore](#restore)
6. [Monitoring](#monitoring)
7. [Troubleshooting](#troubleshooting)
4. [System Dashboard](#system-dashboard)
5. [Backup](#backup)
6. [Restore](#restore)
7. [Audit Log](#audit-log)
8. [Trash Cleanup](#trash-cleanup)
9. [Monitoring](#monitoring)
10. [Incident Response](#incident-response)
11. [Troubleshooting](#troubleshooting)
---
@@ -19,7 +23,7 @@
### Prerequisites
- Docker 24+ and Docker Compose v2
- PostgreSQL 15+ (or use the included Docker container)
- PostgreSQL 16+ (or use the included Docker container with pgvector)
- Redis 7+ (or use the included Docker container)
- A Coolify instance (for managed deployment) or a VPS with Docker
@@ -67,11 +71,11 @@
### Coolify Deployment
See [COOLIFY_SETUP.md](../COOLIFY_SETUP.md) for detailed Coolify deployment instructions.
See [deploy-guide.md](deploy-guide.md) for detailed Coolify deployment instructions.
### Manual Deployment (without Docker)
1. Install Python 3.11+ and PostgreSQL 15+
1. Install Python 3.12+ and PostgreSQL 16+
2. Create a virtual environment: `python3 -m venv .venv && source .venv/bin/activate`
3. Install dependencies: `pip install -r requirements.txt`
4. Configure `.env` (see [Environment Configuration](#environment-configuration))
@@ -183,9 +187,92 @@ LeoCRM supports three environment profiles via the `ENVIRONMENT` variable.
---
## System Dashboard
Das System Dashboard ist die zentrale Monitoring-Oberfläche für Administratoren.
### Zugriff
- **Frontend:** `/system-dashboard` (Admin-only, Sidebar-Eintrag nur für Admins sichtbar)
- **API:** `GET /api/v1/system/dashboard` (Admin-only)
- **Alerts:** `GET /api/v1/system/alerts` (Admin-only)
### Dashboard-Inhalte
| Bereich | Metriken |
|---------|----------|
| **System Health** | Overall status (healthy/degraded/down) |
| **Database** | Connections, Table Count, DB Size |
| **Redis** | Connected Clients, Used Memory, Peak Memory, Uptime |
| **Worker** | Queue Length, Active Workers, Status |
| **API Stats** | Total Requests, Error Count, Error Rate, Avg Response Time |
| **Plugins** | Total Discovered, Active Plugins |
| **Storage** | Disk Usage, File Count, Status |
| **Alert Feed** | System Messages aus Communication-System |
### Alert-Dispatch
Bei Problemen (DB down, Redis down, Worker down, High Error Rate) sendet das System Dashboard automatisch eine System-Message an das Communication-System. Diese erscheint im Alert-Feed und in den Benachrichtigungen der Admins.
```bash
# Dashboard abfragen
curl -b "leocrm_session=<session>" https://crm.media-on.de/api/v1/system/dashboard | jq .
# Aktive Alerts abfragen
curl -b "leocrm_session=<session>" https://crm.media-on.de/api/v1/system/alerts | jq .
```
Siehe auch [monitoring.md](monitoring.md) für Details zu Health Endpoints und Metrics.
---
## Backup
### Database Backup
### Backup-Konfiguration (Settings → Backup)
Die Backup-Konfiguration erfolgt über die System-Settings API oder das Settings-UI.
| Einstellung | Default | Beschreibung |
|-------------|---------|-------------|
| `backup_enabled` | `false` | Backup Automation aktivieren |
| `backup_interval` | `24h` | Backup-Intervall (Cron-Schedule) |
| `backup_retention_days` | `7` | Aufbewahrung in Tagen |
| `backup_destination` | `local` | Backup-Ziel (local, s3, nextcloud) |
#### API-Endpunkte
| Method | Path | Beschreibung |
|--------|------|-------------|
| GET | `/api/v1/system-settings/backup-config` | Backup-Konfiguration abfragen |
| PUT | `/api/v1/system-settings/backup-config` | Backup-Konfiguration aktualisieren |
| POST | `/api/v1/system-settings/backup-now` | Sofortiges Backup auslösen (ARQ-Job) |
| GET | `/api/v1/system-settings/backup-history` | Letzte 10 Backup-Ergebnisse (Audit Log) |
```bash
# Backup-Konfiguration abfragen
curl -b "leocrm_session=<session>" https://crm.media-on.de/api/v1/system-settings/backup-config | jq .
# Backup aktivieren und konfigurieren
curl -X PUT -b "leocrm_session=<session>" \
-H "Content-Type: application/json" \
-d '{"backup_enabled": true, "backup_interval": "24h", "backup_retention_days": 7, "backup_destination": "local"}' \
https://crm.media-on.de/api/v1/system-settings/backup-config
# Sofortiges Backup auslösen
curl -X POST -b "leocrm_session=<session>" https://crm.media-on.de/api/v1/system-settings/backup-now
# Backup-Historie abfragen
curl -b "leocrm_session=<session>" https://crm.media-on.de/api/v1/system-settings/backup-history | jq .
```
### Automated Backup (ARQ Cron-Job)
- **Job:** `auto_backup_job` (täglich 03:00 Uhr, einstellbar)
- **Script:** `scripts/backup.py` (pg_dump + files)
- **Bei Fehler:** System-Message an Communication-System + Audit Log Eintrag
- **Bei Erfolg:** Audit Log Eintrag (`backup_success`)
### Manual Database Backup
```bash
# Full database dump (recommended daily)
@@ -195,14 +282,6 @@ pg_dump -U leocrm -h localhost leocrm > backup_$(date +%Y%m%d).sql
pg_dump -U leocrm -h localhost leocrm | gzip > backup_$(date +%Y%m%d).sql.gz
```
### Automated Backup (Cron)
Add to crontab for daily backup at 2 AM:
```cron
0 2 * * * pg_dump -U leocrm -h localhost leocrm | gzip > /backups/leocrm_$(date +\%Y\%m\%d).sql.gz
```
### File Storage Backup
```bash
@@ -248,14 +327,120 @@ cp /backups/redis_20260101.rdb /var/lib/redis/dump.rdb
systemctl start redis
```
See `scripts/restore.py` for the automated restore solution and `scripts/restore_test.sh` for restore testing.
---
## Audit Log
### Audit Log Export (CSV/JSON)
Audit-Logs können als CSV oder JSON exportiert werden. Der Export erfolgt als Streaming-Response (max. 10.000 Einträge).
```bash
# CSV Export
curl -b "leocrm_session=<session>" \
"https://crm.media-on.de/api/v1/audit-log/export?format=csv" \
-o audit_log_export.csv
# JSON Export
curl -b "leocrm_session=<session>" \
"https://crm.media-on.de/api/v1/audit-log/export?format=json" \
-o audit_log_export.json
# Gefilterter Export (nach Entity-Type und Datum)
curl -b "leocrm_session=<session>" \
"https://crm.media-on.de/api/v1/audit-log/export?format=csv&entity_type=contact&date_from=2026-01-01&date_to=2026-12-31" \
-o audit_log_contacts_2026.csv
```
### Audit Log Retention (365 Tage)
Audit-Logs werden standardmäßig nach 365 Tagen archiviert/gelöscht. Die Retention ist einstellbar.
- **Default:** 365 Tage
- **ARQ-Cron-Job:** `audit_retention_cleanup` (täglich 04:00 Uhr)
- **API:** `DELETE /api/v1/audit-log/retention?retention_days=365`
```bash
# Manuelle Retention-Bereinigung (Admin-only)
curl -X DELETE -b "leocrm_session=<session>" \
"https://crm.media-on.de/api/v1/audit-log/retention?retention_days=365"
# → {"deleted": 1234, "retention_days": 365, "cutoff": "2025-08-20T00:00:00"}
```
### Audit Log Abfrage
```bash
# Audit-Logs abfragen (mit Filter und Pagination)
curl -b "leocrm_session=<session>" \
"https://crm.media-on.de/api/v1/audit-log?entity_type=contact&action=create&page=1&page_size=50" | jq .
```
**Filter-Parameter:**
- `entity_type` — Filter nach Entity-Typ
- `user_id` — Filter nach User-ID
- `action` — Filter nach Action (create/update/delete/login)
- `date_from` — ISO Datum-Start (inklusive)
- `date_to` — ISO Datum-Ende (inklusive)
- `page` / `page_size` — Pagination (max. 200 pro Seite)
### Tamper-Proof
- DELETE auf AuditLog nur mit `?gdpr=true` + Admin-Berechtigung
- Audit-Logs können nicht modifiziert werden (nur erstellen und lesen)
---
## Trash Cleanup
Soft-deleted Entitäten werden nach Ablauf der Retention-Periode endgültig gelöscht.
- **Default:** 90 Tage (`trash_retention_days`)
- **ARQ-Cron-Job:** `cleanup_expired_trash` (täglich 05:00 Uhr)
- **Kriterium:** `deleted_at < now() - retention_days`
- **Bei Löschung:** Audit-Log Eintrag
### Konfiguration
Die Retention-Periode ist über die System-Settings einstellbar:
```bash
# Trash-Retention konfigurieren
curl -X PUT -b "leocrm_session=<session>" \
-H "Content-Type: application/json" \
-d '{"trash_retention_days": 90}' \
https://crm.media-on.de/api/v1/system-settings
```
### Hard-Delete (GDPR)
Einzelne Entitäten können sofort endgültig gelöscht werden (Admin-only):
```bash
# Hard-Delete mit GDPR-Flag
curl -X DELETE -b "leocrm_session=<session>" \
"https://crm.media-on.de/api/v1/contacts/{id}?gdpr=true"
```
---
## Monitoring
### Health Endpoint
### Health Endpoints
```bash
# Liveness
curl http://localhost:8000/health/live
# → {"status":"alive"}
# Readiness
curl http://localhost:8000/health/ready
# → {"status":"ready","checks":{"database":"ok","redis":"ok","storage":"ok"}}
# Full health
curl http://localhost:8000/api/v1/health
# → {"status":"healthy","version":"1.0.0","checks":{...}}
```
Returns JSON with overall status and individual checks:
@@ -305,6 +490,33 @@ python scripts/seed_perf_data.py --count 200000
python scripts/check_indexes.py
```
Siehe [monitoring.md](monitoring.md) für vollständige Monitoring-Dokumentation.
---
## Incident Response
Siehe `docs/incident-response-runbook.md` für detaillierte Notfall-Prozeduren.
### Schnell-Referenz
| Incident | Erste Maßnahme | Eskalation |
|----------|---------------|-----------|
| **Server-Ausfall** | Coolify Restart → Health-Check → System-Message | Hetzner Support |
| **DB-Crash** | PostgreSQL Restart → Migration-Check → Backup-Restore | Coolify DB Restart |
| **Redis-Crash** | Redis Restart → Session-Check | Coolify Service Restart |
| **Security-Breach** | Logs prüfen → Password-Reset → Audit-Log Export | Incident Response Team |
| **Backup Failed** | Backup-Log prüfen → Manueller Retry → Storage prüfen | Admin |
| **Worker Down** | Worker-Container Restart → Queue prüfen | Coolify Service Restart |
### Incident Response Schritte
1. **Erkennen** — Alert im System Dashboard oder externes Monitoring
2. **Eingrenzen** — Health-Checks, Logs, Metrics prüfen
3. **Beheben** — Restart, Restore, Konfiguration anpassen
4. **Verifizieren** — Health-Check grün, System Dashboard ok
5. **Dokumentieren** — Audit Log Eintrag, Post-Mortem bei Critical
---
## Troubleshooting
@@ -340,7 +552,8 @@ python scripts/check_indexes.py
1. Verify ARQ worker is running: `ps aux | grep arq`
2. Start worker: `arq app.core.jobs.WorkerSettings`
3. Check Redis queue: `redis-cli LLEN arq:queue`
3. Check Redis queue: `redis-cli ZCARD arq:queue`
4. Check System Dashboard: `GET /api/v1/system/dashboard` → `.worker`
### Performance Issues
@@ -378,3 +591,13 @@ python scripts/check_indexes.py
3. Review migration files in `alembic/versions/`
4. Check current revision: `alembic current`
5. Reset (DESTRUCTIVE): `alembic downgrade base && alembic upgrade head`
### Backup Issues
**Symptom**: Backup job fails, no backup history
1. Check System Dashboard for backup alerts
2. Check Audit Log for `backup_failed` entries: `GET /api/v1/audit-log?action=backup_failed`
3. Verify storage path has enough disk space
4. Try manual backup: `POST /api/v1/system-settings/backup-now`
5. Check `scripts/backup.py` logs
docs/api-documentation.md
+17 -4
View File
@@ -1,6 +1,6 @@
# LeoCRM API Documentation
> Auto-generated from FastAPI route enumeration. **295 endpoints** across **30 tag groups**.
> Auto-generated from FastAPI route enumeration. **303 endpoints** across **31 tag groups**.
## Overview
@@ -228,12 +228,16 @@ List endpoints use `page` (1-based) and `page_size` (1-100) query parameters. Re
| PATCH | `/api/v1/sequences/{sequence_id}` | Update a sequence. |
| DELETE | `/api/v1/sequences/{sequence_id}` | Delete a sequence. |
### system-settings (2 endpoints)
### system-settings (6 endpoints)
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/v1/system-settings` | Get system settings. **Response**: `SystemSettingsResponse` |
| PUT | `/api/v1/system-settings` | Upsert system settings. **Request**: `SystemSettingsUpsert`, **Response**: `SystemSettingsResponse` |
| GET | `/api/v1/system-settings/backup-config` | Get backup configuration (backup_enabled, backup_interval, backup_retention_days, backup_destination). |
| PUT | `/api/v1/system-settings/backup-config` | Update backup configuration. Admin only. |
| POST | `/api/v1/system-settings/backup-now` | Trigger an immediate backup via ARQ job. Admin only. Returns `{"message": "Backup job enqueued", "job_id": "..."}`. |
| GET | `/api/v1/system-settings/backup-history` | Get last 10 backup results from audit log. Returns `{"history": [...]}`. |
### attachments (4 endpoints)
@@ -253,11 +257,20 @@ List endpoints use `page` (1-based) and `page_size` (1-100) query parameters. Re
| PATCH | `/api/v1/addresses/{address_id}` | Update an address. |
| DELETE | `/api/v1/addresses/{address_id}` | Delete an address. |
### audit (1 endpoint)
### audit (4 endpoints)
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/v1/audit-log` | Query audit log entries. |
| GET | `/api/v1/audit-log` | Query audit log entries (filterable, paginated, admin-only). |
| GET | `/api/v1/audit-log/export` | Export audit log as CSV or JSON (streaming, max 10.000 entries). **Query**: `format` (csv/json), `entity_type`, `action`, `date_from`, `date_to` |
| DELETE | `/api/v1/audit-log/retention` | Delete audit log entries older than retention_days (default 365, admin-only). **Query**: `retention_days` (1-3650) |
### system (2 endpoints)
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/v1/system/dashboard` | Comprehensive system dashboard (admin-only). Returns: health, DB stats, Redis stats, worker stats, API stats, plugin stats, storage stats. |
| GET | `/api/v1/system/alerts` | Active system alerts (admin-only). Returns alerts from Communication-System. |
---
docs/deploy-guide.md
+34 -9
View File
@@ -1,5 +1,8 @@
# LeoCRM Deploy Guide
> **Version:** 2.0
> **Date:** 2026-08-20
## Fast Frontend-Only Deploy (~20s)
```bash
bash /a0/usr/projects/leocrm/scripts/fast-deploy.sh frontend
@@ -27,10 +30,12 @@ bash /a0/usr/projects/leocrm/scripts/fast-deploy.sh full
3. Dann deploy
## Container-Info
- Coolify App UUID: xf7smknlger3 (neu erstellt 2026-08-06)
- Coolify App UUID: xf7smknlger3hvkrsb910tui (neu erstellt 2026-08-06)
- Container-Name aendert sich bei jedem Coolify-Deploy (Suffix)
- Frontend-Pfad im Container: /app/frontend/dist
- Worker: Teil der Docker-Compose-App (crm_worker service)
- DB: Teil der Docker-Compose-App (postgres service, pgvector/pgvector:pg16)
- Redis: Teil der Docker-Compose-App (redis service, redis:7-alpine)
## Server
- Host: 46.225.91.159
@@ -43,7 +48,7 @@ bash /a0/usr/projects/leocrm/scripts/fast-deploy.sh full
- Produktions-DB: postgresql+asyncpg://crm_user:86FkF5vJ_qKYgO6Myj0eQ4Dtm3Dyb1ge@postgres:5432/crm_db
- Redis: redis://default:6VJ7pp8afXXZMnx0JztWFk-OYCLwJfX4@redis:6379/0
- SECRET_KEY: DoYnyh_UnvnYphX-qryiaIpQhm8JB39m_xkat9cNmrGpyKSSZvW9jF1tusIUSP5g
- MAIL_ENCRYPTION_KEY: test-mail-encryption-key (AES-256 Fernet Key für Mail-Passwort-Verschlüsselung, in Coolify als Environment Variable setzen)
- MAIL_ENCRYPTION_KEY: test-mail-encryption-key (AES-256 Fernet Key fuer Mail-Passwort-Verschluesselung, in Coolify als Environment Variable setzen)
## Coolify Resources
- Project UUID: mzu7fvhtad82ujgmbsmyvxzm
@@ -52,13 +57,13 @@ bash /a0/usr/projects/leocrm/scripts/fast-deploy.sh full
## Production Resource Recommendations
Die docker-compose.yaml hat Development-Defaults (PostgreSQL 512m, Redis 128m). Für Produktion mit 50+ Usern, 100k+ Entities, Agenten, Search und Knowledge müssen die Limits erhöht werden.
Die docker-compose.yaml hat Development-Defaults (PostgreSQL 512m, Redis 128m). Fuer Produktion mit 50+ Usern, 100k+ Entities, Agenten, Search und Knowledge muessen die Limits erhoeht werden.
| Service | Development | Production (50+ User) | Begründung |
| Service | Development | Production (50+ User) | Begruendung |
|---|---|---|---|
| **PostgreSQL RAM** | 512m | 1-2GB | pgvector HNSW + FTS + JSONB Snapshots + Outbox + AuditLog |
| **PostgreSQL CPU** | 1.0 | 2.0 | Vector Search + FTS + normale CRM-Queries |
| **PostgreSQL Disk** | Named Volume | 50-100GB | Embeddings (768 dim × 100k = ~300MB), JSONB Snapshots, Outbox |
| **PostgreSQL Disk** | Named Volume | 50-100GB | Embeddings (768 dim x 100k = ~300MB), JSONB Snapshots, Outbox |
| **Redis RAM** | 128m | 256-512m | WS Pub/Sub + Caching + Sessions + ARQ + Rate-Limiting |
| **Redis CPU** | 0.5 | 1.0 | Pub/Sub + Cache + Queue |
| **App (FastAPI) RAM** | Nicht limitiert | 512m-1GB | WebSocket Connections + Async Tasks |
@@ -68,9 +73,9 @@ Die docker-compose.yaml hat Development-Defaults (PostgreSQL 512m, Redis 128m).
### Skalierung bei Bedarf
- **Read-Replicas:** Bei hohem Lese-Aufkommen (Search, FTS, Vector) können Read-Replicas für PostgreSQL eingerichtet werden. Schreib-Last (Outbox, EntityHistory, AuditLog) bleibt auf dem Master.
- **Mehr Worker:** Bei hohem Background-Job-Aufkommen können zusätzliche ARQ-Worker-Container gestartet werden. Queue-Prioritäten verhindern dass wichtige Jobs hinter langen Index-Jobs warten.
- **pgvector auslagern:** Bei sehr großen Datasets (>1M Embeddings) kann pgvector auf einen separaten PostgreSQL-Node ausgelagert werden.
- **Read-Replicas:** Bei hohem Lese-Aufkommen (Search, FTS, Vector) koennen Read-Replicas fuer PostgreSQL eingerichtet werden. Schreib-Last (Outbox, EntityHistory, AuditLog) bleibt auf dem Master.
- **Mehr Worker:** Bei hohem Background-Job-Aufkommen koennen zusaetzliche ARQ-Worker-Container gestartet werden. Queue-Prioritaeten verhindern dass wichtige Jobs hinter langen Index-Jobs warten.
- **pgvector auslagern:** Bei sehr grossen Datasets (>1M Embeddings) kann pgvector auf einen separaten PostgreSQL-Node ausgelagert werden.
- **Redis Cluster:** Bei sehr hohem Cache-/Pub/Sub-Aufkommen kann Redis Cluster eingesetzt werden.
### LLM-Kosten-Management
@@ -78,4 +83,24 @@ Die docker-compose.yaml hat Development-Defaults (PostgreSQL 512m, Redis 128m).
- **Cost-Tracking:** Der zentrale LLM Client (Phase B.1) trackt Kosten pro Call.
- **Budget-Limits:** Pro Agent (Phase F) und pro Tenant (Phase I).
- **Cost-Dashboard:** Phase I zeigt LLM-Kosten pro Agent/Workflow/User.
- **Alerts:** Budget-Alerts bei Überschreitung konfigurierbarer Schwellwerte.
- **Alerts:** Budget-Alerts bei Ueberschreitung konfigurierbarer Schwellwerte.
## ARQ Worker Cron-Jobs
Der crm_worker Service fuehrt folgende Cron-Jobs aus:
| Job | Schedule | Beschreibung |
|-----|----------|-------------|
| `auto_backup_job` | Taeglich 03:00 | Automatisches Backup (ruft scripts/backup.py auf) |
| `audit_retention_cleanup` | Taeglich 04:00 | Audit-Logs aelter als 365 Tage archivieren/loeschen |
| `cleanup_expired_trash` | Taeglich 05:00 | Soft-deleted Entities aelter als 90 Tage endgueltig loeschen |
| `outbox_cleanup` | Stuendlich | Outbox-Eintraege aelter als 30 Tage loeschen |
| `cleanup_expired_sessions` | Stuendlich | Abgelaufene Sessions aus Redis loeschen |
## Container-Entrypoints
| Datei | Service | Funktion |
|------|---------|----------|
| `prestart.sh` | crm_app | Alembic-Migrationen -> DB-Role-Passwoerter -> Plugin-Schema-Sync -> Admin-Seed -> uvicorn Start |
| `worker.sh` | crm_worker | ARQ Worker Start mit Cron-Jobs |
| `healthcheck.sh` | crm_app | HTTP /api/v1/health oder Redis-Ping |
docs/infrastructure.md
+237 -116
View File
@@ -1,22 +1,192 @@
# Infrastructure Guide
> **Version:** 1.0
> **Date:** 2026-07-29
> **Version:** 2.0
> **Date:** 2026-08-20
> **Applies to:** System administrators and DevOps engineers
---
## 1. PgBouncer Setup
## 1. Docker-Compose Stack
PgBouncer is a lightweight connection pooler for PostgreSQL. It reduces the overhead of establishing new database connections by reusing existing ones.
LeoCRM läuft als Docker-Compose-Stack mit 4 Services:
### Why PgBouncer?
| Service | Image | Beschreibung |
|---------|-------|-------------|
| **postgres** | `pgvector/pgvector:pg16` | PostgreSQL 16 mit pgvector Extension für Vector Search |
| **redis** | `redis:7-alpine` | Redis 7 für Sessions, Caching, Pub/Sub, ARQ Queue |
| **crm_app** | Multi-Stage Build | FastAPI Backend (uvicorn), API + WebSocket |
| **crm_worker** | Multi-Stage Build | ARQ Background Worker (Cron-Jobs, Queue Processing) |
- **Connection pooling** — Reduces PostgreSQL connection overhead
- **Resource efficiency** — Handles thousands of client connections with minimal resources
- **Transaction pooling** — Best for stateless applications like FastAPI
- **Session pooling** — For stateful connections
- **Statement pooling** — For specific use cases
### docker-compose.yaml Übersicht
```yaml
services:
postgres:
image: pgvector/pgvector:pg16
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U crm_user -d crm_db"]
interval: 10s
timeout: 5s
retries: 5
redis:
image: redis:7-alpine
volumes:
- redis_data:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 10s
timeout: 5s
retries: 5
crm_app:
build: .
command: ["./prestart.sh"]
depends_on:
postgres: { condition: service_healthy }
redis: { condition: service_healthy }
healthcheck:
test: ["CMD-SHELL", "bash /app/healthcheck.sh"]
interval: 30s
timeout: 10s
retries: 3
start_period: 15s
crm_worker:
build: .
command: ["./worker.sh"]
depends_on:
postgres: { condition: service_healthy }
redis: { condition: service_healthy }
```
### Container-Entrypoints
| Datei | Service | Funktion |
|------|---------|----------|
| `prestart.sh` | crm_app | Alembic-Migrationen → DB-Role-Passwörter → Plugin-Schema-Sync → Admin-Seed → uvicorn Start |
| `worker.sh` | crm_worker | ARQ Worker Start mit Cron-Jobs |
| `healthcheck.sh` | crm_app | HTTP `/api/v1/health` oder Redis-Ping |
### Volumes
| Volume | Service | Beschreibung |
|--------|---------|-------------|
| `postgres_data` | postgres | PostgreSQL Daten (persistent) |
| `redis_data` | redis | Redis Snapshot (persistent) |
| `app_storage` | crm_app | File Storage (DMS, Uploads) |
---
## 2. Coolify Deployment
LeoCRM ist über Coolify auf einem Hetzner VPS deployiert.
### Server-Info
| Eigenschaft | Wert |
|-------------|------|
| **Host** | 46.225.91.159 |
| **Hostname** | coolify-01 |
| **Provider** | Hetzner VPS |
| **Coolify URL** | https://server.media-on.de |
| **App URL** | https://crm.media-on.de |
| **App UUID** | xf7smknlger3hvkrsb910tui |
| **Container-Name** | Ändert sich bei jedem Coolify-Deploy (Suffix) |
### Deploy-Methoden
#### Frontend-Only Deploy (~20s)
```bash
bash /a0/usr/projects/leocrm/scripts/fast-deploy.sh frontend
```
- Baut Frontend lokal, kopiert `dist/` direkt in den laufenden Container
- Kein Coolify-Rebuild, kein Docker-Image-Neubau
- Container wird nicht neu gestartet
#### Full Deploy (~2min, für Backend-Änderungen)
```bash
bash /a0/usr/projects/leocrm/scripts/fast-deploy.sh full
```
- Triggert Coolify-Rebuild über `deploy.py`
- Für Python-Code, Requirements, Migrations
### Server-Container
Auf dem Hetzner VPS laufen ~25 Docker-Container (Coolify + Services):
- Coolify Proxy (Traefik)
- Coolify Dashboard
- Coolify Database
- LeoCRM Stack (postgres, redis, crm_app, crm_worker)
- Weitere Coolify-managed Services
Container-Übersicht auf dem Server:
```bash
ssh -i ~/.ssh/coolify-01-root root@46.225.91.159 'docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"'
```
---
## 3. ARQ Worker & Cron-Jobs
Der `crm_worker` Service läuft ARQ (Async Redis Queue) für Background-Jobs.
### Registrierte Cron-Jobs
| Job | Schedule | Beschreibung |
|-----|----------|-------------|
| `auto_backup_job` | Täglich 03:00 | Automatisches Backup (ruft `scripts/backup.py` auf) |
| `audit_retention_cleanup` | Täglich 04:00 | Audit-Logs älter als 365 Tage archivieren/löschen |
| `cleanup_expired_trash` | Täglich 05:00 | Soft-deleted Entitäten älter als 90 Tage endgültig löschen |
| `outbox_cleanup` | Stündlich | Outbox-Einträge älter als 30 Tage löschen |
| `cleanup_expired_sessions` | Stündlich | Abgelaufene Sessions aus Redis löschen |
### ARQ Worker Konfiguration
```python
# app/core/worker.py
class WorkerSettings:
functions = [...]
cron_jobs = [
cron(auto_backup_job, hour=3, minute=0),
cron(audit_retention_cleanup, hour=4, minute=0),
cron(cleanup_expired_trash, hour=5, minute=0),
cron(outbox_cleanup, hour={0,6,12,18}, minute=0),
cron(cleanup_expired_sessions, hour={0,6,12,18}, minute=0),
]
max_jobs = 10
job_timeout = 300
queue_name = "arq:queue"
```
### Worker-Stats abfragen
```bash
# Queue-Länge
redis-cli ZCARD arq:queue
# Aktive Worker
redis-cli KEYS "arq:heartbeat:*"
# Via API (Admin)
curl -b "leocrm_session=<session>" https://crm.media-on.de/api/v1/system/dashboard | jq '.worker'
```
---
## 4. PgBouncer Setup
PgBouncer ist ein leichter Connection-Pooler für PostgreSQL. Er reduziert den Overhead neuer Datenbankverbindungen durch Wiederverwendung bestehender Verbindungen.
### Warum PgBouncer?
- **Connection pooling** — Reduziert PostgreSQL Connection-Overhead
- **Resource efficiency** — Verwaltet tausende Client-Verbindungen mit minimalen Ressourcen
- **Transaction pooling** — Optimal für stateless Anwendungen wie FastAPI
- **Session pooling** — Für stateful Verbindungen
- **Statement pooling** — Für spezifische Use-Cases
### Installation
@@ -28,7 +198,7 @@ apt-get update && apt-get install -y pgbouncer
pgbouncer --version
```
### Configuration
### Konfiguration
Create `/etc/pgbouncer/pgbouncer.ini`:
@@ -43,8 +213,6 @@ listen_port = 6432
unix_socket_dir = /var/run/pgbouncer
# Authentication
# Use md5 for password-based auth
# Use trust for local development
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt
@@ -67,7 +235,6 @@ log_pooler_errors = 1
stats_period = 60
# Security
# Only allow connections from localhost and Docker network
listen_backlog = 128
```
@@ -167,25 +334,25 @@ echo "SHOW SERVERS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer
---
## 2. Audit Log Partitioning
## 5. Audit Log Partitioning
The `audit_log` table can grow very large over time. PostgreSQL table partitioning helps manage this by splitting the table into smaller, more manageable pieces.
Die `audit_log` Tabelle kann sehr groß werden. PostgreSQL Table Partitioning hilft durch Aufteilung in kleinere, verwaltbare Stücke.
### Why Partition?
### Warum Partitioning?
- **Faster queries** — Queries only scan relevant partitions
- **Easier maintenance** — Drop old partitions instead of DELETE
- **Better vacuum** — Each partition is vacuumed independently
- **Improved performance** — Smaller indexes per partition
- **Schnellere Queries** — Queries scannen nur relevante Partitionen
- **Einfachere Wartung** — Alte Partitionen droppen statt DELETE
- **Besseres Vacuum** — Jede Partition wird unabhängig gevacuumt
- **Bessere Performance** — Kleinere Indexes pro Partition
### Partitioning Strategy
### Partitioning-Strategie
We use **monthly range partitioning** on the `created_at` column:
Wir verwenden **monatliches Range-Partitioning** auf der `created_at` Spalte:
```sql
-- Each partition covers one month
-- Partition name: audit_log_YYYY_MM
-- Example: audit_log_2026_01, audit_log_2026_02, ...
-- Jede Partition deckt einen Monat ab
-- Partitions-Name: audit_log_YYYY_MM
-- Beispiel: audit_log_2026_01, audit_log_2026_02, ...
```
### Creating the Partitioned Table
@@ -221,46 +388,6 @@ CREATE INDEX idx_audit_log_2026_01_tenant ON audit_log_2026_01 (tenant_id);
CREATE INDEX idx_audit_log_2026_01_action ON audit_log_2026_01 (action);
CREATE INDEX idx_audit_log_2026_01_entity ON audit_log_2026_01 (entity_type, entity_id);
CREATE INDEX idx_audit_log_2026_01_created ON audit_log_2026_01 (created_at DESC);
CREATE INDEX idx_audit_log_2026_02_tenant ON audit_log_2026_02 (tenant_id);
CREATE INDEX idx_audit_log_2026_02_action ON audit_log_2026_02 (action);
CREATE INDEX idx_audit_log_2026_02_entity ON audit_log_2026_02 (entity_type, entity_id);
CREATE INDEX idx_audit_log_2026_02_created ON audit_log_2026_02 (created_at DESC);
CREATE INDEX idx_audit_log_2026_03_tenant ON audit_log_2026_03 (tenant_id);
CREATE INDEX idx_audit_log_2026_03_action ON audit_log_2026_03 (action);
CREATE INDEX idx_audit_log_2026_03_entity ON audit_log_2026_03 (entity_type, entity_id);
CREATE INDEX idx_audit_log_2026_03_created ON audit_log_2026_03 (created_at DESC);
```
### Migrating Existing Data
```sql
-- Step 1: Create the partitioned table
-- (see script above)
-- Step 2: Insert existing data
INSERT INTO audit_log_partitioned (
id, tenant_id, user_id, action, entity_type,
entity_id, changes, ip_address, user_agent, created_at
)
SELECT id, tenant_id, user_id, action, entity_type,
entity_id, changes, ip_address, user_agent, created_at
FROM audit_log;
-- Step 3: Verify data integrity
SELECT COUNT(*) FROM audit_log_partitioned;
SELECT COUNT(*) FROM audit_log;
-- Step 4: Rename tables
ALTER TABLE audit_log RENAME TO audit_log_old;
ALTER TABLE audit_log_partitioned RENAME TO audit_log;
-- Step 5: Update sequences and indexes
-- (handled by the partitioned table definition)
-- Step 6: Drop old table after verification
-- DROP TABLE audit_log_old;
```
### Automating Partition Creation
@@ -274,54 +401,19 @@ psql -h localhost -U leocrm -d crm_db -f scripts/setup_audit_partitioning.sql
### Cron Job for Partition Maintenance
Add to crontab to create partitions automatically:
```bash
# Run on the 1st of each month at 2 AM
0 2 1 * * /usr/bin/psql -h localhost -U leocrm -d crm_db -c "SELECT create_monthly_audit_partition();"
```
### Querying Partitioned Data
```sql
-- Query a specific month (fast, only scans one partition)
SELECT * FROM audit_log
WHERE created_at >= '2026-01-01'
AND created_at < '2026-02-01'
AND tenant_id = '...';
-- Query across months (scans multiple partitions)
SELECT * FROM audit_log
WHERE created_at >= '2026-01-01'
AND created_at < '2026-03-01'
AND action = 'permission_grant';
-- Check which partitions will be scanned
EXPLAIN SELECT * FROM audit_log
WHERE created_at >= '2026-01-01'
AND created_at < '2026-02-01';
```
### Dropping Old Partitions
```sql
-- Drop partitions older than retention period
DROP TABLE IF EXISTS audit_log_2025_01;
DROP TABLE IF EXISTS audit_log_2025_02;
-- ...
-- Or use a function
SELECT drop_old_audit_partitions(12); -- Keep last 12 months
```
### Performance Considerations
- **Index each partition** — Don't rely on parent table indexes
- **Use `created_at` in WHERE** — Always filter by date for partition pruning
- **Monitor partition count** — Too many partitions can slow planning
- **Archive old partitions** — Consider moving to cheaper storage
- **Vacuum partitions** — Each partition needs independent vacuum
### Monitoring Partition Health
```sql
@@ -340,18 +432,11 @@ SELECT
FROM pg_catalog.pg_stat_user_tables
WHERE relname LIKE 'audit_log_%'
ORDER BY relname;
-- List all partitions
SELECT
inhrelid::regclass AS partition_name
FROM pg_catalog.pg_inherits
WHERE inhparent = 'audit_log'::regclass
ORDER BY partition_name;
```
---
## 3. Backup and Recovery
## 6. Backup and Recovery
### Database Backup
@@ -361,11 +446,19 @@ pg_dump -h localhost -U leocrm -d crm_db -F c -f /backups/crm_db_$(date +%Y%m%d)
# Backup with compression
pg_dump -h localhost -U leocrm -d crm_db -F c -Z 9 -f /backups/crm_db_$(date +%Y%m%d).dump.gz
# Backup specific schema only
pg_dump -h localhost -U leocrm -d crm_db -n public -F c -f /backups/crm_db_schema_$(date +%Y%m%d).dump
```
### Automated Backup
LeoCRM hat einen automatisierten Backup via ARQ Cron-Job:
- **Job:** `auto_backup_job` (täglich 03:00 Uhr)
- **Script:** `scripts/backup.py` (pg_dump + files)
- **Konfiguration:** Settings → Backup (backup_enabled, backup_interval, backup_retention_days, backup_destination)
- **API:** `POST /api/v1/system-settings/backup-now` (manueller Trigger)
- **History:** `GET /api/v1/system-settings/backup-history` (letzte 10 Backups)
- **Bei Fehler:** System-Message an Communication-System
### Database Restore
```bash
@@ -376,13 +469,11 @@ pg_restore -h localhost -U leocrm -d crm_db -c /backups/crm_db_20260701.dump
pg_restore -h localhost -U leocrm -d crm_db -j 4 -c /backups/crm_db_20260701.dump
```
### Automated Backup Script
See `scripts/backup.py` for the automated backup solution.
See `scripts/restore.py` for the automated restore solution.
---
## 4. Monitoring and Alerts
## 7. Monitoring and Alerts
### Key Metrics
@@ -393,6 +484,8 @@ See `scripts/backup.py` for the automated backup solution.
| Cache hit ratio | > 95% | < 90% |
| Partition size | < 10GB | > 50GB |
| PgBouncer pool usage | < 80% | > 90% |
| Worker queue length | < 100 | > 500 |
| Error rate | < 1% | > 5% |
### Health Checks
@@ -405,4 +498,32 @@ psql -h localhost -U leocrm -d crm_db -c "SELECT count(*) FROM audit_log WHERE c
# Check database size
psql -h localhost -U leocrm -d crm_db -c "SELECT pg_size_pretty(pg_database_size('crm_db'));"
# Via API (Admin)
curl -b "leocrm_session=<session>" https://crm.media-on.de/api/v1/system/dashboard | jq .
```
---
## 8. Production Resource Recommendations
Die `docker-compose.yaml` hat Development-Defaults. Für Produktion mit 50+ Usern müssen die Limits erhöht werden.
| Service | Development | Production (50+ User) | Begründung |
|---|---|---|---|
| **PostgreSQL RAM** | 512m | 1-2GB | pgvector HNSW + FTS + JSONB Snapshots + Outbox + AuditLog |
| **PostgreSQL CPU** | 1.0 | 2.0 | Vector Search + FTS + normale CRM-Queries |
| **PostgreSQL Disk** | Named Volume | 50-100GB | Embeddings (768 dim × 100k = ~300MB), JSONB Snapshots, Outbox |
| **Redis RAM** | 128m | 256-512m | WS Pub/Sub + Caching + Sessions + ARQ + Rate-Limiting |
| **Redis CPU** | 0.5 | 1.0 | Pub/Sub + Cache + Queue |
| **App (FastAPI) RAM** | Nicht limitiert | 512m-1GB | WebSocket Connections + Async Tasks |
| **App CPU** | Nicht limitiert | 1-2 CPUs | API + WS + LLM-Streaming |
| **Worker (ARQ) RAM** | Nicht limitiert | 256-512m | Background Jobs (Indexierung, Agent-Runs, Extraction) |
| **Worker CPU** | Nicht limitiert | 1-2 CPUs | LLM-Calls + Embedding + Text-Extraction |
### Skalierung bei Bedarf
- **Read-Replicas:** Bei hohem Lese-Aufkommen (Search, FTS, Vector) können Read-Replicas für PostgreSQL eingerichtet werden.
- **Mehr Worker:** Bei hohem Background-Job-Aufkommen können zusätzliche ARQ-Worker-Container gestartet werden.
- **pgvector auslagern:** Bei sehr großen Datasets (>1M Embeddings) kann pgvector auf einen separaten PostgreSQL-Node ausgelagert werden.
- **Redis Cluster:** Bei sehr hohem Cache-/Pub/Sub-Aufkommen kann Redis Cluster eingesetzt werden.
docs/monitoring.md
+115 -8
View File
@@ -1,5 +1,8 @@
# Monitoring und Logging
> **Version:** 2.0
> **Date:** 2026-08-20
## Health Endpoints
### `/health/live` — Liveness Probe
@@ -37,6 +40,43 @@ curl https://crm.media-on.de/api/v1/health
# → {"status":"healthy","version":"1.0.0","checks":{...}}
```
---
## System Dashboard (Admin-only)
Das System Dashboard ist die zentrale Monitoring-Oberfläche für Administratoren.
- **URL:** `/system-dashboard` (Frontend, Admin-only)
- **API:** `GET /api/v1/system/dashboard` (Admin-only)
- **Alerts:** `GET /api/v1/system/alerts` (Admin-only)
### Dashboard-Inhalte
| Bereich | Metriken |
|---------|----------|
| **System Health** | Overall status (healthy/degraded/down) |
| **Database** | Connections, Table Count, DB Size (bytes/human) |
| **Redis** | Connected Clients, Used Memory, Peak Memory, Uptime |
| **Worker** | Queue Length, Active Workers, Status (up/degraded/down) |
| **API Stats** | Total Requests, Error Count, Error Rate, Avg Response Time |
| **Plugins** | Total Discovered, Active Plugins (name, version, is_core) |
| **Storage** | Disk Usage (total/used/free), File Count, Status |
| **Alert Feed** | System Messages aus Communication-System |
### Alert-Dispatch
Bei Problemen (DB down, Redis down, Worker down, High Error Rate) sendet das System Dashboard automatisch eine System-Message an das Communication-System (`post_system_message()`). Diese erscheint im Alert-Feed des Dashboards und in den Benachrichtigungen der Admins.
```bash
# Dashboard abfragen
curl -b "leocrm_session=<session>" https://crm.media-on.de/api/v1/system/dashboard | jq .
# Aktive Alerts abfragen
curl -b "leocrm_session=<session>" https://crm.media-on.de/api/v1/system/alerts | jq .
```
---
## Metrics Endpoint
### `/api/v1/metrics` — Prometheus Metrics
@@ -55,16 +95,22 @@ Metriken:
- `leocrm_outbox_pending` — Pending Outbox Events
- `leocrm_outbox_failed` — Failed Outbox Events
## Externes Monitoring
---
### Empfohlene Tools
## Alerting via Notification-System
- **Uptime Kuma** — Einfache Uptime-Überwachung
- **Prometheus + Grafana** — Full Metrics Dashboard
- **Sentry** — Error Tracking
- **Coolify Health Monitoring** — Eingebaut in Coolify
LeoCRM nutzt das integrierte Notification-System für Alerting. Bei System-Problemen werden automatisch System-Messages an das Communication-System gesendet.
### Alerting Regeln
### Alert-Quellen
| Quelle | Trigger | Ziel |
|--------|---------|------|
| System Dashboard | DB/Redis/Worker down, High Error Rate | Communication-System (Alert Feed) |
| Backup Job | Backup fehlgeschlagen | Communication-System + Audit Log |
| ARQ Worker | Job fehlgeschlagen, Queue überlastet | Communication-System |
| Outbox | Delivery failed | Audit Log + Communication-System |
### Alert-Typen
| Alert | Bedingung | Severity |
|-------|-----------|----------|
@@ -77,6 +123,51 @@ Metriken:
| DB Pool Exhausted | Pool checked_out = pool_size | Critical |
| Outbox Backlog | Pending > 100 | Warning |
| Worker Down | Worker heartbeat fehlt | Critical |
| Backup Failed | Backup job returned error | Critical |
| Queue Overload | Queue length > 500 | Warning |
### Alert-Empfang
Alerts erscheinen:
1. Im System Dashboard Alert-Feed (`/system-dashboard`)
2. In den Benachrichtigungen der Admins (Notification-System)
3. Im Audit Log (für Backup/Worker/Outbox Alerts)
---
## Incident Response Runbook
Siehe `docs/incident-response-runbook.md` für detaillierte Notfall-Prozeduren.
### Schnell-Referenz
| Incident | Erste Maßnahme | escalation |
|----------|---------------|------------|
| **Server-Ausfall** | Coolify Restart → Health-Check → System-Message | Hetzner Support |
| **DB-Crash** | PostgreSQL Restart → Migration-Check → Backup-Restore | Coolify DB Restart |
| **Redis-Crash** | Redis Restart → Session-Check | Coolify Service Restart |
| **Security-Breach** | Logs prüfen → Password-Reset → Audit-Log Export | Incident Response Team |
| **Backup Failed** | Backup-Log prüfen → Manueller Retry → Storage prüfen | Admin |
| **Worker Down** | Worker-Container Restart → Queue prüfen | Coolify Service Restart |
### Incident Response Schritte
1. **Erkennen** — Alert im System Dashboard oder externes Monitoring
2. **Eingrenzen** — Health-Checks, Logs, Metrics prüfen
3. **Beheben** — Restart, Restore, Konfiguration anpassen
4. **Verifizieren** — Health-Check grün, System Dashboard ok
5. **Dokumentieren** — Audit Log Eintrag, Post-Mortem bei Critical
---
## Externes Monitoring
### Empfohlene Tools
- **Uptime Kuma** — Einfache Uptime-Überwachung
- **Prometheus + Grafana** — Full Metrics Dashboard
- **Sentry** — Error Tracking
- **Coolify Health Monitoring** — Eingebaut in Coolify
### Coolify Health Check Konfiguration
@@ -111,6 +202,8 @@ scrape_configs:
credentials: '<admin-token>'
```
---
## Strukturiertes Logging
Alle API-Requests werden strukturiert geloggt:
@@ -124,7 +217,7 @@ Alle API-Requests werden strukturiert geloggt:
"tenant_id": "bfe4d09e-...",
"event": "api_request",
"level": "info",
"timestamp": "2026-07-29T16:00:00Z"
"timestamp": "2026-08-20T14:00:00Z"
}
```
@@ -132,3 +225,17 @@ Log-Level:
- `info` — Normale API-Requests
- `warning` — Langsame Requests, Permission denied
- `error` — 500er Fehler, Exceptions
### Error Tracking
```python
# app/core/monitoring.py
record_error(
error_code="DB_CONNECTION_FAILED",
message="Database connection lost",
context={"host": "postgres", "port": 5432},
trace_id="abc-123",
)
```
Fehler werden mit `trace_id` getrackt für Korrelation über Services hinweg.