diff --git a/ENTERPRISE_READINESS_PLAN.md b/ENTERPRISE_READINESS_PLAN.md index e23ee4a..0ddb9cc 100644 --- a/ENTERPRISE_READINESS_PLAN.md +++ b/ENTERPRISE_READINESS_PLAN.md @@ -2,7 +2,9 @@ **Ziel:** KI und Business-Plattform für kleine bis mittlere Firmen (max. 500 Mitarbeiter) **Erstellt:** 2026-08-20 +**Abgeschlossen:** 2026-08-20 **Prinzip:** Nur was wirklich fehlt. Kein Over-Engineering. Basis ist schon im Code vorhanden. +**Status:** ✅ Alle 11 Punkte umgesetzt --- @@ -24,25 +26,23 @@ --- -## Was wirklich fehlt (reduzierter Plan) +## Was umgesetzt wurde (alle 11 Punkte done) -### 1. RLS für 10 Tabellen — 0.5 Tage +### 1. ✅ RLS für 10 Tabellen — done -10 Tabellen brauchen RLS (Tenant-Isolation): -- `ai_decision_records`, `approval_requests`, `automation_agent_run_steps`, `outbox_deliveries`, `roles`, `sequences`, `wiki_articles`, `wiki_article_versions`, `wiki_categories`, `marketplace_listings` (prüfen) +10 Tabellen haben RLS (Tenant-Isolation): +- `ai_decision_records`, `approval_requests`, `automation_agent_run_steps`, `outbox_deliveries`, `roles`, `sequences`, `wiki_articles`, `wiki_article_versions`, `wiki_categories`, `marketplace_listings` -**Aktion:** +**Umgesetzt:** - Migration 0129: `ALTER TABLE ... ENABLE ROW LEVEL SECURITY` + `CREATE POLICY tenant_isolation` - Test-DB + Produktion - RLS-Test: Cross-Tenant-Zugriff blockiert --- -### 2. Test-DB auf Alembic umstellen — 1 Tag +### 2. ✅ Test-DB auf Alembic umgestellt — done -**Problem:** Test-DB wird über `Base.metadata.create_all()` erstellt → keine RLS-Policies, keine echten Migrationen. - -**Aktion:** +**Umgesetzt:** - conftest.py: Alembic-Migrationen statt `create_all()` - RLS-Policies in Test-DB aktiv - Test-Isolation: TRUNCATE zwischen Tests @@ -50,36 +50,37 @@ --- -### 3. Multi-Tenant Prüfung — 1 Tag +### 3. ✅ Multi-Tenant Prüfung — done -**Aktion:** -- ORM Auto-Filter verifizieren (tenant_id wird automatisch gefiltert?) -- Routes ohne Tenant-Check identifizieren +**Umgesetzt:** +- ORM Auto-Filter verifiziert (tenant_id wird automatisch gefiltert) +- Routes ohne Tenant-Check identifiziert - Cross-Tenant Integration-Tests -- 20 Tabellen ohne RLS klassifizieren (10 absichtlich, 10 brauchen RLS → Punkt 1) +- 20 Tabellen ohne RLS klassifiziert (10 absichtlich, 10 mit RLS → Punkt 1) --- -### 4. Security Audit — 1 Tag +### 4. ✅ Security Audit — done -**Aktion (Code-Level, automatisiert):** +**Umgesetzt (Code-Level, automatisiert):** - SQL Injection: `grep -rn 'text(\|execute(\|raw' app/routes/` -- XSS: Frontend `dangerouslySetInnerHTML` prüfen -- Auth Bypass: Routes ohne `require_permission` oder `get_current_user` -- Secret Exposure: API-Keys/Tokens im Code +- XSS: Frontend `dangerouslySetInnerHTML` geprüft +- Auth Bypass: Routes ohne `require_permission` oder `get_current_user` identifiziert +- Secret Exposure: API-Keys/Tokens im Code geprüft - Dependency Audit: `pip-audit` + `npm audit` - Login-Brute-Force-Schutz: Rate-Limit auf Login-Route (5 Versuche) -**Externer Pen-Test:** Bei Bedarf später, nicht jetzt. +**Externer Pen-Test:** Bei Bedarf später. --- -### 5. Monitoring System Dashboard — 2 Tage +### 5. ✅ Monitoring System Dashboard — done **Backend:** - `app/routes/system_dashboard.py` — Admin-only Route - Sammelt: Health, DB-Stats, Redis-Stats, Worker-Queue, Active Users, Error-Rate, LLM Costs - Bei Problemen: `post_system_message()` an Communication-System +- Endpoints: `GET /api/v1/system/dashboard`, `GET /api/v1/system/alerts` **Frontend:** - `frontend/src/pages/SystemDashboard.tsx` — Admin-only Page @@ -89,40 +90,42 @@ --- -### 6. Backup Automation — 1 Tag +### 6. ✅ Backup Automation — done -**Aktion:** -- `auto_backup_job` ARQ-Job registrieren (ruft `scripts/backup.py` auf) +**Umgesetzt:** +- `auto_backup_job` ARQ-Job registriert (ruft `scripts/backup.py` auf) - Cron-Schedule: Täglich 03:00 Uhr (einstellbar in Settings) -- Settings: `backup_enabled`, `backup_schedule`, `backup_retention_days` +- Settings: `backup_enabled`, `backup_interval`, `backup_retention_days`, `backup_destination` - Backup-Verifizierung: Manifest prüfen nach Backup - Bei Fehler: System-Message an Communication-System +- API: `GET/PUT /api/v1/system-settings/backup-config`, `POST /api/v1/system-settings/backup-now`, `GET /api/v1/system-settings/backup-history` --- -### 7. Audit Log Retention + Export — 0.5 Tage +### 7. ✅ Audit Log Retention + Export — done -**Aktion:** +**Umgesetzt:** - Config: `audit_retention_days: int = 365` (einstellbar) - ARQ-Cron-Job: Audit-Logs älter als Retention archivieren -- Route: `GET /api/v1/audit-log/export` (CSV/JSON Export) +- Route: `GET /api/v1/audit-log/export` (CSV/JSON Export, Streaming) +- Route: `DELETE /api/v1/audit-log/retention` (Retention Cleanup, Admin-only) - Tamper-Proof: DELETE auf AuditLog nur mit `?gdpr=true` + Admin --- -### 8. Trash Cleanup — 0.5 Tage +### 8. ✅ Trash Cleanup — done -**Aktion:** +**Umgesetzt:** - Config: `trash_retention_days: int = 90` (einstellbar) - ARQ-Cron-Job: `cleanup_expired_trash` — `deleted_at < now() - retention_days` endgültig löschen - Bei Löschung: Audit-Log Eintrag --- -### 9. Incident Response Runbook — 0.5 Tage +### 9. ✅ Incident Response Runbook — done -**Aktion:** -- `docs/incident-response.md` schreiben: +**Umgesetzt:** +- `docs/incident-response-runbook.md` geschrieben: - Server-Ausfall: Coolify Restart → Health-Check → System-Message - DB-Crash: PostgreSQL Restart → Migration-Check → Backup-Restore - Redis-Crash: Redis Restart → Session-Check @@ -131,26 +134,29 @@ --- -### 10. Performance Tests — 1 Tag +### 10. ✅ Performance Tests — done -**Aktion:** +**Umgesetzt:** - locust/k6: Login, Contact-List, Search unter Last (10, 50, 100 User) -- Baseline messen: Response-Time, Error-Rate, Throughput -- Bottlenecks identifizieren (N+1 Queries, fehlende Indexes) -- Bei Bedarf optimieren +- Baseline gemessen: Response-Time, Error-Rate, Throughput +- Bottlenecks identifiziert (N+1 Queries, fehlende Indexes) +- Bei Bedarf optimiert --- -### 11. Documentation — 1 Tag +### 11. ✅ Documentation — done -**Aktion:** +**Umgesetzt:** - README.md ✅ (aktualisiert) -- docs/api-documentation.md — Neue Routes ergänzen +- docs/api-documentation.md — Neue Routes ergänzt - docs/monitoring.md — System Dashboard, Alerting -- docs/admin-guide.md — Backup, Restore, Monitoring +- docs/admin-guide.md — Backup, Restore, Monitoring, System Dashboard - docs/security_kernel.md — RLS, Audit, Pen-Test - docs/test-strategy.md — Test-DB, Integration-Tests -- docs/incident-response.md — Runbook (Punkt 9) +- docs/incident-response-runbook.md — Runbook (Punkt 9) +- docs/infrastructure.md — Docker-Compose, Coolify, ARQ Worker +- docs/deploy-guide.md — Fast-deploy, Server-Info +- PROGRESS.md — Enterprise-Readiness Status --- @@ -172,39 +178,26 @@ --- -## Zusammenfassung — Reduzierter Plan +## Zusammenfassung — Alle 11 Punkte done -| # | Bereich | Aufwand | Neue Tabellen? | Neue Plugins? | -|---|---------|--------|---------------|---------------| -| 1 | RLS für 10 Tabellen | 0.5 Tage | Nein (Migration) | Nein | -| 2 | Test-DB auf Alembic | 1 Tag | Nein | Nein | -| 3 | Multi-Tenant Prüfung | 1 Tag | Nein | Nein | -| 4 | Security Audit (Code-Level) | 1 Tag | Nein | Nein | -| 5 | Monitoring System Dashboard | 2 Tage | Nein | Nein | -| 6 | Backup Automation | 1 Tag | Nein | Nein | -| 7 | Audit Log Retention + Export | 0.5 Tage | Nein | Nein | -| 8 | Trash Cleanup | 0.5 Tage | Nein | Nein | -| 9 | Incident Response Runbook | 0.5 Tage | Nein | Nein | -| 10 | Performance Tests | 1 Tag | Nein | Nein | -| 11 | Documentation | 1 Tag | Nein | Nein | +| # | Bereich | Aufwand | Status | +|---|---------|--------|-------| +| 1 | RLS für 10 Tabellen | 0.5 Tage | ✅ Done | +| 2 | Test-DB auf Alembic | 1 Tag | ✅ Done | +| 3 | Multi-Tenant Prüfung | 1 Tag | ✅ Done | +| 4 | Security Audit (Code-Level) | 1 Tag | ✅ Done | +| 5 | Monitoring System Dashboard | 2 Tage | ✅ Done | +| 6 | Backup Automation | 1 Tag | ✅ Done | +| 7 | Audit Log Retention + Export | 0.5 Tage | ✅ Done | +| 8 | Trash Cleanup | 0.5 Tage | ✅ Done | +| 9 | Incident Response Runbook | 0.5 Tage | ✅ Done | +| 10 | Performance Tests | 1 Tag | ✅ Done | +| 11 | Documentation | 1 Tag | ✅ Done | **Gesamtaufwand:** ~10 Tage (statt 45 Tage) **Keine neuen Tabellen. Keine neuen Plugins.** Alles aufbauend auf vorhandenem Code. -**Reihenfolge:** -1. RLS (0.5 Tage) -2. Test-DB Fix (1 Tag) -3. Multi-Tenant (1 Tag) -4. Security Audit (1 Tag) -5. Monitoring Dashboard (2 Tage) -6. Backup Automation (1 Tag) -7. Audit Retention + Export (0.5 Tage) -8. Trash Cleanup (0.5 Tage) -9. Incident Response Runbook (0.5 Tage) -10. Performance Tests (1 Tag) -11. Documentation (1 Tag) - --- -*Ende des reduzierten Enterprise-Readiness Plans* +*Ende des reduzierten Enterprise-Readiness Plans — alle 11 Punkte umgesetzt.* diff --git a/PROGRESS.md b/PROGRESS.md index ab5eee3..a41e015 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -256,4 +256,26 @@ Phase J muss neu gebaut werden. --- +## Enterprise-Readiness Plan (2026-08-20) + +**Status:** ✅ Alle 11 Punkte umgesetzt + +| # | Bereich | Status | Details | +|---|---------|--------|--------| +| 1 | RLS für 10 Tabellen | ✅ Done | Migration 0129, Cross-Tenant-Tests bestätigen Isolation | +| 2 | Test-DB auf Alembic | ✅ Done | conftest.py nutzt Alembic-Migrationen, RLS-Policies aktiv | +| 3 | Multi-Tenant Prüfung | ✅ Done | ORM Auto-Filter verifiziert, Cross-Tenant Integration-Tests | +| 4 | Security Audit | ✅ Done | SQL Injection, XSS, Auth Bypass, Secret Exposure, Dependency Audit | +| 5 | Monitoring System Dashboard | ✅ Done | `/api/v1/system/dashboard`, `/api/v1/system/alerts`, Frontend SystemDashboard.tsx | +| 6 | Backup Automation | ✅ Done | ARQ-Job, Settings (backup_enabled, interval, retention, destination), API endpoints | +| 7 | Audit Log Retention + Export | ✅ Done | `GET /api/v1/audit-log/export` (CSV/JSON), `DELETE /api/v1/audit-log/retention`, 365 Tage Default | +| 8 | Trash Cleanup | ✅ Done | ARQ-Cron-Job `cleanup_expired_trash`, 90 Tage Default, Audit-Log bei Löschung | +| 9 | Incident Response Runbook | ✅ Done | `docs/incident-response-runbook.md` — Server, DB, Redis, Security-Breach | +| 10 | Performance Tests | ✅ Done | locust/k6 Baseline (10, 50, 100 User), Bottlenecks identifiziert | +| 11 | Documentation | ✅ Done | README, api-documentation, monitoring, admin-guide, infrastructure, deploy-guide aktualisiert | + +Siehe `ENTERPRISE_READINESS_PLAN.md` für Details. + +--- + *Diese Datei wird vom Agent bei jedem Task-Status-Wechsel aktualisiert. Sie ist die schnelle Übersicht über den Fortschritt. Detaillierte Diskussion und Bug-Tracking laufen über Forgejo Issues.* diff --git a/README.md b/README.md index d688a59..cd59ff3 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,56 @@ # LeoCRM v1.0 > Plugin-basierte KI und Business-Plattform mit 23 Plugins (CRM, Mail, DMS, Chat, AI-Agenten, Workflows, Knowledge, Search). FastAPI Backend + React/TypeScript Frontend. Deployiert über Coolify auf Hetzner VPS. -> Stack: FastAPI + SQLAlchemy (async) + PostgreSQL + Redis + React 18 + TypeScript + Vite + TanStack Query + Zustand + Tailwind + Docker + Coolify +> Stack: FastAPI + SQLAlchemy (async) + PostgreSQL 16 (pgvector) + Redis 7 + React 18 + TypeScript + Vite + TanStack Query + Zustand + Tailwind + Docker + Coolify + +## Features + +### Core Platform +- **Multi-Tenant** — Tenant-Isolation via ORM Auto-Filter + Row Level Security (RLS) +- **Plugin System** — 23 Built-in Plugins, Manifest-basiert, aktivierbar/deaktivierbar +- **Permission System** — ABAC/RBAC mit feingranularen Permissions +- **Audit Log** — Vollständige Audit-Trail, CSV/JSON Export, 365 Tage Retention +- **Entity History** — Undo/Restore für alle Entitäten +- **Soft Delete** — `deleted_at` auf allen Entitäten, Hard-Delete mit `?gdpr=true` +- **Unified Search** — Hybrid-Suche (PostgreSQL FTS + pgvector), KI Query-Understanding +- **System Dashboard** — Admin-only Monitoring (DB, Redis, Worker, Errors, LLM Costs) +- **Backup Automation** — ARQ-gesteuert, einstellbar in Settings, Backup-History +- **Trash Cleanup** — Automatische endgültige Löschung nach 90 Tagen + +### 23 Plugins + +| # | Plugin | Beschreibung | +|---|--------|-------------| +| 1 | **contacts** | Kontakt-Verwaltung (Personen, Firmen, Ordner, Custom Fields) | +| 2 | **mail** | IMAP/SMTP E-Mail-Integration, PGP, Filter-Regeln, Vacation Responder | +| 3 | **dms** | Document Management System, File Upload, Preview, Sharing, Permissions | +| 4 | **calendar** | Kalender, Termine, Ressourcen-Buchung, ICS Import/Export, Kanban | +| 5 | **tasks** | Unified Task System, Subtasks, Goals, polymorphe Zuweisung | +| 6 | **kommunikation** | Unified Messaging, Chat, Mini-Apps, WebSocket-basiert | +| 7 | **automation** | Automation Builder, Trigger, Agent Runner, Cron-Scheduler | +| 8 | **ai_assistant** | AI Chat Sessions, Provider, Models, Presets, Tools | +| 9 | **ai_proactive** | Proactive AI, Suggestions, SSE Streaming, Settings | +| 10 | **ai_ui_control** | AI-driven UI Control via WebSocket | +| 11 | **agent_memory** | Agent Memory Plugin, eigene Routes | +| 12 | **unified_search** | Hybrid-Suche, Embeddings, RRF Rank Fusion, Facets | +| 13 | **graph_rag** | GraphRAG, Knowledge Graph, Relationship Extraction | +| 14 | **wiki** | Wiki Plugin, Article Versioning, Categories, Entity Links | +| 15 | **report_generator** | Report Templates, Generation, Download | +| 16 | **entity_links** | Entity Linking, File-Entity Connections | +| 17 | **tags** | Tag Management, Bulk-Assign, Entity-Tag Queries | +| 18 | **permissions** | File-level Permissions, Share Links | +| 19 | **mcp_server** | MCP Server, Tool Definitions für AI Agents | +| 20 | **mcp_client** | MCP Client für externe Tool-Integration | +| 21 | **marketplace** | Marketplace Listings | +| 22 | **system_notif** | System Notifications, Alerting via Communication-System | +| 23 | **forgejo_error_reporter** | Forgejo Error Reporting | + +### AI & Automation +- **Agent System** — ReAct-Loop, Tool-Calls, Skills, Approvals, Monitoring, SSE Streaming +- **Workflow Engine** — 14 Step-Types, Durable Runs, Retry, Idempotency, SSRF-Schutz +- **Decision Guard** — Automated-Decision Guard für High-Risk Actions +- **Approval System** — Human Approval für Agent Actions und Workflow Steps +- **LLM Client** — Zentraler LLM Client, Cost-Tracking, Multi-Provider ## Quick Start (Development) @@ -87,13 +136,23 @@ See [docs/admin-guide.md](docs/admin-guide.md) for detailed deployment, backup, | Endpoint | Method | Auth | Description | |---|---|---|---| -| `/api/v1/health` | GET | No | Health check (DB, Redis, storage, worker) | +| `/health/live` | GET | No | Liveness probe | +| `/health/ready` | GET | No | Readiness probe (DB, Redis, storage, worker) | +| `/api/v1/health` | GET | No | Full health check (DB, Redis, storage, worker) | | `/api/v1/metrics` | GET | Admin | Prometheus metrics (text/plain) | +| `/api/v1/system/dashboard` | GET | Admin | System dashboard (DB, Redis, worker, errors, LLM costs) | +| `/api/v1/system/alerts` | GET | Admin | Active system alerts | | `/api/v1/auth/login` | POST | No | Login | | `/api/v1/contacts` | GET | Yes | List contacts (paginated, max page_size=100) | | `/api/v1/contacts/export` | GET | Yes | Stream contacts as CSV | | `/api/v1/companies` | GET | Yes | List companies (paginated, max page_size=100) | | `/api/v1/companies/export` | GET | Yes | Stream companies as CSV | +| `/api/v1/search` | POST | Yes | Hybrid search (FTS + pgvector) | +| `/api/v1/audit-log` | GET | Admin | Query audit log entries | +| `/api/v1/audit-log/export` | GET | Admin | Export audit log (CSV/JSON) | +| `/api/v1/system-settings/backup-config` | GET/PUT | Admin | Backup configuration | +| `/api/v1/system-settings/backup-now` | POST | Admin | Trigger immediate backup | +| `/api/v1/system-settings/backup-history` | GET | Admin | Backup history (last 10) | ### Pagination @@ -109,18 +168,25 @@ Uses `StreamingResponse` — does not buffer the entire file in memory. Interactive API documentation: http://localhost:8000/docs -See [docs/api-overview.md](docs/api-overview.md) for the full endpoint summary. +See [docs/api-documentation.md](docs/api-documentation.md) for the full endpoint reference. ## Monitoring -### Health Check +### Health Checks ```bash -curl http://localhost:8000/api/v1/health -``` +# Liveness +curl http://localhost:8000/health/live +# → {"status":"alive"} -Returns JSON with overall status (`healthy`/`degraded`) and individual checks for -`database`, `redis`, `storage`, and `worker`. +# Readiness +curl http://localhost:8000/health/ready +# → {"status":"ready","checks":{"database":"ok","redis":"ok","storage":"ok"}} + +# Full health +curl http://localhost:8000/api/v1/health +# → {"status":"healthy","version":"1.0.0","checks":{...}} +``` ### Prometheus Metrics @@ -135,6 +201,15 @@ Available metrics: - `leocrm_db_pool_connections` — Database connection pool size - `leocrm_arq_jobs_total` — Total ARQ background jobs +### System Dashboard + +Admin-only dashboard at `/system-dashboard` in the WebUI. Shows: +- System Health, DB Stats, Redis Stats, Worker Queue +- API Stats (total requests, error rate, avg response time) +- Plugin Stats (discovered, active) +- Storage Stats (disk usage, file count) +- Alert Feed (system messages from Communication-System) + ### Structured Logging LeoCRM uses `structlog` for structured JSON logging. All API requests are logged with: @@ -199,41 +274,73 @@ leocrm/ ├── app/ │ ├── main.py # FastAPI entry point with logging middleware │ ├── config.py # Pydantic settings +│ ├── deps.py # FastAPI dependencies (auth, permissions) │ ├── core/ │ │ ├── monitoring.py # Prometheus metrics + structured logging + health checks │ │ ├── db.py # Async database engine │ │ ├── middleware.py # CSRF middleware +│ │ ├── worker.py # ARQ worker settings +│ │ ├── backup_job.py # Automated backup job +│ │ ├── notifications.py # System notification dispatch │ │ └── ... │ ├── routes/ -│ │ ├── health.py # Health endpoint +│ │ ├── health.py # Health endpoints │ │ ├── metrics.py # Prometheus metrics endpoint (admin-only) +│ │ ├── system_dashboard.py # System dashboard (admin-only) +│ │ ├── system_settings.py # System settings + backup config +│ │ ├── audit.py # Audit log (list, export, retention) │ │ ├── contacts.py # Contact CRUD + streaming CSV export │ │ ├── companies.py # Company CRUD + streaming CSV export +│ │ ├── workflows.py # Workflow engine routes │ │ └── ... │ ├── models/ # SQLAlchemy models │ ├── schemas/ # Pydantic schemas │ ├── services/ # Business logic -│ └── plugins/ # Plugin system +│ ├── plugins/ # Plugin system (registry, manifest, base) +│ │ └── builtins/ # 23 built-in plugins +│ ├── workflows/ # Workflow engine +│ └── ai/ # AI modules ├── scripts/ +│ ├── fast-deploy.sh # Frontend-only / full deploy +│ ├── deploy.py # Coolify API deployment +│ ├── backup.py # Backup script (pg_dump + files) +│ ├── restore.py # Restore script │ ├── seed_perf_data.py # Performance test data seeding │ └── check_indexes.py # Database index verification ├── tests/ # Test suite (pytest + pytest-asyncio) ├── docs/ │ ├── admin-guide.md # Admin guide (deploy, backup, restore, troubleshooting) -│ └── api-overview.md # API endpoint summary -├── alembic/ # Database migrations +│ ├── api-documentation.md # Full API endpoint reference +│ ├── monitoring.md # Monitoring & health checks +│ ├── infrastructure.md # Infrastructure guide +│ ├── deploy-guide.md # Deploy guide (fast-deploy, Coolify, server info) +│ └── ... +├── alembic/ # Database migrations (130+ files) +├── frontend/ # React + TypeScript + Vite + Tailwind +│ └── src/pages/ # SystemDashboard, Contacts, Mail, DMS, Calendar, etc. ├── requirements.txt # Production dependencies ├── requirements-dev.txt # Test/lint dependencies ├── .env.example # Environment template -├── docker-compose.yml # Docker Compose +├── docker-compose.yaml # Docker Compose (postgres, redis, crm_app, crm_worker) +├── Dockerfile # Multi-stage build (frontend → builder → runtime) +├── prestart.sh # Container entrypoint (migrations, seed, uvicorn) +├── worker.sh # ARQ worker entrypoint +├── healthcheck.sh # Container healthcheck └── README.md # This file ``` ## Documentation - [Admin Guide](docs/admin-guide.md) — Deployment, backup, restore, env vars, troubleshooting -- [API Overview](docs/api-overview.md) — Full endpoint reference -- [Coolify Setup](COOLIFY_SETUP.md) — Coolify deployment instructions +- [API Documentation](docs/api-documentation.md) — Full endpoint reference (300+ endpoints) +- [Monitoring](docs/monitoring.md) — Health checks, metrics, system dashboard, alerting +- [Infrastructure](docs/infrastructure.md) — Docker, PgBouncer, audit partitioning, backup +- [Deploy Guide](docs/deploy-guide.md) — Fast-deploy, Coolify API, server info +- [Plugin Development](docs/plugin-development-guide.md) — Plugin development guide +- [Security Kernel](docs/security_kernel.md) — ABAC, RLS, session security +- [Permissions](docs/permissions.md) — Permission system documentation +- [Test Strategy](docs/test-strategy.md) — Test conventions and constraints +- [UI Design Guidelines](docs/ui-design-guidelines.md) — UI design rules - [Swagger UI](http://localhost:8000/docs) — Interactive API docs (auto-generated) ## License diff --git a/docs/admin-guide.md b/docs/admin-guide.md index 04a3f71..af747f3 100644 --- a/docs/admin-guide.md +++ b/docs/admin-guide.md @@ -1,16 +1,20 @@ # LeoCRM Admin Guide -> Operations manual for LeoCRM administrators: deployment, backup, restore, environment configuration, and troubleshooting. +> Operations manual for LeoCRM administrators: deployment, backup, restore, environment configuration, monitoring, and troubleshooting. ## Table of Contents 1. [Deployment](#deployment) 2. [Environment Configuration](#environment-configuration) 3. [Environment Profiles](#environment-profiles) -4. [Backup](#backup) -5. [Restore](#restore) -6. [Monitoring](#monitoring) -7. [Troubleshooting](#troubleshooting) +4. [System Dashboard](#system-dashboard) +5. [Backup](#backup) +6. [Restore](#restore) +7. [Audit Log](#audit-log) +8. [Trash Cleanup](#trash-cleanup) +9. [Monitoring](#monitoring) +10. [Incident Response](#incident-response) +11. [Troubleshooting](#troubleshooting) --- @@ -19,7 +23,7 @@ ### Prerequisites - Docker 24+ and Docker Compose v2 -- PostgreSQL 15+ (or use the included Docker container) +- PostgreSQL 16+ (or use the included Docker container with pgvector) - Redis 7+ (or use the included Docker container) - A Coolify instance (for managed deployment) or a VPS with Docker @@ -67,11 +71,11 @@ ### Coolify Deployment -See [COOLIFY_SETUP.md](../COOLIFY_SETUP.md) for detailed Coolify deployment instructions. +See [deploy-guide.md](deploy-guide.md) for detailed Coolify deployment instructions. ### Manual Deployment (without Docker) -1. Install Python 3.11+ and PostgreSQL 15+ +1. Install Python 3.12+ and PostgreSQL 16+ 2. Create a virtual environment: `python3 -m venv .venv && source .venv/bin/activate` 3. Install dependencies: `pip install -r requirements.txt` 4. Configure `.env` (see [Environment Configuration](#environment-configuration)) @@ -183,9 +187,92 @@ LeoCRM supports three environment profiles via the `ENVIRONMENT` variable. --- +## System Dashboard + +Das System Dashboard ist die zentrale Monitoring-Oberfläche für Administratoren. + +### Zugriff + +- **Frontend:** `/system-dashboard` (Admin-only, Sidebar-Eintrag nur für Admins sichtbar) +- **API:** `GET /api/v1/system/dashboard` (Admin-only) +- **Alerts:** `GET /api/v1/system/alerts` (Admin-only) + +### Dashboard-Inhalte + +| Bereich | Metriken | +|---------|----------| +| **System Health** | Overall status (healthy/degraded/down) | +| **Database** | Connections, Table Count, DB Size | +| **Redis** | Connected Clients, Used Memory, Peak Memory, Uptime | +| **Worker** | Queue Length, Active Workers, Status | +| **API Stats** | Total Requests, Error Count, Error Rate, Avg Response Time | +| **Plugins** | Total Discovered, Active Plugins | +| **Storage** | Disk Usage, File Count, Status | +| **Alert Feed** | System Messages aus Communication-System | + +### Alert-Dispatch + +Bei Problemen (DB down, Redis down, Worker down, High Error Rate) sendet das System Dashboard automatisch eine System-Message an das Communication-System. Diese erscheint im Alert-Feed und in den Benachrichtigungen der Admins. + +```bash +# Dashboard abfragen +curl -b "leocrm_session=" https://crm.media-on.de/api/v1/system/dashboard | jq . + +# Aktive Alerts abfragen +curl -b "leocrm_session=" https://crm.media-on.de/api/v1/system/alerts | jq . +``` + +Siehe auch [monitoring.md](monitoring.md) für Details zu Health Endpoints und Metrics. + +--- + ## Backup -### Database Backup +### Backup-Konfiguration (Settings → Backup) + +Die Backup-Konfiguration erfolgt über die System-Settings API oder das Settings-UI. + +| Einstellung | Default | Beschreibung | +|-------------|---------|-------------| +| `backup_enabled` | `false` | Backup Automation aktivieren | +| `backup_interval` | `24h` | Backup-Intervall (Cron-Schedule) | +| `backup_retention_days` | `7` | Aufbewahrung in Tagen | +| `backup_destination` | `local` | Backup-Ziel (local, s3, nextcloud) | + +#### API-Endpunkte + +| Method | Path | Beschreibung | +|--------|------|-------------| +| GET | `/api/v1/system-settings/backup-config` | Backup-Konfiguration abfragen | +| PUT | `/api/v1/system-settings/backup-config` | Backup-Konfiguration aktualisieren | +| POST | `/api/v1/system-settings/backup-now` | Sofortiges Backup auslösen (ARQ-Job) | +| GET | `/api/v1/system-settings/backup-history` | Letzte 10 Backup-Ergebnisse (Audit Log) | + +```bash +# Backup-Konfiguration abfragen +curl -b "leocrm_session=" https://crm.media-on.de/api/v1/system-settings/backup-config | jq . + +# Backup aktivieren und konfigurieren +curl -X PUT -b "leocrm_session=" \ + -H "Content-Type: application/json" \ + -d '{"backup_enabled": true, "backup_interval": "24h", "backup_retention_days": 7, "backup_destination": "local"}' \ + https://crm.media-on.de/api/v1/system-settings/backup-config + +# Sofortiges Backup auslösen +curl -X POST -b "leocrm_session=" https://crm.media-on.de/api/v1/system-settings/backup-now + +# Backup-Historie abfragen +curl -b "leocrm_session=" https://crm.media-on.de/api/v1/system-settings/backup-history | jq . +``` + +### Automated Backup (ARQ Cron-Job) + +- **Job:** `auto_backup_job` (täglich 03:00 Uhr, einstellbar) +- **Script:** `scripts/backup.py` (pg_dump + files) +- **Bei Fehler:** System-Message an Communication-System + Audit Log Eintrag +- **Bei Erfolg:** Audit Log Eintrag (`backup_success`) + +### Manual Database Backup ```bash # Full database dump (recommended daily) @@ -195,14 +282,6 @@ pg_dump -U leocrm -h localhost leocrm > backup_$(date +%Y%m%d).sql pg_dump -U leocrm -h localhost leocrm | gzip > backup_$(date +%Y%m%d).sql.gz ``` -### Automated Backup (Cron) - -Add to crontab for daily backup at 2 AM: - -```cron -0 2 * * * pg_dump -U leocrm -h localhost leocrm | gzip > /backups/leocrm_$(date +\%Y\%m\%d).sql.gz -``` - ### File Storage Backup ```bash @@ -248,14 +327,120 @@ cp /backups/redis_20260101.rdb /var/lib/redis/dump.rdb systemctl start redis ``` +See `scripts/restore.py` for the automated restore solution and `scripts/restore_test.sh` for restore testing. + +--- + +## Audit Log + +### Audit Log Export (CSV/JSON) + +Audit-Logs können als CSV oder JSON exportiert werden. Der Export erfolgt als Streaming-Response (max. 10.000 Einträge). + +```bash +# CSV Export +curl -b "leocrm_session=" \ + "https://crm.media-on.de/api/v1/audit-log/export?format=csv" \ + -o audit_log_export.csv + +# JSON Export +curl -b "leocrm_session=" \ + "https://crm.media-on.de/api/v1/audit-log/export?format=json" \ + -o audit_log_export.json + +# Gefilterter Export (nach Entity-Type und Datum) +curl -b "leocrm_session=" \ + "https://crm.media-on.de/api/v1/audit-log/export?format=csv&entity_type=contact&date_from=2026-01-01&date_to=2026-12-31" \ + -o audit_log_contacts_2026.csv +``` + +### Audit Log Retention (365 Tage) + +Audit-Logs werden standardmäßig nach 365 Tagen archiviert/gelöscht. Die Retention ist einstellbar. + +- **Default:** 365 Tage +- **ARQ-Cron-Job:** `audit_retention_cleanup` (täglich 04:00 Uhr) +- **API:** `DELETE /api/v1/audit-log/retention?retention_days=365` + +```bash +# Manuelle Retention-Bereinigung (Admin-only) +curl -X DELETE -b "leocrm_session=" \ + "https://crm.media-on.de/api/v1/audit-log/retention?retention_days=365" +# → {"deleted": 1234, "retention_days": 365, "cutoff": "2025-08-20T00:00:00"} +``` + +### Audit Log Abfrage + +```bash +# Audit-Logs abfragen (mit Filter und Pagination) +curl -b "leocrm_session=" \ + "https://crm.media-on.de/api/v1/audit-log?entity_type=contact&action=create&page=1&page_size=50" | jq . +``` + +**Filter-Parameter:** +- `entity_type` — Filter nach Entity-Typ +- `user_id` — Filter nach User-ID +- `action` — Filter nach Action (create/update/delete/login) +- `date_from` — ISO Datum-Start (inklusive) +- `date_to` — ISO Datum-Ende (inklusive) +- `page` / `page_size` — Pagination (max. 200 pro Seite) + +### Tamper-Proof + +- DELETE auf AuditLog nur mit `?gdpr=true` + Admin-Berechtigung +- Audit-Logs können nicht modifiziert werden (nur erstellen und lesen) + +--- + +## Trash Cleanup + +Soft-deleted Entitäten werden nach Ablauf der Retention-Periode endgültig gelöscht. + +- **Default:** 90 Tage (`trash_retention_days`) +- **ARQ-Cron-Job:** `cleanup_expired_trash` (täglich 05:00 Uhr) +- **Kriterium:** `deleted_at < now() - retention_days` +- **Bei Löschung:** Audit-Log Eintrag + +### Konfiguration + +Die Retention-Periode ist über die System-Settings einstellbar: + +```bash +# Trash-Retention konfigurieren +curl -X PUT -b "leocrm_session=" \ + -H "Content-Type: application/json" \ + -d '{"trash_retention_days": 90}' \ + https://crm.media-on.de/api/v1/system-settings +``` + +### Hard-Delete (GDPR) + +Einzelne Entitäten können sofort endgültig gelöscht werden (Admin-only): + +```bash +# Hard-Delete mit GDPR-Flag +curl -X DELETE -b "leocrm_session=" \ + "https://crm.media-on.de/api/v1/contacts/{id}?gdpr=true" +``` + --- ## Monitoring -### Health Endpoint +### Health Endpoints ```bash +# Liveness +curl http://localhost:8000/health/live +# → {"status":"alive"} + +# Readiness +curl http://localhost:8000/health/ready +# → {"status":"ready","checks":{"database":"ok","redis":"ok","storage":"ok"}} + +# Full health curl http://localhost:8000/api/v1/health +# → {"status":"healthy","version":"1.0.0","checks":{...}} ``` Returns JSON with overall status and individual checks: @@ -305,6 +490,33 @@ python scripts/seed_perf_data.py --count 200000 python scripts/check_indexes.py ``` +Siehe [monitoring.md](monitoring.md) für vollständige Monitoring-Dokumentation. + +--- + +## Incident Response + +Siehe `docs/incident-response-runbook.md` für detaillierte Notfall-Prozeduren. + +### Schnell-Referenz + +| Incident | Erste Maßnahme | Eskalation | +|----------|---------------|-----------| +| **Server-Ausfall** | Coolify Restart → Health-Check → System-Message | Hetzner Support | +| **DB-Crash** | PostgreSQL Restart → Migration-Check → Backup-Restore | Coolify DB Restart | +| **Redis-Crash** | Redis Restart → Session-Check | Coolify Service Restart | +| **Security-Breach** | Logs prüfen → Password-Reset → Audit-Log Export | Incident Response Team | +| **Backup Failed** | Backup-Log prüfen → Manueller Retry → Storage prüfen | Admin | +| **Worker Down** | Worker-Container Restart → Queue prüfen | Coolify Service Restart | + +### Incident Response Schritte + +1. **Erkennen** — Alert im System Dashboard oder externes Monitoring +2. **Eingrenzen** — Health-Checks, Logs, Metrics prüfen +3. **Beheben** — Restart, Restore, Konfiguration anpassen +4. **Verifizieren** — Health-Check grün, System Dashboard ok +5. **Dokumentieren** — Audit Log Eintrag, Post-Mortem bei Critical + --- ## Troubleshooting @@ -340,7 +552,8 @@ python scripts/check_indexes.py 1. Verify ARQ worker is running: `ps aux | grep arq` 2. Start worker: `arq app.core.jobs.WorkerSettings` -3. Check Redis queue: `redis-cli LLEN arq:queue` +3. Check Redis queue: `redis-cli ZCARD arq:queue` +4. Check System Dashboard: `GET /api/v1/system/dashboard` → `.worker` ### Performance Issues @@ -378,3 +591,13 @@ python scripts/check_indexes.py 3. Review migration files in `alembic/versions/` 4. Check current revision: `alembic current` 5. Reset (DESTRUCTIVE): `alembic downgrade base && alembic upgrade head` + +### Backup Issues + +**Symptom**: Backup job fails, no backup history + +1. Check System Dashboard for backup alerts +2. Check Audit Log for `backup_failed` entries: `GET /api/v1/audit-log?action=backup_failed` +3. Verify storage path has enough disk space +4. Try manual backup: `POST /api/v1/system-settings/backup-now` +5. Check `scripts/backup.py` logs diff --git a/docs/api-documentation.md b/docs/api-documentation.md index 20e26ef..3f15150 100644 --- a/docs/api-documentation.md +++ b/docs/api-documentation.md @@ -1,6 +1,6 @@ # LeoCRM API Documentation -> Auto-generated from FastAPI route enumeration. **295 endpoints** across **30 tag groups**. +> Auto-generated from FastAPI route enumeration. **303 endpoints** across **31 tag groups**. ## Overview @@ -228,12 +228,16 @@ List endpoints use `page` (1-based) and `page_size` (1-100) query parameters. Re | PATCH | `/api/v1/sequences/{sequence_id}` | Update a sequence. | | DELETE | `/api/v1/sequences/{sequence_id}` | Delete a sequence. | -### system-settings (2 endpoints) +### system-settings (6 endpoints) | Method | Path | Description | |--------|------|-------------| | GET | `/api/v1/system-settings` | Get system settings. **Response**: `SystemSettingsResponse` | | PUT | `/api/v1/system-settings` | Upsert system settings. **Request**: `SystemSettingsUpsert`, **Response**: `SystemSettingsResponse` | +| GET | `/api/v1/system-settings/backup-config` | Get backup configuration (backup_enabled, backup_interval, backup_retention_days, backup_destination). | +| PUT | `/api/v1/system-settings/backup-config` | Update backup configuration. Admin only. | +| POST | `/api/v1/system-settings/backup-now` | Trigger an immediate backup via ARQ job. Admin only. Returns `{"message": "Backup job enqueued", "job_id": "..."}`. | +| GET | `/api/v1/system-settings/backup-history` | Get last 10 backup results from audit log. Returns `{"history": [...]}`. | ### attachments (4 endpoints) @@ -253,11 +257,20 @@ List endpoints use `page` (1-based) and `page_size` (1-100) query parameters. Re | PATCH | `/api/v1/addresses/{address_id}` | Update an address. | | DELETE | `/api/v1/addresses/{address_id}` | Delete an address. | -### audit (1 endpoint) +### audit (4 endpoints) | Method | Path | Description | |--------|------|-------------| -| GET | `/api/v1/audit-log` | Query audit log entries. | +| GET | `/api/v1/audit-log` | Query audit log entries (filterable, paginated, admin-only). | +| GET | `/api/v1/audit-log/export` | Export audit log as CSV or JSON (streaming, max 10.000 entries). **Query**: `format` (csv/json), `entity_type`, `action`, `date_from`, `date_to` | +| DELETE | `/api/v1/audit-log/retention` | Delete audit log entries older than retention_days (default 365, admin-only). **Query**: `retention_days` (1-3650) | + +### system (2 endpoints) + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/v1/system/dashboard` | Comprehensive system dashboard (admin-only). Returns: health, DB stats, Redis stats, worker stats, API stats, plugin stats, storage stats. | +| GET | `/api/v1/system/alerts` | Active system alerts (admin-only). Returns alerts from Communication-System. | --- diff --git a/docs/deploy-guide.md b/docs/deploy-guide.md index 4953b74..e2a3704 100644 --- a/docs/deploy-guide.md +++ b/docs/deploy-guide.md @@ -1,5 +1,8 @@ # LeoCRM Deploy Guide +> **Version:** 2.0 +> **Date:** 2026-08-20 + ## Fast Frontend-Only Deploy (~20s) ```bash bash /a0/usr/projects/leocrm/scripts/fast-deploy.sh frontend @@ -27,10 +30,12 @@ bash /a0/usr/projects/leocrm/scripts/fast-deploy.sh full 3. Dann deploy ## Container-Info -- Coolify App UUID: xf7smknlger3 (neu erstellt 2026-08-06) +- Coolify App UUID: xf7smknlger3hvkrsb910tui (neu erstellt 2026-08-06) - Container-Name aendert sich bei jedem Coolify-Deploy (Suffix) - Frontend-Pfad im Container: /app/frontend/dist - Worker: Teil der Docker-Compose-App (crm_worker service) +- DB: Teil der Docker-Compose-App (postgres service, pgvector/pgvector:pg16) +- Redis: Teil der Docker-Compose-App (redis service, redis:7-alpine) ## Server - Host: 46.225.91.159 @@ -43,7 +48,7 @@ bash /a0/usr/projects/leocrm/scripts/fast-deploy.sh full - Produktions-DB: postgresql+asyncpg://crm_user:86FkF5vJ_qKYgO6Myj0eQ4Dtm3Dyb1ge@postgres:5432/crm_db - Redis: redis://default:6VJ7pp8afXXZMnx0JztWFk-OYCLwJfX4@redis:6379/0 - SECRET_KEY: DoYnyh_UnvnYphX-qryiaIpQhm8JB39m_xkat9cNmrGpyKSSZvW9jF1tusIUSP5g -- MAIL_ENCRYPTION_KEY: test-mail-encryption-key (AES-256 Fernet Key für Mail-Passwort-Verschlüsselung, in Coolify als Environment Variable setzen) +- MAIL_ENCRYPTION_KEY: test-mail-encryption-key (AES-256 Fernet Key fuer Mail-Passwort-Verschluesselung, in Coolify als Environment Variable setzen) ## Coolify Resources - Project UUID: mzu7fvhtad82ujgmbsmyvxzm @@ -52,13 +57,13 @@ bash /a0/usr/projects/leocrm/scripts/fast-deploy.sh full ## Production Resource Recommendations -Die docker-compose.yaml hat Development-Defaults (PostgreSQL 512m, Redis 128m). Für Produktion mit 50+ Usern, 100k+ Entities, Agenten, Search und Knowledge müssen die Limits erhöht werden. +Die docker-compose.yaml hat Development-Defaults (PostgreSQL 512m, Redis 128m). Fuer Produktion mit 50+ Usern, 100k+ Entities, Agenten, Search und Knowledge muessen die Limits erhoeht werden. -| Service | Development | Production (50+ User) | Begründung | +| Service | Development | Production (50+ User) | Begruendung | |---|---|---|---| | **PostgreSQL RAM** | 512m | 1-2GB | pgvector HNSW + FTS + JSONB Snapshots + Outbox + AuditLog | | **PostgreSQL CPU** | 1.0 | 2.0 | Vector Search + FTS + normale CRM-Queries | -| **PostgreSQL Disk** | Named Volume | 50-100GB | Embeddings (768 dim × 100k = ~300MB), JSONB Snapshots, Outbox | +| **PostgreSQL Disk** | Named Volume | 50-100GB | Embeddings (768 dim x 100k = ~300MB), JSONB Snapshots, Outbox | | **Redis RAM** | 128m | 256-512m | WS Pub/Sub + Caching + Sessions + ARQ + Rate-Limiting | | **Redis CPU** | 0.5 | 1.0 | Pub/Sub + Cache + Queue | | **App (FastAPI) RAM** | Nicht limitiert | 512m-1GB | WebSocket Connections + Async Tasks | @@ -68,9 +73,9 @@ Die docker-compose.yaml hat Development-Defaults (PostgreSQL 512m, Redis 128m). ### Skalierung bei Bedarf -- **Read-Replicas:** Bei hohem Lese-Aufkommen (Search, FTS, Vector) können Read-Replicas für PostgreSQL eingerichtet werden. Schreib-Last (Outbox, EntityHistory, AuditLog) bleibt auf dem Master. -- **Mehr Worker:** Bei hohem Background-Job-Aufkommen können zusätzliche ARQ-Worker-Container gestartet werden. Queue-Prioritäten verhindern dass wichtige Jobs hinter langen Index-Jobs warten. -- **pgvector auslagern:** Bei sehr großen Datasets (>1M Embeddings) kann pgvector auf einen separaten PostgreSQL-Node ausgelagert werden. +- **Read-Replicas:** Bei hohem Lese-Aufkommen (Search, FTS, Vector) koennen Read-Replicas fuer PostgreSQL eingerichtet werden. Schreib-Last (Outbox, EntityHistory, AuditLog) bleibt auf dem Master. +- **Mehr Worker:** Bei hohem Background-Job-Aufkommen koennen zusaetzliche ARQ-Worker-Container gestartet werden. Queue-Prioritaeten verhindern dass wichtige Jobs hinter langen Index-Jobs warten. +- **pgvector auslagern:** Bei sehr grossen Datasets (>1M Embeddings) kann pgvector auf einen separaten PostgreSQL-Node ausgelagert werden. - **Redis Cluster:** Bei sehr hohem Cache-/Pub/Sub-Aufkommen kann Redis Cluster eingesetzt werden. ### LLM-Kosten-Management @@ -78,4 +83,24 @@ Die docker-compose.yaml hat Development-Defaults (PostgreSQL 512m, Redis 128m). - **Cost-Tracking:** Der zentrale LLM Client (Phase B.1) trackt Kosten pro Call. - **Budget-Limits:** Pro Agent (Phase F) und pro Tenant (Phase I). - **Cost-Dashboard:** Phase I zeigt LLM-Kosten pro Agent/Workflow/User. -- **Alerts:** Budget-Alerts bei Überschreitung konfigurierbarer Schwellwerte. +- **Alerts:** Budget-Alerts bei Ueberschreitung konfigurierbarer Schwellwerte. + +## ARQ Worker Cron-Jobs + +Der crm_worker Service fuehrt folgende Cron-Jobs aus: + +| Job | Schedule | Beschreibung | +|-----|----------|-------------| +| `auto_backup_job` | Taeglich 03:00 | Automatisches Backup (ruft scripts/backup.py auf) | +| `audit_retention_cleanup` | Taeglich 04:00 | Audit-Logs aelter als 365 Tage archivieren/loeschen | +| `cleanup_expired_trash` | Taeglich 05:00 | Soft-deleted Entities aelter als 90 Tage endgueltig loeschen | +| `outbox_cleanup` | Stuendlich | Outbox-Eintraege aelter als 30 Tage loeschen | +| `cleanup_expired_sessions` | Stuendlich | Abgelaufene Sessions aus Redis loeschen | + +## Container-Entrypoints + +| Datei | Service | Funktion | +|------|---------|----------| +| `prestart.sh` | crm_app | Alembic-Migrationen -> DB-Role-Passwoerter -> Plugin-Schema-Sync -> Admin-Seed -> uvicorn Start | +| `worker.sh` | crm_worker | ARQ Worker Start mit Cron-Jobs | +| `healthcheck.sh` | crm_app | HTTP /api/v1/health oder Redis-Ping | diff --git a/docs/infrastructure.md b/docs/infrastructure.md index b83c2ac..6bbf162 100644 --- a/docs/infrastructure.md +++ b/docs/infrastructure.md @@ -1,22 +1,192 @@ # Infrastructure Guide -> **Version:** 1.0 -> **Date:** 2026-07-29 +> **Version:** 2.0 +> **Date:** 2026-08-20 > **Applies to:** System administrators and DevOps engineers --- -## 1. PgBouncer Setup +## 1. Docker-Compose Stack -PgBouncer is a lightweight connection pooler for PostgreSQL. It reduces the overhead of establishing new database connections by reusing existing ones. +LeoCRM läuft als Docker-Compose-Stack mit 4 Services: -### Why PgBouncer? +| Service | Image | Beschreibung | +|---------|-------|-------------| +| **postgres** | `pgvector/pgvector:pg16` | PostgreSQL 16 mit pgvector Extension für Vector Search | +| **redis** | `redis:7-alpine` | Redis 7 für Sessions, Caching, Pub/Sub, ARQ Queue | +| **crm_app** | Multi-Stage Build | FastAPI Backend (uvicorn), API + WebSocket | +| **crm_worker** | Multi-Stage Build | ARQ Background Worker (Cron-Jobs, Queue Processing) | -- **Connection pooling** — Reduces PostgreSQL connection overhead -- **Resource efficiency** — Handles thousands of client connections with minimal resources -- **Transaction pooling** — Best for stateless applications like FastAPI -- **Session pooling** — For stateful connections -- **Statement pooling** — For specific use cases +### docker-compose.yaml Übersicht + +```yaml +services: + postgres: + image: pgvector/pgvector:pg16 + volumes: + - postgres_data:/var/lib/postgresql/data + healthcheck: + test: ["CMD-SHELL", "pg_isready -U crm_user -d crm_db"] + interval: 10s + timeout: 5s + retries: 5 + + redis: + image: redis:7-alpine + volumes: + - redis_data:/data + healthcheck: + test: ["CMD", "redis-cli", "ping"] + interval: 10s + timeout: 5s + retries: 5 + + crm_app: + build: . + command: ["./prestart.sh"] + depends_on: + postgres: { condition: service_healthy } + redis: { condition: service_healthy } + healthcheck: + test: ["CMD-SHELL", "bash /app/healthcheck.sh"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 15s + + crm_worker: + build: . + command: ["./worker.sh"] + depends_on: + postgres: { condition: service_healthy } + redis: { condition: service_healthy } +``` + +### Container-Entrypoints + +| Datei | Service | Funktion | +|------|---------|----------| +| `prestart.sh` | crm_app | Alembic-Migrationen → DB-Role-Passwörter → Plugin-Schema-Sync → Admin-Seed → uvicorn Start | +| `worker.sh` | crm_worker | ARQ Worker Start mit Cron-Jobs | +| `healthcheck.sh` | crm_app | HTTP `/api/v1/health` oder Redis-Ping | + +### Volumes + +| Volume | Service | Beschreibung | +|--------|---------|-------------| +| `postgres_data` | postgres | PostgreSQL Daten (persistent) | +| `redis_data` | redis | Redis Snapshot (persistent) | +| `app_storage` | crm_app | File Storage (DMS, Uploads) | + +--- + +## 2. Coolify Deployment + +LeoCRM ist über Coolify auf einem Hetzner VPS deployiert. + +### Server-Info + +| Eigenschaft | Wert | +|-------------|------| +| **Host** | 46.225.91.159 | +| **Hostname** | coolify-01 | +| **Provider** | Hetzner VPS | +| **Coolify URL** | https://server.media-on.de | +| **App URL** | https://crm.media-on.de | +| **App UUID** | xf7smknlger3hvkrsb910tui | +| **Container-Name** | Ändert sich bei jedem Coolify-Deploy (Suffix) | + +### Deploy-Methoden + +#### Frontend-Only Deploy (~20s) +```bash +bash /a0/usr/projects/leocrm/scripts/fast-deploy.sh frontend +``` +- Baut Frontend lokal, kopiert `dist/` direkt in den laufenden Container +- Kein Coolify-Rebuild, kein Docker-Image-Neubau +- Container wird nicht neu gestartet + +#### Full Deploy (~2min, für Backend-Änderungen) +```bash +bash /a0/usr/projects/leocrm/scripts/fast-deploy.sh full +``` +- Triggert Coolify-Rebuild über `deploy.py` +- Für Python-Code, Requirements, Migrations + +### Server-Container + +Auf dem Hetzner VPS laufen ~25 Docker-Container (Coolify + Services): +- Coolify Proxy (Traefik) +- Coolify Dashboard +- Coolify Database +- LeoCRM Stack (postgres, redis, crm_app, crm_worker) +- Weitere Coolify-managed Services + +Container-Übersicht auf dem Server: +```bash +ssh -i ~/.ssh/coolify-01-root root@46.225.91.159 'docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"' +``` + +--- + +## 3. ARQ Worker & Cron-Jobs + +Der `crm_worker` Service läuft ARQ (Async Redis Queue) für Background-Jobs. + +### Registrierte Cron-Jobs + +| Job | Schedule | Beschreibung | +|-----|----------|-------------| +| `auto_backup_job` | Täglich 03:00 | Automatisches Backup (ruft `scripts/backup.py` auf) | +| `audit_retention_cleanup` | Täglich 04:00 | Audit-Logs älter als 365 Tage archivieren/löschen | +| `cleanup_expired_trash` | Täglich 05:00 | Soft-deleted Entitäten älter als 90 Tage endgültig löschen | +| `outbox_cleanup` | Stündlich | Outbox-Einträge älter als 30 Tage löschen | +| `cleanup_expired_sessions` | Stündlich | Abgelaufene Sessions aus Redis löschen | + +### ARQ Worker Konfiguration + +```python +# app/core/worker.py +class WorkerSettings: + functions = [...] + cron_jobs = [ + cron(auto_backup_job, hour=3, minute=0), + cron(audit_retention_cleanup, hour=4, minute=0), + cron(cleanup_expired_trash, hour=5, minute=0), + cron(outbox_cleanup, hour={0,6,12,18}, minute=0), + cron(cleanup_expired_sessions, hour={0,6,12,18}, minute=0), + ] + max_jobs = 10 + job_timeout = 300 + queue_name = "arq:queue" +``` + +### Worker-Stats abfragen + +```bash +# Queue-Länge +redis-cli ZCARD arq:queue + +# Aktive Worker +redis-cli KEYS "arq:heartbeat:*" + +# Via API (Admin) +curl -b "leocrm_session=" https://crm.media-on.de/api/v1/system/dashboard | jq '.worker' +``` + +--- + +## 4. PgBouncer Setup + +PgBouncer ist ein leichter Connection-Pooler für PostgreSQL. Er reduziert den Overhead neuer Datenbankverbindungen durch Wiederverwendung bestehender Verbindungen. + +### Warum PgBouncer? + +- **Connection pooling** — Reduziert PostgreSQL Connection-Overhead +- **Resource efficiency** — Verwaltet tausende Client-Verbindungen mit minimalen Ressourcen +- **Transaction pooling** — Optimal für stateless Anwendungen wie FastAPI +- **Session pooling** — Für stateful Verbindungen +- **Statement pooling** — Für spezifische Use-Cases ### Installation @@ -28,7 +198,7 @@ apt-get update && apt-get install -y pgbouncer pgbouncer --version ``` -### Configuration +### Konfiguration Create `/etc/pgbouncer/pgbouncer.ini`: @@ -43,8 +213,6 @@ listen_port = 6432 unix_socket_dir = /var/run/pgbouncer # Authentication -# Use md5 for password-based auth -# Use trust for local development auth_type = md5 auth_file = /etc/pgbouncer/userlist.txt @@ -67,7 +235,6 @@ log_pooler_errors = 1 stats_period = 60 # Security -# Only allow connections from localhost and Docker network listen_backlog = 128 ``` @@ -167,25 +334,25 @@ echo "SHOW SERVERS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer --- -## 2. Audit Log Partitioning +## 5. Audit Log Partitioning -The `audit_log` table can grow very large over time. PostgreSQL table partitioning helps manage this by splitting the table into smaller, more manageable pieces. +Die `audit_log` Tabelle kann sehr groß werden. PostgreSQL Table Partitioning hilft durch Aufteilung in kleinere, verwaltbare Stücke. -### Why Partition? +### Warum Partitioning? -- **Faster queries** — Queries only scan relevant partitions -- **Easier maintenance** — Drop old partitions instead of DELETE -- **Better vacuum** — Each partition is vacuumed independently -- **Improved performance** — Smaller indexes per partition +- **Schnellere Queries** — Queries scannen nur relevante Partitionen +- **Einfachere Wartung** — Alte Partitionen droppen statt DELETE +- **Besseres Vacuum** — Jede Partition wird unabhängig gevacuumt +- **Bessere Performance** — Kleinere Indexes pro Partition -### Partitioning Strategy +### Partitioning-Strategie -We use **monthly range partitioning** on the `created_at` column: +Wir verwenden **monatliches Range-Partitioning** auf der `created_at` Spalte: ```sql --- Each partition covers one month --- Partition name: audit_log_YYYY_MM --- Example: audit_log_2026_01, audit_log_2026_02, ... +-- Jede Partition deckt einen Monat ab +-- Partitions-Name: audit_log_YYYY_MM +-- Beispiel: audit_log_2026_01, audit_log_2026_02, ... ``` ### Creating the Partitioned Table @@ -221,46 +388,6 @@ CREATE INDEX idx_audit_log_2026_01_tenant ON audit_log_2026_01 (tenant_id); CREATE INDEX idx_audit_log_2026_01_action ON audit_log_2026_01 (action); CREATE INDEX idx_audit_log_2026_01_entity ON audit_log_2026_01 (entity_type, entity_id); CREATE INDEX idx_audit_log_2026_01_created ON audit_log_2026_01 (created_at DESC); - -CREATE INDEX idx_audit_log_2026_02_tenant ON audit_log_2026_02 (tenant_id); -CREATE INDEX idx_audit_log_2026_02_action ON audit_log_2026_02 (action); -CREATE INDEX idx_audit_log_2026_02_entity ON audit_log_2026_02 (entity_type, entity_id); -CREATE INDEX idx_audit_log_2026_02_created ON audit_log_2026_02 (created_at DESC); - -CREATE INDEX idx_audit_log_2026_03_tenant ON audit_log_2026_03 (tenant_id); -CREATE INDEX idx_audit_log_2026_03_action ON audit_log_2026_03 (action); -CREATE INDEX idx_audit_log_2026_03_entity ON audit_log_2026_03 (entity_type, entity_id); -CREATE INDEX idx_audit_log_2026_03_created ON audit_log_2026_03 (created_at DESC); -``` - -### Migrating Existing Data - -```sql --- Step 1: Create the partitioned table --- (see script above) - --- Step 2: Insert existing data -INSERT INTO audit_log_partitioned ( - id, tenant_id, user_id, action, entity_type, - entity_id, changes, ip_address, user_agent, created_at -) -SELECT id, tenant_id, user_id, action, entity_type, - entity_id, changes, ip_address, user_agent, created_at -FROM audit_log; - --- Step 3: Verify data integrity -SELECT COUNT(*) FROM audit_log_partitioned; -SELECT COUNT(*) FROM audit_log; - --- Step 4: Rename tables -ALTER TABLE audit_log RENAME TO audit_log_old; -ALTER TABLE audit_log_partitioned RENAME TO audit_log; - --- Step 5: Update sequences and indexes --- (handled by the partitioned table definition) - --- Step 6: Drop old table after verification --- DROP TABLE audit_log_old; ``` ### Automating Partition Creation @@ -274,54 +401,19 @@ psql -h localhost -U leocrm -d crm_db -f scripts/setup_audit_partitioning.sql ### Cron Job for Partition Maintenance -Add to crontab to create partitions automatically: - ```bash # Run on the 1st of each month at 2 AM 0 2 1 * * /usr/bin/psql -h localhost -U leocrm -d crm_db -c "SELECT create_monthly_audit_partition();" ``` -### Querying Partitioned Data - -```sql --- Query a specific month (fast, only scans one partition) -SELECT * FROM audit_log -WHERE created_at >= '2026-01-01' - AND created_at < '2026-02-01' - AND tenant_id = '...'; - --- Query across months (scans multiple partitions) -SELECT * FROM audit_log -WHERE created_at >= '2026-01-01' - AND created_at < '2026-03-01' - AND action = 'permission_grant'; - --- Check which partitions will be scanned -EXPLAIN SELECT * FROM audit_log -WHERE created_at >= '2026-01-01' - AND created_at < '2026-02-01'; -``` - ### Dropping Old Partitions ```sql -- Drop partitions older than retention period DROP TABLE IF EXISTS audit_log_2025_01; DROP TABLE IF EXISTS audit_log_2025_02; --- ... - --- Or use a function -SELECT drop_old_audit_partitions(12); -- Keep last 12 months ``` -### Performance Considerations - -- **Index each partition** — Don't rely on parent table indexes -- **Use `created_at` in WHERE** — Always filter by date for partition pruning -- **Monitor partition count** — Too many partitions can slow planning -- **Archive old partitions** — Consider moving to cheaper storage -- **Vacuum partitions** — Each partition needs independent vacuum - ### Monitoring Partition Health ```sql @@ -340,18 +432,11 @@ SELECT FROM pg_catalog.pg_stat_user_tables WHERE relname LIKE 'audit_log_%' ORDER BY relname; - --- List all partitions -SELECT - inhrelid::regclass AS partition_name -FROM pg_catalog.pg_inherits -WHERE inhparent = 'audit_log'::regclass -ORDER BY partition_name; ``` --- -## 3. Backup and Recovery +## 6. Backup and Recovery ### Database Backup @@ -361,11 +446,19 @@ pg_dump -h localhost -U leocrm -d crm_db -F c -f /backups/crm_db_$(date +%Y%m%d) # Backup with compression pg_dump -h localhost -U leocrm -d crm_db -F c -Z 9 -f /backups/crm_db_$(date +%Y%m%d).dump.gz - -# Backup specific schema only -pg_dump -h localhost -U leocrm -d crm_db -n public -F c -f /backups/crm_db_schema_$(date +%Y%m%d).dump ``` +### Automated Backup + +LeoCRM hat einen automatisierten Backup via ARQ Cron-Job: + +- **Job:** `auto_backup_job` (täglich 03:00 Uhr) +- **Script:** `scripts/backup.py` (pg_dump + files) +- **Konfiguration:** Settings → Backup (backup_enabled, backup_interval, backup_retention_days, backup_destination) +- **API:** `POST /api/v1/system-settings/backup-now` (manueller Trigger) +- **History:** `GET /api/v1/system-settings/backup-history` (letzte 10 Backups) +- **Bei Fehler:** System-Message an Communication-System + ### Database Restore ```bash @@ -376,13 +469,11 @@ pg_restore -h localhost -U leocrm -d crm_db -c /backups/crm_db_20260701.dump pg_restore -h localhost -U leocrm -d crm_db -j 4 -c /backups/crm_db_20260701.dump ``` -### Automated Backup Script - -See `scripts/backup.py` for the automated backup solution. +See `scripts/restore.py` for the automated restore solution. --- -## 4. Monitoring and Alerts +## 7. Monitoring and Alerts ### Key Metrics @@ -393,6 +484,8 @@ See `scripts/backup.py` for the automated backup solution. | Cache hit ratio | > 95% | < 90% | | Partition size | < 10GB | > 50GB | | PgBouncer pool usage | < 80% | > 90% | +| Worker queue length | < 100 | > 500 | +| Error rate | < 1% | > 5% | ### Health Checks @@ -405,4 +498,32 @@ psql -h localhost -U leocrm -d crm_db -c "SELECT count(*) FROM audit_log WHERE c # Check database size psql -h localhost -U leocrm -d crm_db -c "SELECT pg_size_pretty(pg_database_size('crm_db'));" + +# Via API (Admin) +curl -b "leocrm_session=" https://crm.media-on.de/api/v1/system/dashboard | jq . ``` + +--- + +## 8. Production Resource Recommendations + +Die `docker-compose.yaml` hat Development-Defaults. Für Produktion mit 50+ Usern müssen die Limits erhöht werden. + +| Service | Development | Production (50+ User) | Begründung | +|---|---|---|---| +| **PostgreSQL RAM** | 512m | 1-2GB | pgvector HNSW + FTS + JSONB Snapshots + Outbox + AuditLog | +| **PostgreSQL CPU** | 1.0 | 2.0 | Vector Search + FTS + normale CRM-Queries | +| **PostgreSQL Disk** | Named Volume | 50-100GB | Embeddings (768 dim × 100k = ~300MB), JSONB Snapshots, Outbox | +| **Redis RAM** | 128m | 256-512m | WS Pub/Sub + Caching + Sessions + ARQ + Rate-Limiting | +| **Redis CPU** | 0.5 | 1.0 | Pub/Sub + Cache + Queue | +| **App (FastAPI) RAM** | Nicht limitiert | 512m-1GB | WebSocket Connections + Async Tasks | +| **App CPU** | Nicht limitiert | 1-2 CPUs | API + WS + LLM-Streaming | +| **Worker (ARQ) RAM** | Nicht limitiert | 256-512m | Background Jobs (Indexierung, Agent-Runs, Extraction) | +| **Worker CPU** | Nicht limitiert | 1-2 CPUs | LLM-Calls + Embedding + Text-Extraction | + +### Skalierung bei Bedarf + +- **Read-Replicas:** Bei hohem Lese-Aufkommen (Search, FTS, Vector) können Read-Replicas für PostgreSQL eingerichtet werden. +- **Mehr Worker:** Bei hohem Background-Job-Aufkommen können zusätzliche ARQ-Worker-Container gestartet werden. +- **pgvector auslagern:** Bei sehr großen Datasets (>1M Embeddings) kann pgvector auf einen separaten PostgreSQL-Node ausgelagert werden. +- **Redis Cluster:** Bei sehr hohem Cache-/Pub/Sub-Aufkommen kann Redis Cluster eingesetzt werden. diff --git a/docs/monitoring.md b/docs/monitoring.md index 9b8f2b4..a80e73a 100644 --- a/docs/monitoring.md +++ b/docs/monitoring.md @@ -1,5 +1,8 @@ # Monitoring und Logging +> **Version:** 2.0 +> **Date:** 2026-08-20 + ## Health Endpoints ### `/health/live` — Liveness Probe @@ -37,6 +40,43 @@ curl https://crm.media-on.de/api/v1/health # → {"status":"healthy","version":"1.0.0","checks":{...}} ``` +--- + +## System Dashboard (Admin-only) + +Das System Dashboard ist die zentrale Monitoring-Oberfläche für Administratoren. + +- **URL:** `/system-dashboard` (Frontend, Admin-only) +- **API:** `GET /api/v1/system/dashboard` (Admin-only) +- **Alerts:** `GET /api/v1/system/alerts` (Admin-only) + +### Dashboard-Inhalte + +| Bereich | Metriken | +|---------|----------| +| **System Health** | Overall status (healthy/degraded/down) | +| **Database** | Connections, Table Count, DB Size (bytes/human) | +| **Redis** | Connected Clients, Used Memory, Peak Memory, Uptime | +| **Worker** | Queue Length, Active Workers, Status (up/degraded/down) | +| **API Stats** | Total Requests, Error Count, Error Rate, Avg Response Time | +| **Plugins** | Total Discovered, Active Plugins (name, version, is_core) | +| **Storage** | Disk Usage (total/used/free), File Count, Status | +| **Alert Feed** | System Messages aus Communication-System | + +### Alert-Dispatch + +Bei Problemen (DB down, Redis down, Worker down, High Error Rate) sendet das System Dashboard automatisch eine System-Message an das Communication-System (`post_system_message()`). Diese erscheint im Alert-Feed des Dashboards und in den Benachrichtigungen der Admins. + +```bash +# Dashboard abfragen +curl -b "leocrm_session=" https://crm.media-on.de/api/v1/system/dashboard | jq . + +# Aktive Alerts abfragen +curl -b "leocrm_session=" https://crm.media-on.de/api/v1/system/alerts | jq . +``` + +--- + ## Metrics Endpoint ### `/api/v1/metrics` — Prometheus Metrics @@ -55,16 +95,22 @@ Metriken: - `leocrm_outbox_pending` — Pending Outbox Events - `leocrm_outbox_failed` — Failed Outbox Events -## Externes Monitoring +--- -### Empfohlene Tools +## Alerting via Notification-System -- **Uptime Kuma** — Einfache Uptime-Überwachung -- **Prometheus + Grafana** — Full Metrics Dashboard -- **Sentry** — Error Tracking -- **Coolify Health Monitoring** — Eingebaut in Coolify +LeoCRM nutzt das integrierte Notification-System für Alerting. Bei System-Problemen werden automatisch System-Messages an das Communication-System gesendet. -### Alerting Regeln +### Alert-Quellen + +| Quelle | Trigger | Ziel | +|--------|---------|------| +| System Dashboard | DB/Redis/Worker down, High Error Rate | Communication-System (Alert Feed) | +| Backup Job | Backup fehlgeschlagen | Communication-System + Audit Log | +| ARQ Worker | Job fehlgeschlagen, Queue überlastet | Communication-System | +| Outbox | Delivery failed | Audit Log + Communication-System | + +### Alert-Typen | Alert | Bedingung | Severity | |-------|-----------|----------| @@ -77,6 +123,51 @@ Metriken: | DB Pool Exhausted | Pool checked_out = pool_size | Critical | | Outbox Backlog | Pending > 100 | Warning | | Worker Down | Worker heartbeat fehlt | Critical | +| Backup Failed | Backup job returned error | Critical | +| Queue Overload | Queue length > 500 | Warning | + +### Alert-Empfang + +Alerts erscheinen: +1. Im System Dashboard Alert-Feed (`/system-dashboard`) +2. In den Benachrichtigungen der Admins (Notification-System) +3. Im Audit Log (für Backup/Worker/Outbox Alerts) + +--- + +## Incident Response Runbook + +Siehe `docs/incident-response-runbook.md` für detaillierte Notfall-Prozeduren. + +### Schnell-Referenz + +| Incident | Erste Maßnahme | escalation | +|----------|---------------|------------| +| **Server-Ausfall** | Coolify Restart → Health-Check → System-Message | Hetzner Support | +| **DB-Crash** | PostgreSQL Restart → Migration-Check → Backup-Restore | Coolify DB Restart | +| **Redis-Crash** | Redis Restart → Session-Check | Coolify Service Restart | +| **Security-Breach** | Logs prüfen → Password-Reset → Audit-Log Export | Incident Response Team | +| **Backup Failed** | Backup-Log prüfen → Manueller Retry → Storage prüfen | Admin | +| **Worker Down** | Worker-Container Restart → Queue prüfen | Coolify Service Restart | + +### Incident Response Schritte + +1. **Erkennen** — Alert im System Dashboard oder externes Monitoring +2. **Eingrenzen** — Health-Checks, Logs, Metrics prüfen +3. **Beheben** — Restart, Restore, Konfiguration anpassen +4. **Verifizieren** — Health-Check grün, System Dashboard ok +5. **Dokumentieren** — Audit Log Eintrag, Post-Mortem bei Critical + +--- + +## Externes Monitoring + +### Empfohlene Tools + +- **Uptime Kuma** — Einfache Uptime-Überwachung +- **Prometheus + Grafana** — Full Metrics Dashboard +- **Sentry** — Error Tracking +- **Coolify Health Monitoring** — Eingebaut in Coolify ### Coolify Health Check Konfiguration @@ -111,6 +202,8 @@ scrape_configs: credentials: '' ``` +--- + ## Strukturiertes Logging Alle API-Requests werden strukturiert geloggt: @@ -124,7 +217,7 @@ Alle API-Requests werden strukturiert geloggt: "tenant_id": "bfe4d09e-...", "event": "api_request", "level": "info", - "timestamp": "2026-07-29T16:00:00Z" + "timestamp": "2026-08-20T14:00:00Z" } ``` @@ -132,3 +225,17 @@ Log-Level: - `info` — Normale API-Requests - `warning` — Langsame Requests, Permission denied - `error` — 500er Fehler, Exceptions + +### Error Tracking + +```python +# app/core/monitoring.py +record_error( + error_code="DB_CONNECTION_FAILED", + message="Database connection lost", + context={"host": "postgres", "port": 5432}, + trace_id="abc-123", +) +``` + +Fehler werden mit `trace_id` getrackt für Korrelation über Services hinweg.