Leopoldadmin/leocrm
1
0
Fork 0
Code Issues 363 Pull Requests Actions 1 Packages Projects Releases 1 Wiki Activity

docs: punkt 11 (documentation) — README, infrastructure, monitoring, admin-guide, deploy-guide, api-docs, PROGRESS, ENTERPRISE_READINESS_PLAN all updated

Browse Source
This commit is contained in:
Agent Zero
2026-08-20 14:03:35 +02:00
parent e6790d9b81
commit f79eb9354a
8 changed files with 853 additions and 242 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/monitoring.md
+115 -8
View File
@@ -1,5 +1,8 @@
# Monitoring und Logging
> **Version:** 2.0
> **Date:** 2026-08-20
## Health Endpoints
### `/health/live` — Liveness Probe
@@ -37,6 +40,43 @@ curl https://crm.media-on.de/api/v1/health
# → {"status":"healthy","version":"1.0.0","checks":{...}}
```
---
## System Dashboard (Admin-only)
Das System Dashboard ist die zentrale Monitoring-Oberfläche für Administratoren.
- **URL:** `/system-dashboard` (Frontend, Admin-only)
- **API:** `GET /api/v1/system/dashboard` (Admin-only)
- **Alerts:** `GET /api/v1/system/alerts` (Admin-only)
### Dashboard-Inhalte
| Bereich | Metriken |
|---------|----------|
| **System Health** | Overall status (healthy/degraded/down) |
| **Database** | Connections, Table Count, DB Size (bytes/human) |
| **Redis** | Connected Clients, Used Memory, Peak Memory, Uptime |
| **Worker** | Queue Length, Active Workers, Status (up/degraded/down) |
| **API Stats** | Total Requests, Error Count, Error Rate, Avg Response Time |
| **Plugins** | Total Discovered, Active Plugins (name, version, is_core) |
| **Storage** | Disk Usage (total/used/free), File Count, Status |
| **Alert Feed** | System Messages aus Communication-System |
### Alert-Dispatch
Bei Problemen (DB down, Redis down, Worker down, High Error Rate) sendet das System Dashboard automatisch eine System-Message an das Communication-System (`post_system_message()`). Diese erscheint im Alert-Feed des Dashboards und in den Benachrichtigungen der Admins.
```bash
# Dashboard abfragen
curl -b "leocrm_session=<session>" https://crm.media-on.de/api/v1/system/dashboard | jq .
# Aktive Alerts abfragen
curl -b "leocrm_session=<session>" https://crm.media-on.de/api/v1/system/alerts | jq .
```
---
## Metrics Endpoint
### `/api/v1/metrics` — Prometheus Metrics
@@ -55,16 +95,22 @@ Metriken:
- `leocrm_outbox_pending` — Pending Outbox Events
- `leocrm_outbox_failed` — Failed Outbox Events
## Externes Monitoring
---
### Empfohlene Tools
## Alerting via Notification-System
- **Uptime Kuma** — Einfache Uptime-Überwachung
- **Prometheus + Grafana** — Full Metrics Dashboard
- **Sentry** — Error Tracking
- **Coolify Health Monitoring** — Eingebaut in Coolify
LeoCRM nutzt das integrierte Notification-System für Alerting. Bei System-Problemen werden automatisch System-Messages an das Communication-System gesendet.
### Alerting Regeln
### Alert-Quellen
| Quelle | Trigger | Ziel |
|--------|---------|------|
| System Dashboard | DB/Redis/Worker down, High Error Rate | Communication-System (Alert Feed) |
| Backup Job | Backup fehlgeschlagen | Communication-System + Audit Log |
| ARQ Worker | Job fehlgeschlagen, Queue überlastet | Communication-System |
| Outbox | Delivery failed | Audit Log + Communication-System |
### Alert-Typen
| Alert | Bedingung | Severity |
|-------|-----------|----------|
@@ -77,6 +123,51 @@ Metriken:
| DB Pool Exhausted | Pool checked_out = pool_size | Critical |
| Outbox Backlog | Pending > 100 | Warning |
| Worker Down | Worker heartbeat fehlt | Critical |
| Backup Failed | Backup job returned error | Critical |
| Queue Overload | Queue length > 500 | Warning |
### Alert-Empfang
Alerts erscheinen:
1. Im System Dashboard Alert-Feed (`/system-dashboard`)
2. In den Benachrichtigungen der Admins (Notification-System)
3. Im Audit Log (für Backup/Worker/Outbox Alerts)
---
## Incident Response Runbook
Siehe `docs/incident-response-runbook.md` für detaillierte Notfall-Prozeduren.
### Schnell-Referenz
| Incident | Erste Maßnahme | escalation |
|----------|---------------|------------|
| **Server-Ausfall** | Coolify Restart → Health-Check → System-Message | Hetzner Support |
| **DB-Crash** | PostgreSQL Restart → Migration-Check → Backup-Restore | Coolify DB Restart |
| **Redis-Crash** | Redis Restart → Session-Check | Coolify Service Restart |
| **Security-Breach** | Logs prüfen → Password-Reset → Audit-Log Export | Incident Response Team |
| **Backup Failed** | Backup-Log prüfen → Manueller Retry → Storage prüfen | Admin |
| **Worker Down** | Worker-Container Restart → Queue prüfen | Coolify Service Restart |
### Incident Response Schritte
1. **Erkennen** — Alert im System Dashboard oder externes Monitoring
2. **Eingrenzen** — Health-Checks, Logs, Metrics prüfen
3. **Beheben** — Restart, Restore, Konfiguration anpassen
4. **Verifizieren** — Health-Check grün, System Dashboard ok
5. **Dokumentieren** — Audit Log Eintrag, Post-Mortem bei Critical
---
## Externes Monitoring
### Empfohlene Tools
- **Uptime Kuma** — Einfache Uptime-Überwachung
- **Prometheus + Grafana** — Full Metrics Dashboard
- **Sentry** — Error Tracking
- **Coolify Health Monitoring** — Eingebaut in Coolify
### Coolify Health Check Konfiguration
@@ -111,6 +202,8 @@ scrape_configs:
credentials: '<admin-token>'
```
---
## Strukturiertes Logging
Alle API-Requests werden strukturiert geloggt:
@@ -124,7 +217,7 @@ Alle API-Requests werden strukturiert geloggt:
"tenant_id": "bfe4d09e-...",
"event": "api_request",
"level": "info",
"timestamp": "2026-07-29T16:00:00Z"
"timestamp": "2026-08-20T14:00:00Z"
}
```
@@ -132,3 +225,17 @@ Log-Level:
- `info` — Normale API-Requests
- `warning` — Langsame Requests, Permission denied
- `error` — 500er Fehler, Exceptions
### Error Tracking
```python
# app/core/monitoring.py
record_error(
error_code="DB_CONNECTION_FAILED",
message="Database connection lost",
context={"host": "postgres", "port": 5432},
trace_id="abc-123",
)
```
Fehler werden mit `trace_id` getrackt für Korrelation über Services hinweg.