485fbd9877
Phase 8.3+8.4: Restore-Test Script und Coolify-Endabnahme
...
8.3 Restore-Test:
- restore_test.sh: PostgreSQL Backup restore, Migrationen, Data Integrity, RLS Re-test
- Prueft Alembic Version, Table Count, RLS >= 100, Contacts > 0
- RLS Re-test: 0 rows ohne/fake tenant context
- Erfordert TEST_DATABASE_URL (separate Test-DB)
8.4 Coolify-Endabnahme (live verifiziert):
- API healthy: DB up, Redis up, Worker up
- Worker healthy: running:healthy
- Login: admin@media-on.de , admin, Default Org
- Workspace Wechsel: 1 Workspace, Context modules mit is_visible
- DMS Upload + Download: HTTP 200, Content korrekt
- MCP Read: 1 Tool (call_crm_api), Auth api-token
- Outbox: 5 published events
- Token CRUD: Create, List, Revoke (204)
2026-08-03 15:50:11 +02:00
f4364f30e0
Phase 8.1+8.2: CI Pipeline und Migrations-Release-Gate
...
8.1 Merge-CI:
- Backend Tests und Frontend Tests zu ci_pipeline.sh hinzugefuegt
- Migration Hash Check (<=0092) mit check_migration_hashes.py
- npm ci --legacy-peer-deps in Forgejo Workflow und ci_pipeline.sh
- 93 Migration-Hashes generiert und verifiziert
8.2 Migrations-Release-Gate:
- migration_release_gate.sh: Fresh Install, Schema Snapshot, RLS/Grants Check, Cross-Tenant Test, Data Integrity
- Prueft leere DB Installation mit Alembic Head + Plugin-Migrationen
- Verifiziert RLS >= 100 Tabellen, 4 DB-Rollen, kein BYPASSRLS auf crm_api
- Cross-Tenant: 0 rows ohne/fake tenant context
2026-08-03 15:49:03 +02:00
0260f3410d
Phase 7: Plugin-Gate, Event-Envelope, Pro-Handler Outbox-Verarbeitung
...
7.1 Plugin-Gate korrigiert:
- require_active_plugin nutzt current_user fuer tenant_id statt current_setting()
- Keine neue DB-Session mehr — nutzt bestehende get_db Dependency
- Fail-closed bei Fehlern
7.4 Einheitlicher Event-Envelope:
- Sauberes Envelope mit event_id, event_name, tenant_id, aggregate_type, aggregate_id, occurred_at, correlation_id, schema_version, data
- Keine _-Praefixe mehr im payload
- Handler empfangen envelope statt rohes payload
7.6 Verarbeitung pro Handler:
- Globaler consumer_inbox Check entfernt
- Pro-Handler Idempotency: outbox_deliveries pruefen ob Handler bereits erfolgreich
- Bereits erfolgreiche Handler werden uebersprungen
- consumer_inbox pro Handler geschrieben
7.7 no_handlers: Bereits implementiert (terminaler Status)
7.8 Cron-Jobs: Bereits mit Redis SET NX Locking implementiert
Tests: 23/23 Outbox-Tests bestanden
2026-08-03 15:20:06 +02:00
8d82df3076
Fix: LocalStorage top-level import in DMS routes
Check Cross-Plugin Imports / check (push) Has been cancelled
2026-08-03 15:08:13 +02:00
8b683c7da7
Phase 6.5 Fix: DMS Download Endpoint fuer alle Dateitypen
...
Check Cross-Plugin Imports / check (push) Has been cancelled
- GET /api/v1/dms/files/{file_id}/download streamt alle Dateitypen
- FileResponse fuer LocalStorage (automatisches Streaming)
- StreamingResponse Fallback fuer S3
- Prueft dms:read Permission und entity access
2026-08-03 15:02:39 +02:00
29d55cb187
Phase 6: DMS & Attachments — Streaming, Deduplikation, API-Bereinigung
...
Check Cross-Plugin Imports / check (push) Has been cancelled
6.4 Upload streamen:
- attachment_service.save_attachment: Streamt in 1MB Chunks statt await file.read()
- routes/attachments.py: Uebergibt UploadFile direkt statt bytes
6.5 Download streamen:
- DMS preview_file: FileResponse fuer LocalStorage (automatisches Streaming)
- Kein storage.read() mehr fuer LocalStorage
6.6 Tenantlokale Deduplikation:
- DMS Upload: Prueft content_hash vor Erstellung, wiederverwendet existierendes File
- attachment_service: Dedup bereits vorhanden, jetzt mit Streaming kompatibel
- Migration 0098: Partial Unique Index (tenant_id, content_hash) WHERE content_hash IS NOT NULL AND deleted_at IS NULL
6.7 API-Ausgabe bereinigt:
- attachment_service: storage_path und content_hash aus API-Ausgaben entfernt
- DMS routes: content_hash aus 4 API-Endpunkten entfernt
Tests: 54/54 bestanden (17 Workspace + 13 API Token + 24 Command)
2026-08-03 14:21:43 +02:00
ff975ca0a6
Fix: MCP list_mcp_tools + config Routes auf Bearer-Auth umstellen
Check Cross-Plugin Imports / check (push) Has been cancelled
2026-08-03 14:12:44 +02:00
4efdc8e036
Fix: Migration 0097 — api_tokens.updated_at Spalte hinzufuegen
...
ApiToken Modell erbt von TenantMixin (TimestampMixin) das updated_at erwartet.
Migration 0001 hat api_tokens ohne updated_at erstellt.
Migration 0083 hat deleted_at hinzugefuegt aber updated_at verpasst.
2026-08-03 14:10:04 +02:00
8ad0a19f25
Phase 5: AI/MCP Bearer-Auth + Delegationstoken + Audit
...
Check Cross-Plugin Imports / check (push) Has been cancelled
5.1 Delegationstoken (app/core/delegation_token.py):
- HMAC-SHA256 signiert mit SECRET_KEY, max 60s Lifetime
- Payload: user_id, tenant_id, agent_id, audience, expires_at, token_id
- Statelose Verifikation, Audience-Check, Expiry-Check
5.2 MCP Bearer-Auth:
- app/core/api_token.py: Token Service (create, verify, revoke, list)
- app/deps.py: get_current_user_bearer + get_current_user_or_bearer
- app/routes/api_tokens.py: Token CRUD Routes (create, list, revoke)
- MCP Server Routes: get_current_user_or_bearer akzeptiert Session + Bearer
5.3 Methodenrechte:
- MCP nutzt bereits mcp:read/mcp:write basierend auf tool_def.required_permission
5.5 Audit:
- MCP Tool-Ausfuehrung wird protokolliert (log_audit mit correlation_id)
Tests: 13/13 bestanden (7 API Token + 6 Delegation Token)
2026-08-03 14:06:55 +02:00
ea797b033a
Phase 4.5+4.6: Modul-Konfiguration pro Workspace + Sidebar useMemo Fix
...
4.5 Modul-Konfiguration pro Workspace:
- WorkspaceManager: Config-Editor pro Modul (JSON textarea)
- Pro Modul kann JSON config bearbeitet werden (z.B. sichtbare Ordner-IDs)
- Generisch: jedes Modul definiert selbst was in seiner config steht
4.6 Bugfixes:
- Sidebar useMemo: isModuleVisible zu Abhaengigkeiten hinzugefuegt
- Bei Workspacewechsel wird Sidebar jetzt sofort neu berechnet
Tests: 17 Backend + 13 Frontend = 30/30 bestanden
2026-08-03 13:58:27 +02:00
07d4587499
Plan anpassen: 4.5/4.6 entfernt, neue generelle 4.5 Modul-Konfiguration pro Workspace
2026-08-03 13:55:14 +02:00
3eb11b1745
Phase 1: Migrationsaudit + Forward-Migrationen 0093-0096
...
Audit (docs/migration_history_audit.md):
- files.size_bytes: INTEGER (Alembic) vs BIGINT (Produktion/Plugin)
- GIN-Indizes: Fehlendes USING GIN in Alembic 0002
- guest_users: ix_guest_users_email_tenant fehlt UNIQUE in Alembic 0059
- plugins.name: Doppelter Unique-Index in Produktion
Forward-Migrationen:
- 0093: files.size_bytes INTEGER → BIGINT
- 0094: GIN-Indizes reparieren + plugins.name doppelten Index entfernen
- 0095: guest_users email+tenant_id UNIQUE INDEX (mit Dubletten-Check)
- 0096: Workspace tenant_integrity (tenant-bound FKs)
Tests: 41/41 bestanden (17 Workspace + 24 Command)
Alembic Head: 0096
2026-08-03 13:29:16 +02:00
a760a759eb
Phase 0+3: Stand sichern, alte Doku einfrieren, doppelte Command-Struktur entfernen
...
Phase 0:
- Git Tag: pre-recovery-current (3cbf921
2026-08-03 13:25:48 +02:00
3cbf92191e
Reparaturplan Fixes: Widget workspace_id check, total bug, context is_visible, permissions, fallbacks
...
Check Cross-Plugin Imports / check (push) Has been cancelled
Backend:
- Widget total: 0 bug fixed (now returns len(widgets))
- Widget update/delete: now verifies workspace_id + tenant_id (was only tenant_id)
- Workspace context: returns all modules with is_visible flag (was only visible modules)
- is_workspace_manager() removed (Plan 4.2: no manager checks)
- seed_default_workspace: removed hardcoded modules (Plan 4.7: no hardcoded tiles)
- Workspace permissions registered in CORE_PERMISSIONS (Plan 2.3)
Frontend:
- Permission fallback removed: Sidebar/TopBar show nothing while loading (Plan 2.4)
- workspaceStore isModuleVisible: fail-closed when isSystemAdmin undefined
- WorkspaceManager: AVAILABLE_MODULES replaced with dynamic core+plugin items (Plan 4.4)
Tests:
- 17 backend tests (removed is_workspace_manager test, adapted widget/context tests)
- 13 frontend tests (added undefined-isSystemAdmin test, adapted visibility tests)
2026-08-03 12:44:02 +02:00
9f41da3d10
Update SANIERUNGS_FORTSCHRITT.md: Phase 6 Workspaces abgeschlossen
2026-08-03 03:45:52 +02:00
310a9f0542
Phase 6: Workspaces — Widget CRUD, Manager-Check, Cross-Tenant, Zustand Store, Settings Route
...
Backend:
- Widget CRUD: get_widgets, create_widget, update_widget, delete_widget
- Manager role check: is_workspace_manager
- Cross-tenant validation: verify_user_same_tenant (UserTenant)
- Default workspace seeding: seed_default_workspace with 12 standard modules
- Set user default workspace: set_user_default_workspace
- Fix create_workspace default uniqueness (unset others before insert)
- Widget CRUD routes: GET/POST/PUT/DELETE /{workspace_id}/widgets
- Set-default route: POST /{workspace_id}/set-default
- Cross-tenant validation in assign_user route
Frontend:
- workspaceStore (Zustand): central state with sessionStorage persistence
- API client interceptor: X-Workspace-ID header on all requests
- useWorkspace hook refactored to use workspaceStore
- Widget API hooks: useWorkspaceWidgets, useCreateWorkspaceWidget, etc.
- useSetDefaultWorkspace hook
- Settings route: /settings/workspaces with WorkspaceManagerPage
- Settings nav item for Workspaces
Tests:
- 25 backend tests (CRUD, modules, widgets, users, manager, seeding, context, isolation)
- 12 frontend tests (workspaceStore state, visibility, persistence, reset)
- 48/48 backend tests passing
- 12/12 frontend tests passing
2026-08-03 03:39:27 +02:00
236f0d2a5d
deploy.py: create_api_application ueber /applications/private-deploy-key (Git-basiert)
2026-08-03 02:18:23 +02:00
95972d2cdd
deploy.py: --initial mit API-UUID fuer Worker-Image und Deploy
2026-08-03 02:16:31 +02:00
0c789f7660
deploy.py: create_api_application ueber /applications/dockerfile (base64)
2026-08-03 02:10:38 +02:00
cd48d99c65
deploy.py: --initial Modus fuer vollautomatische Erstinstallation ueber Coolify API
2026-08-03 02:05:19 +02:00
f775405a01
deploy.py: 409 Conflict Handling (POST -> PATCH bei existierenden ENVs)
2026-08-03 01:32:41 +02:00
7e5e0dd8bd
deploy.py: ENV-Variablen ueber Coolify API setzen, keine manuelle .env-Datei mehr
2026-08-03 01:30:03 +02:00
8ac90e4dd6
deploy.py: .env nach update_service schreiben + _wait_service_healthy Bug fix
2026-08-03 01:22:44 +02:00
5eec2fdde8
deploy.py: ENV-Variablen statt hardcoded Passwoerter + .env auf Server schreiben
2026-08-03 01:17:25 +02:00
c63ab9b45a
Fix deploy.py: Use /deploy endpoint for Worker Service + connect_to_docker_network
2026-08-03 01:00:02 +02:00
2b50f528f3
Fix deploy.py: head -1 statt tail -1 fuer Image-Tag (neuestes Image zuerst)
2026-08-03 00:47:30 +02:00
bb6ea4001a
Update SANIERUNGS_FORTSCHRITT.md: Phase 5 produktionsverifiziert
2026-08-03 00:13:49 +02:00
ceb06600c5
Fix deploy.py: Worker-Deploy repariert
...
- Tag :latest auf neuestes Commit-Image (Coolify taggt mit Hash, nicht latest)
- Verbinde Worker mit coolify Netzwerk nach Restart (für Redis/Postgres DNS)
- Kein update_service mehr (überschreibt Coolify-Konfiguration)
- Worker-Compose auf Server korrigiert (coolify Netzwerk in Service-Definition)
2026-08-03 00:12:00 +02:00
e2b3cf081b
Fix deploy.py: Worker-Deploy war kaputt
...
- Bug 1: WORKER_COMPOSE_YAML hatte PW Platzhalter statt echter Passwörter
- Bug 2: deploy_worker rief deploy_application auf Service-UUID auf (falsche API)
- Bug 3: verify_worker_service akzeptierte nicht running:healthy Status
- Fix: Echte Passwörter, update_service+restart statt deploy_application, Status-Check korrigiert
2026-08-02 23:57:16 +02:00
74936b3972
Phase 5 (v2): Processing-Recovery, Retention-Cleanup, Replay-Delivery-Reset
...
- recover_stuck_events: Reset processing events stuck >120s back to pending
- cleanup_published_events: Delete published events older than 30 days
- Replay now resets outbox_deliveries for clean retry
- Worker: hourly retention cleanup cron job
- API: /recover-stuck and /cleanup-published endpoints
- process_outbox_batch: auto-recovery at start of each tenant iteration
- 23/23 tests passing (5 new tests)
2026-08-02 23:47:29 +02:00
4b0d32f8f0
Update SANIERUNGS_FORTSCHRITT.md: Phase 5 abgeschlossen
2026-08-02 23:29:21 +02:00
07a99975ec
Phase 5: Outbox DLQ, Monitoring, Consumer-Registry
...
- Migration 0092: DLQ columns (error_message, failed_at) + consumer_inbox RLS fix
- outbox.py: DLQ logic, replay functions, stats, consumer registry
- app/routes/outbox.py: 5 API endpoints (stats, failed, replay, replay-all, consumer-registry)
- outbox_deliveries tracking per consumer handler
- 18/18 tests passing
2026-08-02 23:25:54 +02:00
24cb10a7a2
docs: SANIERUNGS_FORTSCHRITT.md — kompakter Fortschritts-Tracker
...
- Phasen-Status: Phase 0-3 abgeschlossen, 4-10 offen
- Gates: Alle 5 bestanden
- Produktions-Setup: Coolify Ressourcen, DB-Rollen, Volumes
- Deployment: deploy.py Befehle dokumentiert
- Wichtige Dateien und Regeln für nächsten Agenten
- Was erledigt ist und was als nächstes zu tun ist
2026-08-02 23:05:33 +02:00
dfd9e778c5
test: Phase 3 — Plugin lifecycle tests (14/14 passed)
...
Tests:
- Registry initialization and engine requirement
- Plugin registration and discovery
- Load order with and without dependencies
- Core plugin deactivation blocked
- Deactivation blocked by active dependents
- Event handler registration on activate
- Event handler unregistration on deactivate
- Activate → deactivate → reactivate cycle
- Idempotent activate when already active
- Idempotent deactivate when already inactive
Phase 3 (Plugin-Lifecycle) verified:
- install: idempotent, dependency checks, migrations via crm_migration
- activate: idempotent, per-tenant with RLS context, event handlers
- deactivate: idempotent, core protection, dependency check, handler cleanup
- uninstall: deactivate first, then optional drop tables
- main.py: per-tenant activation with set_tenant_context
- Worker: event handlers only for active plugins (Gate 5)
- Router: only in API, not in worker
2026-08-01 23:29:20 +02:00
745bc4f2d8
feat: Phase 2 — Migration 0091: FK-Constraints für 74 Tenant-Tabellen
...
- 74 Tabellen erhalten FOREIGN KEY (tenant_id) REFERENCES tenants(id) ON DELETE CASCADE
- 10 globale Tabellen ausgeschlossen (sequences, system_settings, currencies, etc.)
- Orphan-Cleanup: SET tenant_id = NULL für verwaiste Einträge
- Idempotent: IF NOT EXISTS für alle Constraints
- Downgrade: Drop aller FK-Constraints
2026-08-01 23:04:15 +02:00
a922408e49
fix: add Origin header to login test in deploy.py
2026-08-01 21:23:52 +02:00
b3f40bacd2
fix: deploy.py rewrite — everything via Coolify API, no manual docker
2026-08-01 21:22:55 +02:00
a7b3424eee
docs: Vollständige Installationsanleitung (INSTALL.md)
...
- Komplette Schritt-für-Schritt-Installation von Grund auf
- Alle DB-Rollen und Bootstrap-Reihenfolge dokumentiert
- Alle ENV-Variablen für API, Worker, DB dokumentiert
- Vollständige docker-compose.yml Referenz
- Coolify-Setup-Anleitung
- prestart.sh Startup-Ablauf
- seed_admin.py Admin-Erstellung
- Verifikationsschritte (Health, Login, Alembic, RLS, DDL)
- SMTP-Konfiguration
- Backup und Restore mit Grant-Hinweis
- Häufige Probleme und Lösungen
- Architektur-Übersicht und Datei-Struktur
2026-08-01 20:37:53 +02:00
be20a8545e
docs: Abschlussbericht Phase 0+1 und vollständiger Sanierungsplan
...
- Kompletter Statusbericht mit allen 5 Gates
- Datenbankrollen-Architektur dokumentiert
- RLS-Architektur dokumentiert
- Verifizierte Sicherheitsnachweise
- Durchgeführte Code-Änderungen und Migrationen
- Offene Risiken
- Vollständiger Sanierungsplan Phase 2-10
- Gesamtschätzung: 120-210h verbleibend
- Empfohlene Reihenfolge
2026-08-01 07:28:11 +02:00
733fa1c807
docs: Gate 3 acceptance — restore test verified
...
Gate 3 (Restore-Test) bestanden:
- Backup aus Forgejo-Release heruntergeladen, MD5 verifiziert
- pg_restore in separate Test-DB (crm_restore_test)
- alembic upgrade head: 0086 → 0090
- Datenintegrität: 9 Contacts, 2 Tenants, 1 User, 479 Sessions
- RLS: 0 rows ohne Kontext, 8 rows Tenant B, 2 rows Tenant A
- Cross-Tenant INSERT blockiert, DDL blockiert
- 108 RLS-Tabellen, 112 Policies, 0 Legacy Policies
2026-08-01 00:27:12 +02:00
9b4ee3b8ca
docs: Gate 5 acceptance — worker event handlers verified
...
Gate 5 (Worker und Eventhandler) bestanden:
- Worker healthy, verarbeitet Outbox-Jobs und enqueued Jobs
- 18 Worker-Funktionen registriert
- Plugin-Eventhandler nur für aktive Plugins
- Per-Tenant Outbox-Processing mit RLS-Kontext
- Worker verwendet crm_worker (get_worker_session_factory)
- Keine Plugin-Router im Worker
2026-07-31 23:15:32 +02:00
94847ea515
fix: PluginModel.is_active → PluginModel.active (worker crash fix)
2026-07-31 23:12:05 +02:00
cea21ff576
fix: Gate 5 — worker event handlers and per-tenant outbox processing
...
Worker fixes:
- registry.initialize uses get_migration_engine() for DDL (not worker_engine)
- Worker session uses get_worker_session_factory() (crm_worker, not crm_api)
- Event handlers only registered for active plugins (is_active check)
- Outbox processing per-tenant with set_config(app.current_tenant_id)
- process_outbox_job uses get_worker_session_factory() and loads tenant_ids
- Removed unused get_engine import
Outbox fixes:
- process_outbox_batch iterates over tenants, sets RLS context per tenant
- _process_single_outbox_event extracted for clarity
- Events claimed per-tenant (RLS-compatible, no BYPASSRLS needed)
- Commit after each tenant to release locks
Gate 5 requirements met:
- Plugin event handlers registered for active plugins only
- No plugin routers registered in worker
- Outbox events without handlers marked as no_handlers
- Failed consumers trigger retry with exponential backoff
- Processing is idempotent (consumer_inbox check)
- Every worker DB access sets app.current_tenant_id
- Worker cannot read/write other tenant data (RLS enforced)
2026-07-31 23:09:25 +02:00
89fe7a4750
docs: Gate 2 acceptance — fresh DB install verified
...
Gate 2 (Neuinstallation auf leerer Datenbank) bestanden:
- Alembic-Head 0090, 124 Tabellen, 47 RLS-Tabellen
- 0 legacy app.tenant_id policies
- Alle 4 DB-Rollen korrekt (NOSUPERUSER, crm_migration BYPASSRLS)
- RLS fail-closed: 0 rows ohne Kontext
- Cross-Tenant INSERT blockiert
- crm_api DDL blockiert
- seed_admin.py funktioniert
- Login erfolgreich (200 OK)
- Keine manuellen Schemaänderungen
2026-07-31 22:33:07 +02:00
89b775b9ef
fix: legacy app.tenant_id policies on _old tables + seed_admin.py rewrite
...
- Migration 0090: Drop legacy tenant_isolation policies on companies_old,
company_contacts_old, contacts_old that used app.tenant_id variable.
Create new policies using app.current_tenant_id for crm_api/crm_worker.
- seed_admin.py: Rewrite to use migration engine (crm_migration) for
bootstrap, set tenant context, create Tenant + Role + User + UserTenant.
No longer passes tenant_id as User parameter.
Fixes: 3 legacy app.tenant_id policies found in Gate 2 verification.
Fixes: seed_admin.py incompatible with current User model.
2026-07-31 22:23:38 +02:00
b5191f0d11
gate2: migration 0089 — add updated_at to sessions table (model uses TimestampMixin but table was missing column)
2026-07-31 22:15:29 +02:00
569476b993
gate2: fix prestart.sh shell quote conflict — use temp Python file instead of python3 -c
2026-07-31 21:57:04 +02:00
68db50544c
gate2: fix shell quote conflict in prestart.sh — use string concat instead of f-string for ALTER ROLE
2026-07-31 21:49:45 +02:00
2a7412e49f
gate2: fix prestart.sh — inline password for ALTER ROLE (prepared statements dont work with ALTER ROLE)
2026-07-31 21:42:55 +02:00
9124b17a8e
gate2: prestart.sh sets passwords for all DB roles (crm_api, crm_auth, crm_worker, crm_migration) after migration
...
Migration 0070 creates roles without passwords. On fresh DB, API cannot authenticate.
prestart.sh now extracts password from MIGRATION_DATABASE_URL and sets it for all roles.
2026-07-31 21:31:18 +02:00
Agent Zero