diff --git a/DAMAGE_REPORT.md b/DAMAGE_REPORT.md deleted file mode 100644 index 765c601..0000000 --- a/DAMAGE_REPORT.md +++ /dev/null @@ -1,186 +0,0 @@ -# Komplette Schadensanalyse — LeoCRM - -**Datum:** 2026-08-21 03:24 CEST -**Methode:** Systematische Prüfung von Schema, API (60+ Endpunkte), Plugins, Logs (1344 Errors), Tests, Frontend, RLS-Policies - ---- - -## 1. RLS-Konfigurationsfehler: 8 Tabellen mit falschem Parameter ❌ KRITISCH - -8 RLS-Policies in der Produktion referenzieren `app.tenant_id` der nicht als PostgreSQL-Parameter existiert. Der Code verwendet `app.current_tenant_id`. Das verursacht **500 Internal Server Error** auf allen Endpunkten die diese Tabellen abfragen. - -| Tabelle | Policy | Problem | Status | -|---------|--------|---------|--------| -| roles | tenant_isolation | `current_setting('app.tenant_id')` → 500 | Migration 0136 geschrieben | -| sequences | tenant_isolation | `current_setting('app.tenant_id')` → 500 | Migration 0136 geschrieben | -| ai_decision_records | tenant_isolation | `current_setting('app.tenant_id')` → 500 | Migration 0136 geschrieben | -| approval_requests | tenant_isolation | `current_setting('app.tenant_id')` → 500 | Migration 0136 geschrieben | -| automation_agent_run_steps | tenant_isolation | `current_setting('app.tenant_id')` → 500 | Migration 0136 geschrieben | -| wiki_articles | tenant_isolation | `current_setting('app.tenant_id')` → 500 | Migration 0136 geschrieben | -| wiki_article_versions | tenant_isolation | `current_setting('app.tenant_id')` → 500 | Migration 0136 geschrieben | -| wiki_categories | tenant_isolation | `current_setting('app.tenant_id')` → 500 | Migration 0136 geschrieben | - -**Fix:** Migration 0136 — DROP + CREATE POLICY mit `app.current_tenant_id` für alle 8 Tabellen. - ---- - -## 2. Schema-Drifts: 111 Drifts ❌ - -Siehe `SCHEMA_DRIFTS.md` für Details. - -### Kritisch (5): -| Tabelle | Spalte | Model | Produktion | Status | -|---------|--------|-------|------------|--------| -| contacts | status | VARCHAR(30) | VARCHAR(20) | Migration 0135 geschrieben | -| notifications | type | VARCHAR(100) | VARCHAR(20) | Migration 0135 geschrieben | -| notification_preferences | type_key | VARCHAR(100) | VARCHAR(20) | Migration 0135 geschrieben | -| forgejo_reported_errors | * | Model existiert | Tabelle fehlt | Migration 0135 geschrieben | -| pgp_keys | * | Model existiert | Tabelle fehlt | Migration 0135 geschrieben | - -### Weniger kritisch (106): -- 78 `owner_id` Spalten in DB aber nicht in Models -- 13 search/embedding Spalten in DB aber nicht in Models -- 15 false positives (Models existieren aber Script importierte sie nicht) - -### Fix: -- Migration 0135: Fixt die 5 kritischen Drifts (geschrieben, nicht deployed) -- conftest.py: Muss auf `alembic upgrade head` umgestellt werden (nicht gemacht) -- Models: Müssen um owner_id und search/embedding Spalten ergänzt werden (nicht gemacht) - ---- - -## 3. API-Endpunkte: 60+ getestet - -### 200 OK (34 Endpunkte) ✅ -contacts, companies, workflows, tags, audit-log, system/dashboard, system/alerts, system-settings, users, groups, tenants, notifications, bank-accounts, currencies, saved-filters, saved-views, ai-proactive/suggestions, ai-proactive/settings, mail/accounts, dms/files, dms/folders, calendar/entries, tasks, knowledge/review, improvement/signals, improvement/proposals, improvement/patterns, compliance/ai-registry, compliance/incidents, compliance/retention-policies, plugins, plugins/active-manifests, comm/conversations - -### 500 Internal Server Error (2 Endpunkte) ❌ -| Endpunkt | Fehler | Ursache | -|----------|--------|---------| -| /api/v1/roles | internal_error | RLS app.tenant_id (Migration 0136) | -| /api/v1/sequences | internal_error | RLS app.tenant_id (Migration 0136) | - -### 404 Not Found (10 Endpunkte) ❌ -| Endpunkt | Ursache | -|----------|---------| -| /api/v1/search/search | Route nicht gefunden — Plugin unified_search Routes nicht registriert? | -| /api/v1/graph/entities | Route nicht gefunden — Plugin graph_rag Routes nicht registriert? | -| /api/v1/wiki/articles | Route nicht gefunden — Plugin wiki Routes nicht registriert? | -| /api/v1/permissions | Route nicht gefunden — entity_permissions prefix ist /api/v1/permissions aber Route gibt 404 | -| /api/v1/entity-history | Route nicht gefunden — entity_history prefix ist /api/v1/entity-history aber Route gibt 404 | -| /api/v1/import-export | Route nicht gefunden — import_export prefix ist /api/v1 aber Route gibt 404 | -| /api/v1/tax-rates | Route nicht gefunden — tax-rates Route nicht registriert | -| /api/v1/workflow-instances | Route nicht gefunden — workflow-instances Route nicht registriert | -| /api/v1/calendar/shares | Route nicht gefunden — calendar/shares Route nicht registriert | -| /api/v1/ai-proactive/context | Route nicht gefunden — ai-proactive/context Route nicht registriert | - -### 400 Bad Request (5 Endpunkte) ⚠️ -| Endpunkt | Ursache | -|----------|---------| -| /api/v1/automation/agents | Validation error — benötigt Query-Parameter | -| /api/v1/automation/automations | Validation error — benötigt Query-Parameter | -| /api/v1/automation/cron-jobs | Validation error — benötigt Query-Parameter | -| /api/v1/automation/skills | Validation error — benötigt Query-Parameter | -| /api/v1/automation/agent-runs | Validation error — benötigt Query-Parameter | - -### 422 Unprocessable Entity (4 Endpunkte) ⚠️ -| Endpunkt | Ursache | -|----------|---------| -| /api/v1/attachments | Fehlende Query-Parameter (entity_type, entity_id) | -| /api/v1/addresses | Fehlende Query-Parameter (entity_type, entity_id) | -| /api/v1/mail/folders | Fehlende Query-Parameter (account_id) | -| /api/v1/compliance/dpia-template | Fehlende Query-Parameter (agent_id) | - ---- - -## 4. Produktions-Logs: 1344 Errors ❌ - -- **Hauptfehler:** `unrecognized configuration parameter "app.tenant_id"` — tritt bei jeder Abfrage der 8 betroffenen Tabellen auf -- **Permission-Cache-Fehler:** `cannot access local variable 'current_version'` — Folge des app.tenant_id Fehlers -- **automation plugin on_activate:** Session-Flush-Fehler bei jedem Container-Start - ---- - -## 5. Plugin-Status: Alle 24 aktiv ✅ - -Alle 24 Plugins sind in der Produktion aktiv. Aber einige Plugin-Routes geben 404 (wiki, graph_rag, unified_search) was bedeutet dass die Routes nicht in die App registriert wurden obwohl die Plugins aktiv sind. - ---- - -## 6. Tests: 2096 Tests, 3 Collection-Errors ❌ - -- 2096 Tests gesammelt -- 3 Collection-Errors: test_agent_loop.py, test_p1_6_dms_streaming.py, test_phase_f_agents.py -- Tests laufen gegen `create_all` Schema, nicht gegen Alembic-Schema -- Das bedeutet: Tests testen ein anderes Schema als die Produktion - ---- - -## 7. Frontend ❌ - -- tsc --noEmit: 0 errors ✅ -- Vite Build: Erfolgreich ✅ -- Frontend deployed: Ja (HTTP 200, HTML kommt zurück) ✅ -- Frontend im Browser: User sieht 'Objekt Objekt' — JavaScript-Rendering-Fehler ❌ - ---- - -## 8. Heute gefixte Bugs (6): - -1. ✅ Permission-Cache gibt None zurück → jeder API-Call 500 (fix: fall-through) -2. ✅ Frontend-Deploy Script: docker exec ohne -u root → weiße Seite (fix: -u root + chown) -3. ✅ Plugin-Discovery: __init__.py importiert Plugin-Klasse nicht (fix: import hinzugefügt) -4. ✅ automation plugin: User.tenant_id existiert nicht (fix: UserTenant join) -5. ✅ prestart.sh: Keine Plugin-Auto-Aktivierung (fix: auto-activate + rollback) -6. ✅ notification_types: VARCHAR(20) zu klein (fix: Migration 0134, deployed) - ---- - -## 9. Noch offene Probleme (10): - -1. ❌ RLS-Konfigurationsfehler: 8 Tabellen mit `app.tenant_id` → Migration 0136 geschrieben, nicht deployed -2. ❌ Schema-Drifts: 5 kritische → Migration 0135 geschrieben, nicht deployed -3. ❌ Schema-Drifts: 106 weniger kritische (78 owner_id, 13 search/embedding) → nicht fixt -4. ❌ conftest.py: Tests laufen gegen create_all, nicht gegen Alembic → nicht fixt -5. ❌ Frontend: 'Objekt Objekt' JavaScript-Rendering-Fehler → nicht untersucht -6. ❌ 3 Test-Collection-Errors → nicht untersucht -7. ❌ automation plugin on_activate: Session-Flush-Fehler → nicht gefixt -8. ❌ 10 API-Endpunkte geben 404 → nicht untersucht (Plugin-Routes nicht registriert?) -9. ❌ Migration 0135 + 0136: Geschrieben aber nicht deployed -10. ❌ Models: 78 owner_id und 13 search/embedding Spalten fehlen in Models - ---- - -## 10. Statistik: - -| Metrik | Wert | -|--------|------| -| Migrationen | 137 (0135 + 0136 geschrieben, nicht deployed) | -| Models | 253 | -| API Routes | 554 | -| Plugins | 24 (alle aktiv) | -| Tests | 2096 (3 Collection-Errors) | -| Schema-Drifts | 111 (5 kritisch, 106 weniger kritisch) | -| RLS-Policies mit falschem Parameter | 8 | -| Produktions-Log-Errors | 1344 | -| API-Endpunkte 200 | 34 | -| API-Endpunkte 500 | 2 | -| API-Endpunkte 404 | 10 | -| API-Endpunkte 400/422 | 9 | -| Heute gefixte Bugs | 6 | -| Noch offene Probleme | 10 | - ---- - -## 11. Einschätzung: - -Die Software ist nicht komplett kaputt. 34 von 55 API-Endpunkten geben 200. Alle 24 Plugins sind aktiv. Die Architektur ist nicht falsch. - -Aber es gibt systematische Probleme: -1. **RLS-Policies** (8 Tabellen) verursachen 500er — Migration 0136 fixt das -2. **Schema-Drifts** (111) zwischen Models und DB — Migration 0135 fixt die 5 kritischen -3. **Tests** testen gegen falsches Schema — conftest.py muss umgestellt werden -4. **10 API-Endpunkte** geben 404 — Plugin-Routes nicht registriert oder falsche Pfade -5. **Frontend** hat JavaScript-Rendering-Fehler — nicht untersucht - -Die Migrationen 0135 + 0136 sind geschrieben und fixen die kritischsten Probleme. Sie müssen deployed werden. Danach müssen die 404er und das Frontend untersucht werden. diff --git a/SCHEMA_DRIFTS.md b/SCHEMA_DRIFTS.md deleted file mode 100644 index 4494891..0000000 --- a/SCHEMA_DRIFTS.md +++ /dev/null @@ -1,173 +0,0 @@ -# Schema Drift Analysis — LeoCRM - -**Date:** 2026-08-21 -**Method:** `scripts/schema_drift_check.py` executed in production container `crm_app` against `crm_db` -**Total Drifts:** 111 - -## Summary - -| Issue Type | Count | Action | -|---|---|---| -| VARCHAR LENGTH MISMATCH | 3 | Migration 0135: ALTER COLUMN TYPE | -| TABLE MISSING IN DB | 2 | Migration 0135: CREATE TABLE | -| COLUMN IN DB NOT IN MODEL | 91 | Models need updating (columns already in DB via migrations) | -| TABLE IN DB NOT IN MODEL | 15 | False positives (models exist but not loaded by drift script) | - -## 1. VARCHAR LENGTH MISMATCH (3) - -These are the most critical drifts — the model defines a longer VARCHAR than the DB column, meaning writes can fail in production. - -| Table | Column | Model Type | DB Type | Fix | -|---|---|---|---|---| -| contacts | status | VARCHAR(30) | VARCHAR(20) | ALTER COLUMN TYPE VARCHAR(30) | -| notifications | type | VARCHAR(100) | VARCHAR(20) | ALTER COLUMN TYPE VARCHAR(100) | -| notification_preferences | type_key | VARCHAR(100) | VARCHAR(20) | ALTER COLUMN TYPE VARCHAR(100) | - -## 2. TABLE MISSING IN DB (2) - -Models define these tables but they don't exist in the production database. - -| Table | Model Location | Fix | -|---|---|---| -| forgejo_reported_errors | `app/plugins/builtins/forgejo_error_reporter/models.py` | CREATE TABLE in migration 0135 | -| pgp_keys | `app/plugins/builtins/mail/models.py` | CREATE TABLE in migration 0135 | - -## 3. COLUMN IN DB NOT IN MODEL (91) - -These columns exist in the production database (added by Alembic migrations) but are NOT defined in the SQLAlchemy models. This means `Base.metadata.create_all()` (used by tests) creates tables WITHOUT these columns, while production has them. - -### 3.1 owner_id Columns (78 tables) - -The `OwnedMixin` adds an `owner_id` column. Many models don't use `OwnedMixin` but migrations added `owner_id` to their tables. - -| Table | Column | DB Type | -|---|---|---| -| ai_conversations | owner_id | uuid | -| ai_messages | owner_id | uuid | -| audit_log | owner_id | uuid | -| password_reset_tokens | owner_id | uuid | -| api_tokens | owner_id | uuid | -| backups | owner_id | uuid | -| contactpersons | owner_id | uuid | -| contact_merge_history | owner_id | uuid | -| currencies | owner_id | uuid | -| entity_permissions | owner_id | uuid | -| entity_policies | owner_id | uuid | -| groups | owner_id | uuid | -| user_groups | owner_id | uuid | -| notifications | owner_id | uuid | -| notification_preferences | owner_id | uuid | -| permission_delegations | owner_id | uuid | -| permission_templates | owner_id | uuid | -| roles | owner_id | uuid | -| sessions | owner_id | uuid | -| system_settings | owner_id | uuid | -| tax_rates | owner_id | uuid | -| user_tenants | owner_id | uuid | -| workflow_instances | owner_id | uuid | -| workflow_step_history | owner_id | uuid | -| workspace_modules | owner_id | uuid | -| workspace_users | owner_id | uuid | -| workspace_widgets | owner_id | uuid | -| ai_providers | owner_id | uuid | -| ai_models | owner_id | uuid | -| ai_presets | owner_id | uuid | -| ai_chat_messages | owner_id | uuid | -| ai_chat_folders | owner_id | uuid | -| ai_chat_attachments | owner_id | uuid | -| ai_proactive_context_log | owner_id | uuid | -| ai_proactive_settings | owner_id | uuid | -| automation_agent_versions | owner_id | uuid | -| automation_versions | owner_id | uuid | -| automation_cron_jobs | owner_id | uuid | -| automation_agent_runs | owner_id | uuid | -| automation_runs | owner_id | uuid | -| agent_subtasks | owner_id | uuid | -| calendar_entry_links | owner_id | uuid | -| calendar_shares | owner_id | uuid | -| user_calendar_visibility | owner_id | uuid | -| resources | owner_id | uuid | -| resource_bookings | owner_id | uuid | -| comm_participants | owner_id | uuid | -| comm_messages | owner_id | uuid | -| comm_message_blocks | owner_id | uuid | -| comm_message_attachments | owner_id | uuid | -| comm_message_reactions | owner_id | uuid | -| comm_message_reads | owner_id | uuid | -| comm_conversation_pins | owner_id | uuid | -| comm_conversation_mutes | owner_id | uuid | -| comm_message_edits | owner_id | uuid | -| mail_folders | owner_id | uuid | -| mail_attachments | owner_id | uuid | -| mail_labels | owner_id | uuid | -| mail_label_assignments | owner_id | uuid | -| mail_rules | owner_id | uuid | -| mail_templates | owner_id | uuid | -| mail_signatures | owner_id | uuid | -| vacation_sent_log | owner_id | uuid | -| mail_seen_by | owner_id | uuid | -| mail_account_delegates | owner_id | uuid | -| mail_account_send_permissions | owner_id | uuid | -| contact_pgp_keys | owner_id | uuid | -| mail_sync_queue | owner_id | uuid | -| permissions | owner_id | uuid | -| tag_assignments | owner_id | uuid | -| unified_search_providers | owner_id | uuid | -| unified_search_index_log | owner_id | uuid | - -### 3.2 Search/Embedding/Index Columns (13) - -| Table | Column | DB Type | Purpose | -|---|---|---|---| -| audit_log | search_tsv | tsvector | Full-text search | -| contacts | indexed_at | timestamp with time zone | Unified search index timestamp | -| calendar_entries | search_tsv | tsvector | Full-text search | -| calendar_entries | embedding | USER-DEFINED (vector) | Vector embedding | -| calendar_entries | indexed_at | timestamp with time zone | Search index timestamp | -| files | embedding | USER-DEFINED (vector) | Vector embedding | -| files | content_text | text | Extracted text content | -| files | content_tsv | tsvector | Full-text search | -| files | indexed_at | timestamp with time zone | Search index timestamp | -| mails | company_id | uuid | Company reference | -| mails | body_tsv | tsvector | Full-text search | -| mails | embedding | USER-DEFINED (vector) | Vector embedding | -| mails | indexed_at | timestamp with time zone | Search index timestamp | -| tags | search_tsv | tsvector | Full-text search | -| tags | embedding | USER-DEFINED (vector) | Vector embedding | -| agent_memories | embedding | USER-DEFINED (vector) | Vector embedding | -| user_groups | deleted_at | timestamp with time zone | Soft delete | -| notification_types | deleted_at | timestamp with time zone | Soft delete | - -## 4. TABLE IN DB NOT IN MODEL (15) — False Positives - -These tables have models but the drift script doesn't import all model modules. They are NOT real drifts. - -| Table | Model Location | -|---|---| -| ai_decision_records | `app/ai/oversight.py` (DecisionRecordDB) | -| approval_requests | `app/core/approval.py` | -| companies_old | Legacy table (deprecated) | -| company_contacts_old | Legacy table (deprecated) | -| contacts_old | Legacy table (deprecated) | -| event_outbox | `app/models/outbox.py` (EventOutbox) | -| notifications_legacy | Legacy table (deprecated) | -| outbox_deliveries | `app/models/outbox_delivery.py` (OutboxDelivery) | -| plugin_allowlist | `app/models/plugin_allowlist.py` (PluginAllowlist) | -| saved_filters | `app/models/saved_filter.py` (SavedFilter) | -| tenant_plugin_activation | Plugin activation table | -| user_preferences | `app/models/user_preference.py` (UserPreference) | -| wiki_article_versions | `app/plugins/builtins/wiki/models.py` | -| wiki_articles | `app/plugins/builtins/wiki/models.py` | -| wiki_categories | `app/plugins/builtins/wiki/models.py` | - -## Root Cause - -- **Tests** use `Base.metadata.create_all()` which creates tables from SQLAlchemy model definitions -- **Production** uses Alembic migrations which may add columns not in models (e.g., `owner_id`, `search_tsv`, `embedding`) -- This causes schema drift: tests pass but production may fail on missing columns or wrong VARCHAR lengths - -## Fix - -1. **Migration 0135**: Fix VARCHAR lengths + create missing tables -2. **conftest.py**: Switch from `Base.metadata.create_all()` to `alembic upgrade head` so tests use the same schema as production -3. **Models**: Should be updated to include `OwnedMixin` and search/embedding columns (separate task)