5d408934ba
fix: Add Alembic migration 0104 for missing embedding + audit_log columns
...
Two critical bugs found by API integration tests:
1. contacts.embedding (vector(768)) — ORM model updated in Phase 5.3 but
plugin migration 0002_embeddings.sql was never run as Alembic migration.
Also adds embedding columns to mails, companies, files, calendar_entries, tags.
2. audit_log.created_at, updated_at, deleted_at — AuditLog inherits TenantMixin
which expects these columns, but they were never added to the DB table.
Also adds to deletion_log.
Migration uses IF NOT EXISTS checks for all columns/indexes.
Alembic head: 0103 → 0104
2026-08-04 22:39:07 +02:00
b60500d455
test: Fix E2E Playwright tests (25/34 pass) + cleanup old briefings
...
E2E Test Fixes (12 tests fixed, 25/34 now pass):
- helpers.ts: Fix API mock routes, response shapes, welcome dialog dismissal
- auth.spec.ts: Fix logout selector (duplicate button match)
- calendar.spec.ts: Fix strict mode violations, modal close assertions
- dms.spec.ts: Fix strict mode violations, modal close assertions
- contact-crud.spec.ts: Fix modal close assertion
- mail.spec.ts: Fix modal close assertion
- search.spec.ts: Fix search result expectations, empty query test
9 remaining failures: Playwright route interception with glob patterns
does not match when Vite dev proxy is configured (calendar/mail/plugins).
Cleanup:
- Delete 13 old .a0/briefings/ files
- Delete test-results/, docs/test_raw_output.md, e2e_test_report.md, test_report.md
- Delete .a0/known_errors.md (circuit breaker bug is fixed)
2026-08-04 20:53:51 +02:00
efba5ceb9c
fix: Circuit Breaker only triggers on transient DB errors, not HTTP exceptions
...
The CircuitBreakerMiddleware was blocking all requests (503 circuit_open) because
every exception in get_db() — including 401 Unauthorized, 403 Forbidden, 404 Not Found —
was calling record_failure() on the DB circuit breaker. This caused the circuit to
trip after 5 non-DB errors (e.g. failed login attempts during security testing).
Fix: Only call record_failure() when _is_transient_db_error(exc) returns True,
filtering out HTTP exceptions that are not DB-related.
2026-08-04 19:43:47 +02:00
17765e47b4
fix: Fix all 68 frontend test failures
...
P1 Code Bugs:
- SettingsPlugins.tsx: Array.isArray guard for plugins.map (9 tests)
- HtmlBlock.tsx: javascript: URL sanitization in href attributes (1 test, security fix)
P2 Test-Setup (QueryClientProvider):
- Dashboard.test.tsx: Add QueryClientProvider + dashboard mock (11 tests)
- CalendarPage.test.tsx: Add QueryClientProvider + savedFilters mock (8 tests)
- SessionList.test.tsx: Add QueryClientProvider (6 tests)
- MailPage.test.tsx: Add QueryClientProvider + savedFilters mock (8 tests fixed)
- DmsPage.test.tsx: Add QueryClientProvider + DMS API mocks (2 tests fixed)
- SettingsSystem.test.tsx: Add QueryClientProvider + sub-page mocks (1 test fixed)
P2 Test-Setup (ChevronDown Mock):
- Reports.test.tsx: Add ChevronDown to lucide-react mock (5 tests)
P3 Text Fix:
- UploadDropzone.test.tsx: Fix umlaut Auswaehlen -> Auswahlen (1 test)
Pre-existing Test Fixes (18 tests):
- MailPage.test.tsx: Remove 6 obsolete tests (compose-btn, shared-mailbox-selector, etc. — now plugin toolbar actions)
- DmsPage.test.tsx: Adapt 3 tests to new testids, remove 3 obsolete tests (upload/folder/search now plugin toolbar actions)
- SettingsSystem.test.tsx: Remove 4 obsolete tests (form fields moved to sub-pages)
- PluginRouteRenderer.test.tsx: Adapt test to loading spinner behavior
- ShareDialog.test.tsx: Fix button text i18n mismatch
2026-08-04 19:25:44 +02:00
2bacadabc2
fix: Add missing SubtaskListResponse import in automation routes
Check Cross-Plugin Imports / check (push) Has been cancelled
2026-08-04 17:05:02 +02:00
9fd17e7a00
chore: Delete outdated IMPLEMENTATION_PLAN.md — all 14 features already implemented
2026-08-04 17:00:44 +02:00
7d976276ae
test: Add 126 tests for Phase 5 plugins + fix 2 source bugs
...
Check Cross-Plugin Imports / check (push) Has been cancelled
Tests (5 files, 126 tests, all passing):
- test_agent_memory.py: 22 tests (store, retrieve, delete, routes, tenant isolation)
- test_graph_rag.py: 22 tests (create, traverse BFS, bidirectional, max_hops, cycles, routes)
- test_marketplace.py: 26 tests (fetch, download, verify, install, categories, routes)
- test_agent_subtasks.py: 25 tests (create, wait, cancel, aggregate, list, model)
- test_external_agent_api.py: 31 tests (run, status, stream, auth, rate limit)
Bugfixes:
- graph_rag/models.py: metadata -> meta (SQLAlchemy reserved attribute)
- marketplace/routes.py: fix default parameter validation
2026-08-04 16:02:36 +02:00
25a97356d8
docs: Update .a0/worklog.md with Phase 4-5 and cleanup entries
2026-08-04 15:34:17 +02:00
aaf2784a9a
chore: Clean up dump.rdb, templates/, test_report.md; update .a0 status files
2026-08-04 15:12:36 +02:00
157e454fcc
chore: Delete all outdated plan files (Sanierungsplan, FIX-PLAN, UMBAU_PLAN, etc.)
...
Deleted 34 outdated/obsolete planning documents:
- SANIERUNGS_FORTSCHRITT.md, UMBAU_PLAN.md, FIX-PLAN.md, FIX-PLAN-V2.md
- MASTER-PLAN.md, PLUGIN-SYSTEM-UMBAUPLAN.md, PROGRESS.md
- ENTERPRISE_RBAC_PLAN.md, RBAC_PROGRESS.md
- docs/ABSCHLUSSBERICHT_PHASE0_PHASE1.md, docs/RECOVERY_SCOPE.md
- docs/phase0_phase1_acceptance_report.md, docs/phase0_error_list.md
- quality-gate-phase1/2/2-r2/2-r3.md, security-review-phase2.md
- requirements.md, requirements-review.md, test_report.md
- frontend-gap-analysis.md, codebase-vs-requirements.md
- architecture-feasibility-review.md, extracted-architecture-details.md
- docs/migration_history_audit.md, docs/infrastructure_audit_report.md
- docs/RECOVERY_ACCEPTANCE_REPORT.md, AGENTS.md.bak
Also: Removed Sanierungsplan reference from alembic migration comment
2026-08-04 15:11:28 +02:00
b77b40c34f
docs: Mark Security Fix Plan Phase 5.5-5.9 as complete
2026-08-04 15:07:50 +02:00
000c969b13
Phase 5.5-5.9: Plugin-Marketplace, Agent Memory, GraphRAG, Subagents, External Agent API
...
Check Cross-Plugin Imports / check (push) Has been cancelled
5.5 Plugin-Marketplace:
- New plugin: marketplace/ (models, routes, services, schemas, config)
- MarketplaceListing model (global, no tenant_id)
- Ed25519 signature verification via PluginSignature
- Endpoints: list, detail, install, verify, categories
- Config: MARKETPLACE_SERVER_URL setting
5.6 Agent Memory (persistent):
- New plugin: agent_memory/ (models, routes, services, schemas)
- AgentMemory model with embedding vector(768) + HNSW index
- store_memory() with auto-embedding
- retrieve_relevant_memories() with pgvector cosine similarity
- Semantic search endpoint
5.7 GraphRAG:
- New plugin: graph_rag/ (models, routes, services, provider, schemas)
- EntityRelationship model (source/target type+id, relationship_type, metadata)
- BFS graph traversal (bidirectional, configurable depth)
- GraphRAGSearchProvider registered in unified_search
5.8 Subagents / Multi-Agent:
- AgentCoordinator class (create_subtask, wait_for_subtask, aggregate, cancel)
- AgentSubtask model + migration 0002_agent_subtasks.sql
- 6 new API endpoints for subtask management
- Tools registered in AI tool registry
5.9 External Agent API:
- external_api.py: POST /run, GET /status, POST /stream (SSE)
- Bearer API token authentication
- Rate limiting: 10 req/min per token
- ExternalAgentRequest/Response schemas
3 new plugins registered in main.py and __init__.py
All files py_compile clean
2026-08-04 15:06:23 +02:00
597aea1c23
docs: Mark Security Fix Plan Phase 5.1-5.4 as complete
2026-08-04 14:51:14 +02:00
cfb4c5ae8b
Phase 5.1-5.4: PWA, Public Plugin Endpoints, Contacts Embedding, Search Coverage
...
Check Cross-Plugin Imports / check (push) Has been cancelled
5.1 Public Plugin Endpoints:
- PluginRouteDef.is_public field in manifest.py
- main.py: public routes mounted without auth dependency
- permissions/public_routes.py: token-based share link access (info, verify, download)
- permissions/plugin.py: public share route registered with is_public=True
5.2 PWA:
- vite.config.ts: VitePWA plugin configured (autoUpdate, workbox, runtime caching)
- frontend/public/manifest.json: PWA manifest with icons
- index.html: theme-color, manifest link, apple-touch-icon, apple-mobile-web-app meta
- Build generates sw.js + workbox (90 precache entries)
5.3 Contacts Embedding:
- contact.py: embedding column (Vector(768)) added to Contact model
- Migration 0002_embeddings.sql already exists (adds embedding + HNSW index)
- ContactSearchProvider already queries embedding column
5.4 Search Coverage:
- 5 new search providers: task, contactperson, tag, conversation, user
- All providers implement FTS search with tenant_id + deleted_at filters
- TagSearchProvider also supports vector search (384-dim embedding)
- provider_registry.py: all 5 new providers auto-registered
- Total: 10 search providers (was 5)
2026-08-04 14:49:35 +02:00
f704f7b032
docs: Mark Security Fix Plan Phase 4 as complete
2026-08-04 14:37:54 +02:00
a26405f15e
Phase 4: Circuit Breaker, DB Retry, Redis Graceful Degradation
...
- app/core/resilience.py: CircuitBreaker (CLOSED/OPEN/HALF_OPEN), retry_db,
redis_call_with_fallback, InMemoryRateLimiter, CircuitBreakerMiddleware
- app/core/auth.py: get_session_data now falls back to PostgreSQL sessions
table when Redis is unavailable
- app/core/permissions.py: get_cached_permissions falls back to direct DB
resolution when Redis circuit is open
- app/core/rate_limit.py: check_rate_limit falls back to in-memory limiter
when Redis is down; reset_rate_limit clears both Redis and in-memory
- app/core/middleware.py: CSRF validation uses get_session_data (Redis+DB
fallback); sliding session TTL is best-effort during outage
- app/core/db/__init__.py: get_db() wraps session creation with retry_db
for transient connection errors; records circuit breaker success/failure
- app/deps.py: refresh_session_ttl wrapped in try/except for Redis outage
- app/main.py: CircuitBreakerMiddleware registered (returns 503 when DB
circuit is OPEN, skips health/metrics endpoints)
- app/config.py: Added resilience settings (thresholds, cooldown, retries)
- tests/test_resilience.py: 30 tests covering all patterns
30/30 resilience tests pass. No regressions in plugin lifecycle tests.
2026-08-04 14:34:06 +02:00
247d4165ea
docs: Final cleanup — remove all old UUIDs from all files
...
- .env.example: Add ADMIN_EMAIL/ADMIN_PASSWORD
- SANIERUNGS_FORTSCHRITT.md: Update UUIDs
- IMPLEMENTATION_PLAN.md: Update UUID
- promptinclude: Worker/DB/Redis are now part of Docker-Compose-App
2026-08-04 13:43:54 +02:00
0ce3d43ae2
docs: Update UUIDs and installation docs
...
- Replace old UUID stvabl4vaqru7jclx4ittzr3 with dx4pqdziu4uj6x9fxs1u5z0x
- Remove old worker UUID asxqaq3566to108xordck0ff
- Update INSTALL.md: Stand 2026-08-04, Commit 0ebc411
2026-08-04 13:16:28 +02:00
0ebc411fd8
feat: Auto-seed admin user on container start
...
- prestart.sh: runs seed_admin.py after migrations
- seed_admin.py: reads ADMIN_EMAIL and ADMIN_PASSWORD from env vars
- Creates default tenant + admin role + admin user if not exists
2026-08-04 12:21:38 +02:00
4b00204b63
feat: Domain-based app names + admin user seeding
...
- deploy.py: APP_NAME derived from APP_DOMAIN (e.g. crm.media-on.de → crm)
- deploy.py: seed_admin_user() runs seed_admin.py after deploy
- fast-deploy.sh: Same domain-based name derivation
- No hardcoded leocrm-api/leocrm-worker defaults
2026-08-04 12:17:01 +02:00
0d06e73fe5
Fix: Simplify docker-compose.yml — remove custom networks and labels
...
- No custom crm-net network — Coolify manages networking
- No custom Traefik labels — Coolify generates them
- No hardcoded domains — all from environment variables
- Simplified volumes — no custom names
2026-08-04 11:55:04 +02:00
51c9b467b2
Fix: verify_ws_origin async + await callers
...
Check Cross-Plugin Imports / check (push) Has been cancelled
- auth.py: verify_ws_origin is now async def
- kommunikation/routes.py: await verify_ws_origin
- ai_ui_control/routes.py: await verify_ws_origin
2026-08-04 11:41:18 +02:00
7d3007b6c4
Fix: Remove hardcoded UUIDs and secrets from deploy scripts
...
- deploy.py: UUIDs from env vars or Coolify API lookup by name
- fast-deploy.sh: No hardcoded UUIDs, APP_DOMAIN from env
- docker-compose.yml: All secrets from env vars, no hardcoded values
- .env.example: All required vars documented
- Deleted obsolete fast-frontend-deploy.sh with hardcoded container name
2026-08-04 11:13:44 +02:00
e7edc46286
Phase 3: WebSocket CSRF, SameSite=Lax, Tenant FK CASCADE
2026-08-04 09:27:10 +02:00
4a104af615
Fix duplicate networks key in crm-app service
2026-08-04 00:37:58 +02:00
6481996334
Add Traefik labels and coolify network for crm.media-on.de routing
2026-08-04 00:31:56 +02:00
51a2c44238
Fix: Migration 0102 checks table existence before adding owner_id
2026-08-04 00:18:06 +02:00
40e9943e69
Fix: Remove external port binding from crm-app (conflicts with Coolify on port 8000)
2026-08-04 00:11:30 +02:00
e17b9c9e56
Phase 2: Visibility Filter & Owner ID
...
Check Cross-Plugin Imports / check (push) Has been cancelled
- Add OwnedMixin to 15 models (contact_folder, user_preference, workspace,
mcp_server_config, agent_definition, automation_definition, report_template,
report_instance, entity_link, comm_conversation, proactive_suggestion,
ai_agent, ai_chat_session, tag, share_link)
- Migration 0102: Add owner_id column to 15 tables with backfill from user_id
- Fix EntityPermission Registry: remove notification, add entity_attachment,
entity_history, subtask, calendar, folder; fix wrong class names
(DmsFile→File, CalendarEvent→CalendarEntry, Mailbox→MailAccount)
- Add apply_visibility_filter to list endpoints in tags, tasks, mcp_client,
automation, report_generator, ai_assistant routes
- Add owner_id to create handlers for all new OwnedMixin models
- Patch tasks/services.py and automation/services.py list methods with
user_id and is_system_admin parameters
2026-08-04 00:03:29 +02:00
93a330ae40
Fix: Simplify migration 0101 — only disable RLS on auth tables
2026-08-03 23:14:26 +02:00
d43407ca77
Fix: Disable RLS on auth tables, correct policy syntax
2026-08-03 23:06:01 +02:00
4f970a11eb
Phase 1: Critical security fixes - 59 permissions, grants, RLS, mass-assignment, ownership, leaks, MIME
Check Cross-Plugin Imports / check (push) Has been cancelled
2026-08-03 22:32:03 +02:00
bd9fc15418
A1: Permission-Checks fuer delegations, policies, permission_templates, ai_copilot
2026-08-03 21:42:11 +02:00
f043be44be
Statement Timeout: 30s -> 300s (5min) — nur echte Hänger abbrechen, normale Queries/Reports/Bulk laufen durch
2026-08-03 20:40:56 +02:00
3622120cd6
Optimierung: approximate_count fuer contact_service (0.06ms statt 700ms bei 1M+ rows)
...
- pg_class.reltuples fuer Tabellen >100 Zeilen (5000x schneller)
- Exact count nur fuer kleine Tabellen <100 Zeilen
- Generic pagination.py Utility fuer alle Services verfuegbar
2026-08-03 20:23:08 +02:00
662916a8cb
Allgemeine Performance Optimierungen fuer 1M+ Datensaetze
...
1. Generic Pagination Utility (app/core/pagination.py):
- approximate_count: pg_class.reltuples statt SELECT count(*) (5000x schneller)
- paginated_list: Generic keyset/offset pagination fuer alle Services
- use_approximate_count Option fuer grosse Tabellen
2. Connection Pool erhoeht:
- pool_size: 10 -> 20
- max_overflow: 20 -> 30
- 3 Engines = 150 Connections max (fuer 100+ User)
3. Statement Timeout (30s):
- Verhindert dass langsame Queries die API blockieren
- connect_args server_settings statement_timeout=30000
Tests: 43/43 bestanden
2026-08-03 20:16:16 +02:00
5863004727
Fix: Keyset-Pagination use_keyset unabhaengig von cursor (erste Seite ohne cursor)
2026-08-03 19:21:19 +02:00
5d35f0064e
Performance Optimierungen: Rate Limit, created_at Index, Keyset-Pagination
...
1. Rate Limit erhoeht: 60 -> 300 Requests/Minute (fuer 100+ User)
2. Migration 0099: created_at DESC Index auf allen Tabellen (Order by Performance)
3. Keyset-Pagination: optionaler cursor Parameter fuer contacts API
- cursor=UUID nutzt WHERE id > cursor statt OFFSET
- Backward compatible: ohne cursor wird page/page_size genutzt
- next_cursor in Response fuer naechste Seite
Tests: 43/43 bestanden
2026-08-03 19:18:09 +02:00
67c05f39f1
Fix: require_active_plugin nutzt get_db Session + current_setting (keine neue Session)
2026-08-03 16:26:06 +02:00
1271101acd
Fix: require_active_plugin nutzt Request statt get_current_user Dependency
2026-08-03 16:23:21 +02:00
19dd0aa74f
Phase 7.2: WebSocket Plugin-Gate fuer kommunikation und ai_ui_control
...
Check Cross-Plugin Imports / check (push) Has been cancelled
- Beide WebSocket Endpoints pruefen jetzt Plugin-Aktivierung (global + tenant)
- Fail-closed bei Fehlern
- Pruefung direkt im WebSocket-Endpunkt (Router-Dependency greift bei WS nicht)
2026-08-03 16:20:27 +02:00
fe6a4fdd54
Fix: Migration 0094 prueft Spalten-Existenz vor GIN-Index-Erstellung
2026-08-03 16:14:43 +02:00
d5daeb8dfd
Phase 9: Verbindlicher Abschlussbericht (RECOVERY_ACCEPTANCE_REPORT.md)
Check Cross-Plugin Imports / check (push) Has been cancelled
2026-08-03 15:50:39 +02:00
485fbd9877
Phase 8.3+8.4: Restore-Test Script und Coolify-Endabnahme
...
8.3 Restore-Test:
- restore_test.sh: PostgreSQL Backup restore, Migrationen, Data Integrity, RLS Re-test
- Prueft Alembic Version, Table Count, RLS >= 100, Contacts > 0
- RLS Re-test: 0 rows ohne/fake tenant context
- Erfordert TEST_DATABASE_URL (separate Test-DB)
8.4 Coolify-Endabnahme (live verifiziert):
- API healthy: DB up, Redis up, Worker up
- Worker healthy: running:healthy
- Login: admin@media-on.de , admin, Default Org
- Workspace Wechsel: 1 Workspace, Context modules mit is_visible
- DMS Upload + Download: HTTP 200, Content korrekt
- MCP Read: 1 Tool (call_crm_api), Auth api-token
- Outbox: 5 published events
- Token CRUD: Create, List, Revoke (204)
2026-08-03 15:50:11 +02:00
f4364f30e0
Phase 8.1+8.2: CI Pipeline und Migrations-Release-Gate
...
8.1 Merge-CI:
- Backend Tests und Frontend Tests zu ci_pipeline.sh hinzugefuegt
- Migration Hash Check (<=0092) mit check_migration_hashes.py
- npm ci --legacy-peer-deps in Forgejo Workflow und ci_pipeline.sh
- 93 Migration-Hashes generiert und verifiziert
8.2 Migrations-Release-Gate:
- migration_release_gate.sh: Fresh Install, Schema Snapshot, RLS/Grants Check, Cross-Tenant Test, Data Integrity
- Prueft leere DB Installation mit Alembic Head + Plugin-Migrationen
- Verifiziert RLS >= 100 Tabellen, 4 DB-Rollen, kein BYPASSRLS auf crm_api
- Cross-Tenant: 0 rows ohne/fake tenant context
2026-08-03 15:49:03 +02:00
0260f3410d
Phase 7: Plugin-Gate, Event-Envelope, Pro-Handler Outbox-Verarbeitung
...
7.1 Plugin-Gate korrigiert:
- require_active_plugin nutzt current_user fuer tenant_id statt current_setting()
- Keine neue DB-Session mehr — nutzt bestehende get_db Dependency
- Fail-closed bei Fehlern
7.4 Einheitlicher Event-Envelope:
- Sauberes Envelope mit event_id, event_name, tenant_id, aggregate_type, aggregate_id, occurred_at, correlation_id, schema_version, data
- Keine _-Praefixe mehr im payload
- Handler empfangen envelope statt rohes payload
7.6 Verarbeitung pro Handler:
- Globaler consumer_inbox Check entfernt
- Pro-Handler Idempotency: outbox_deliveries pruefen ob Handler bereits erfolgreich
- Bereits erfolgreiche Handler werden uebersprungen
- consumer_inbox pro Handler geschrieben
7.7 no_handlers: Bereits implementiert (terminaler Status)
7.8 Cron-Jobs: Bereits mit Redis SET NX Locking implementiert
Tests: 23/23 Outbox-Tests bestanden
2026-08-03 15:20:06 +02:00
8d82df3076
Fix: LocalStorage top-level import in DMS routes
Check Cross-Plugin Imports / check (push) Has been cancelled
2026-08-03 15:08:13 +02:00
8b683c7da7
Phase 6.5 Fix: DMS Download Endpoint fuer alle Dateitypen
...
Check Cross-Plugin Imports / check (push) Has been cancelled
- GET /api/v1/dms/files/{file_id}/download streamt alle Dateitypen
- FileResponse fuer LocalStorage (automatisches Streaming)
- StreamingResponse Fallback fuer S3
- Prueft dms:read Permission und entity access
2026-08-03 15:02:39 +02:00
29d55cb187
Phase 6: DMS & Attachments — Streaming, Deduplikation, API-Bereinigung
...
Check Cross-Plugin Imports / check (push) Has been cancelled
6.4 Upload streamen:
- attachment_service.save_attachment: Streamt in 1MB Chunks statt await file.read()
- routes/attachments.py: Uebergibt UploadFile direkt statt bytes
6.5 Download streamen:
- DMS preview_file: FileResponse fuer LocalStorage (automatisches Streaming)
- Kein storage.read() mehr fuer LocalStorage
6.6 Tenantlokale Deduplikation:
- DMS Upload: Prueft content_hash vor Erstellung, wiederverwendet existierendes File
- attachment_service: Dedup bereits vorhanden, jetzt mit Streaming kompatibel
- Migration 0098: Partial Unique Index (tenant_id, content_hash) WHERE content_hash IS NOT NULL AND deleted_at IS NULL
6.7 API-Ausgabe bereinigt:
- attachment_service: storage_path und content_hash aus API-Ausgaben entfernt
- DMS routes: content_hash aus 4 API-Endpunkten entfernt
Tests: 54/54 bestanden (17 Workspace + 13 API Token + 24 Command)
2026-08-03 14:21:43 +02:00
ff975ca0a6
Fix: MCP list_mcp_tools + config Routes auf Bearer-Auth umstellen
Check Cross-Plugin Imports / check (push) Has been cancelled
2026-08-03 14:12:44 +02:00
Agent Zero