662916a8cb
Allgemeine Performance Optimierungen fuer 1M+ Datensaetze
...
1. Generic Pagination Utility (app/core/pagination.py):
- approximate_count: pg_class.reltuples statt SELECT count(*) (5000x schneller)
- paginated_list: Generic keyset/offset pagination fuer alle Services
- use_approximate_count Option fuer grosse Tabellen
2. Connection Pool erhoeht:
- pool_size: 10 -> 20
- max_overflow: 20 -> 30
- 3 Engines = 150 Connections max (fuer 100+ User)
3. Statement Timeout (30s):
- Verhindert dass langsame Queries die API blockieren
- connect_args server_settings statement_timeout=30000
Tests: 43/43 bestanden
2026-08-03 20:16:16 +02:00
5863004727
Fix: Keyset-Pagination use_keyset unabhaengig von cursor (erste Seite ohne cursor)
2026-08-03 19:21:19 +02:00
5d35f0064e
Performance Optimierungen: Rate Limit, created_at Index, Keyset-Pagination
...
1. Rate Limit erhoeht: 60 -> 300 Requests/Minute (fuer 100+ User)
2. Migration 0099: created_at DESC Index auf allen Tabellen (Order by Performance)
3. Keyset-Pagination: optionaler cursor Parameter fuer contacts API
- cursor=UUID nutzt WHERE id > cursor statt OFFSET
- Backward compatible: ohne cursor wird page/page_size genutzt
- next_cursor in Response fuer naechste Seite
Tests: 43/43 bestanden
2026-08-03 19:18:09 +02:00
67c05f39f1
Fix: require_active_plugin nutzt get_db Session + current_setting (keine neue Session)
2026-08-03 16:26:06 +02:00
1271101acd
Fix: require_active_plugin nutzt Request statt get_current_user Dependency
2026-08-03 16:23:21 +02:00
19dd0aa74f
Phase 7.2: WebSocket Plugin-Gate fuer kommunikation und ai_ui_control
...
Check Cross-Plugin Imports / check (push) Has been cancelled
- Beide WebSocket Endpoints pruefen jetzt Plugin-Aktivierung (global + tenant)
- Fail-closed bei Fehlern
- Pruefung direkt im WebSocket-Endpunkt (Router-Dependency greift bei WS nicht)
2026-08-03 16:20:27 +02:00
0260f3410d
Phase 7: Plugin-Gate, Event-Envelope, Pro-Handler Outbox-Verarbeitung
...
7.1 Plugin-Gate korrigiert:
- require_active_plugin nutzt current_user fuer tenant_id statt current_setting()
- Keine neue DB-Session mehr — nutzt bestehende get_db Dependency
- Fail-closed bei Fehlern
7.4 Einheitlicher Event-Envelope:
- Sauberes Envelope mit event_id, event_name, tenant_id, aggregate_type, aggregate_id, occurred_at, correlation_id, schema_version, data
- Keine _-Praefixe mehr im payload
- Handler empfangen envelope statt rohes payload
7.6 Verarbeitung pro Handler:
- Globaler consumer_inbox Check entfernt
- Pro-Handler Idempotency: outbox_deliveries pruefen ob Handler bereits erfolgreich
- Bereits erfolgreiche Handler werden uebersprungen
- consumer_inbox pro Handler geschrieben
7.7 no_handlers: Bereits implementiert (terminaler Status)
7.8 Cron-Jobs: Bereits mit Redis SET NX Locking implementiert
Tests: 23/23 Outbox-Tests bestanden
2026-08-03 15:20:06 +02:00
8d82df3076
Fix: LocalStorage top-level import in DMS routes
Check Cross-Plugin Imports / check (push) Has been cancelled
2026-08-03 15:08:13 +02:00
8b683c7da7
Phase 6.5 Fix: DMS Download Endpoint fuer alle Dateitypen
...
Check Cross-Plugin Imports / check (push) Has been cancelled
- GET /api/v1/dms/files/{file_id}/download streamt alle Dateitypen
- FileResponse fuer LocalStorage (automatisches Streaming)
- StreamingResponse Fallback fuer S3
- Prueft dms:read Permission und entity access
2026-08-03 15:02:39 +02:00
29d55cb187
Phase 6: DMS & Attachments — Streaming, Deduplikation, API-Bereinigung
...
Check Cross-Plugin Imports / check (push) Has been cancelled
6.4 Upload streamen:
- attachment_service.save_attachment: Streamt in 1MB Chunks statt await file.read()
- routes/attachments.py: Uebergibt UploadFile direkt statt bytes
6.5 Download streamen:
- DMS preview_file: FileResponse fuer LocalStorage (automatisches Streaming)
- Kein storage.read() mehr fuer LocalStorage
6.6 Tenantlokale Deduplikation:
- DMS Upload: Prueft content_hash vor Erstellung, wiederverwendet existierendes File
- attachment_service: Dedup bereits vorhanden, jetzt mit Streaming kompatibel
- Migration 0098: Partial Unique Index (tenant_id, content_hash) WHERE content_hash IS NOT NULL AND deleted_at IS NULL
6.7 API-Ausgabe bereinigt:
- attachment_service: storage_path und content_hash aus API-Ausgaben entfernt
- DMS routes: content_hash aus 4 API-Endpunkten entfernt
Tests: 54/54 bestanden (17 Workspace + 13 API Token + 24 Command)
2026-08-03 14:21:43 +02:00
ff975ca0a6
Fix: MCP list_mcp_tools + config Routes auf Bearer-Auth umstellen
Check Cross-Plugin Imports / check (push) Has been cancelled
2026-08-03 14:12:44 +02:00
8ad0a19f25
Phase 5: AI/MCP Bearer-Auth + Delegationstoken + Audit
...
Check Cross-Plugin Imports / check (push) Has been cancelled
5.1 Delegationstoken (app/core/delegation_token.py):
- HMAC-SHA256 signiert mit SECRET_KEY, max 60s Lifetime
- Payload: user_id, tenant_id, agent_id, audience, expires_at, token_id
- Statelose Verifikation, Audience-Check, Expiry-Check
5.2 MCP Bearer-Auth:
- app/core/api_token.py: Token Service (create, verify, revoke, list)
- app/deps.py: get_current_user_bearer + get_current_user_or_bearer
- app/routes/api_tokens.py: Token CRUD Routes (create, list, revoke)
- MCP Server Routes: get_current_user_or_bearer akzeptiert Session + Bearer
5.3 Methodenrechte:
- MCP nutzt bereits mcp:read/mcp:write basierend auf tool_def.required_permission
5.5 Audit:
- MCP Tool-Ausfuehrung wird protokolliert (log_audit mit correlation_id)
Tests: 13/13 bestanden (7 API Token + 6 Delegation Token)
2026-08-03 14:06:55 +02:00
a760a759eb
Phase 0+3: Stand sichern, alte Doku einfrieren, doppelte Command-Struktur entfernen
...
Phase 0:
- Git Tag: pre-recovery-current (3cbf921
2026-08-03 13:25:48 +02:00
3cbf92191e
Reparaturplan Fixes: Widget workspace_id check, total bug, context is_visible, permissions, fallbacks
...
Check Cross-Plugin Imports / check (push) Has been cancelled
Backend:
- Widget total: 0 bug fixed (now returns len(widgets))
- Widget update/delete: now verifies workspace_id + tenant_id (was only tenant_id)
- Workspace context: returns all modules with is_visible flag (was only visible modules)
- is_workspace_manager() removed (Plan 4.2: no manager checks)
- seed_default_workspace: removed hardcoded modules (Plan 4.7: no hardcoded tiles)
- Workspace permissions registered in CORE_PERMISSIONS (Plan 2.3)
Frontend:
- Permission fallback removed: Sidebar/TopBar show nothing while loading (Plan 2.4)
- workspaceStore isModuleVisible: fail-closed when isSystemAdmin undefined
- WorkspaceManager: AVAILABLE_MODULES replaced with dynamic core+plugin items (Plan 4.4)
Tests:
- 17 backend tests (removed is_workspace_manager test, adapted widget/context tests)
- 13 frontend tests (added undefined-isSystemAdmin test, adapted visibility tests)
2026-08-03 12:44:02 +02:00
310a9f0542
Phase 6: Workspaces — Widget CRUD, Manager-Check, Cross-Tenant, Zustand Store, Settings Route
...
Backend:
- Widget CRUD: get_widgets, create_widget, update_widget, delete_widget
- Manager role check: is_workspace_manager
- Cross-tenant validation: verify_user_same_tenant (UserTenant)
- Default workspace seeding: seed_default_workspace with 12 standard modules
- Set user default workspace: set_user_default_workspace
- Fix create_workspace default uniqueness (unset others before insert)
- Widget CRUD routes: GET/POST/PUT/DELETE /{workspace_id}/widgets
- Set-default route: POST /{workspace_id}/set-default
- Cross-tenant validation in assign_user route
Frontend:
- workspaceStore (Zustand): central state with sessionStorage persistence
- API client interceptor: X-Workspace-ID header on all requests
- useWorkspace hook refactored to use workspaceStore
- Widget API hooks: useWorkspaceWidgets, useCreateWorkspaceWidget, etc.
- useSetDefaultWorkspace hook
- Settings route: /settings/workspaces with WorkspaceManagerPage
- Settings nav item for Workspaces
Tests:
- 25 backend tests (CRUD, modules, widgets, users, manager, seeding, context, isolation)
- 12 frontend tests (workspaceStore state, visibility, persistence, reset)
- 48/48 backend tests passing
- 12/12 frontend tests passing
2026-08-03 03:39:27 +02:00
74936b3972
Phase 5 (v2): Processing-Recovery, Retention-Cleanup, Replay-Delivery-Reset
...
- recover_stuck_events: Reset processing events stuck >120s back to pending
- cleanup_published_events: Delete published events older than 30 days
- Replay now resets outbox_deliveries for clean retry
- Worker: hourly retention cleanup cron job
- API: /recover-stuck and /cleanup-published endpoints
- process_outbox_batch: auto-recovery at start of each tenant iteration
- 23/23 tests passing (5 new tests)
2026-08-02 23:47:29 +02:00
07a99975ec
Phase 5: Outbox DLQ, Monitoring, Consumer-Registry
...
- Migration 0092: DLQ columns (error_message, failed_at) + consumer_inbox RLS fix
- outbox.py: DLQ logic, replay functions, stats, consumer registry
- app/routes/outbox.py: 5 API endpoints (stats, failed, replay, replay-all, consumer-registry)
- outbox_deliveries tracking per consumer handler
- 18/18 tests passing
2026-08-02 23:25:54 +02:00
94847ea515
fix: PluginModel.is_active → PluginModel.active (worker crash fix)
2026-07-31 23:12:05 +02:00
cea21ff576
fix: Gate 5 — worker event handlers and per-tenant outbox processing
...
Worker fixes:
- registry.initialize uses get_migration_engine() for DDL (not worker_engine)
- Worker session uses get_worker_session_factory() (crm_worker, not crm_api)
- Event handlers only registered for active plugins (is_active check)
- Outbox processing per-tenant with set_config(app.current_tenant_id)
- process_outbox_job uses get_worker_session_factory() and loads tenant_ids
- Removed unused get_engine import
Outbox fixes:
- process_outbox_batch iterates over tenants, sets RLS context per tenant
- _process_single_outbox_event extracted for clarity
- Events claimed per-tenant (RLS-compatible, no BYPASSRLS needed)
- Commit after each tenant to release locks
Gate 5 requirements met:
- Plugin event handlers registered for active plugins only
- No plugin routers registered in worker
- Outbox events without handlers marked as no_handlers
- Failed consumers trigger retry with exponential backoff
- Processing is idempotent (consumer_inbox check)
- Every worker DB access sets app.current_tenant_id
- Worker cannot read/write other tenant data (RLS enforced)
2026-07-31 23:09:25 +02:00
48ddd78e9e
gate2: fix mail plugin migration 0009 — guard UPDATE for missing deleted_at column on fresh DB
Check Cross-Plugin Imports / check (push) Has been cancelled
2026-07-31 20:56:06 +02:00
4a5c905934
P0-fix: plugin migrations use migration engine (crm_migration) instead of API engine (crm_api)
...
Check Cross-Plugin Imports / check (push) Has been cancelled
- main.py: registry.initialize(get_migration_engine()) instead of get_engine()
- main.py: plugin migrations run via get_migration_session_factory() not async_session()
- registry.py: upgrade_plugin, install_plugin, uninstall_plugin all use migration session for DDL
- db/__init__.py: get_migration_engine() raises RuntimeError if MIGRATION_DATABASE_URL missing (no fallback)
- Fixes fresh-install failure: crm_api has no DDL rights, plugin migrations need crm_migration
2026-07-31 20:45:16 +02:00
a303a4e455
gate4: use separate API session for audit log in confirm_password_reset
2026-07-31 12:01:11 +02:00
a721db5214
gate4: set tenant context before audit log in confirm_password_reset
2026-07-31 11:58:11 +02:00
ce0e9ab12a
gate4: fix SMTP TLS mode for port 465 (implicit TLS instead of STARTTLS)
2026-07-31 11:37:11 +02:00
31408670e6
gate4: register app.core.jobs in worker for send_password_reset_email
2026-07-31 11:32:47 +02:00
fa96466a50
gate: fresh session per plugin activation to isolate RLS errors
2026-07-31 09:43:29 +02:00
ab8d878bc7
gate: db.expunge_all() after rollback to clear pending objects from failed INSERTs
2026-07-31 09:41:32 +02:00
d114fd7d4c
gate: wrap db.commit() in try/except after plugin activation
2026-07-31 09:39:45 +02:00
79d132b66d
gate: fully resilient plugin activation in API startup
2026-07-31 09:37:44 +02:00
01aa31a3e0
gate: API startup resilient to RLS errors, dont fail on duplicate cron job inserts
2026-07-31 09:33:41 +02:00
31d11efd33
gate: API main.py flush+rollback after plugin activation for RLS error handling
2026-07-31 09:32:15 +02:00
9d7b160e2a
gate: fix password_reset RLS policy for crm_auth, set tenant context before token creation
2026-07-31 09:24:04 +02:00
1deb852ff3
gate: worker skips plugin activation, only registers event handlers
Check Cross-Plugin Imports / check (push) Has been cancelled
2026-07-31 09:20:54 +02:00
ab61c81d2b
gate: worker flush after plugin activation to detect swallowed RLS errors
2026-07-31 09:19:45 +02:00
ec0cf6f588
gate: worker resilient to RLS errors during plugin activation, use crm_worker engine
2026-07-31 09:12:42 +02:00
5ce85f4324
gate: fix worker on_startup to set tenant context per-tenant for plugin activation
2026-07-31 09:09:23 +02:00
1a980ba9d8
gate: migration 0086, crm_migration BYPASSRLS, audit_log fix, CI test for app.tenant_id
...
- Migration 0086: Remove FORCE RLS from 5 global tables
- Migration 0085: crm_migration keeps BYPASSRLS for data migrations
- Migration 0085: Remove audit_log from crm_auth grants
- auth_service.py: Audit log via separate API session (crm_api with tenant context)
- tests/test_no_legacy_tenant_var.py: CI test for app.tenant_id in policies
2026-07-31 09:02:40 +02:00
15f0a07d4e
phase1: fix auth_service tenant context for audit_log, add sessions+audit_log to crm_auth grants
...
- auth_service.py: set tenant context before audit log write in login
- migration 0085: add sessions and audit_log to AUTH_TABLES for crm_auth
- Login now works on production with RLS enabled
2026-07-31 02:28:29 +02:00
100b9f705c
phase1: separate DB roles, RLS restoration, login on crm_auth
...
Check Cross-Plugin Imports / check (push) Has been cancelled
- config.py: add auth_database_url, worker_database_url, migration_database_url
- db/__init__.py: separate engines for auth/worker/migration + get_auth_db/get_worker_db
- auth.py: all auth endpoints use get_auth_db (crm_auth role)
- auth_service.py: remove login fallback, require active membership, check status
- auth_service.py: switch_tenant checks active membership status
- alembic/env.py: use migration_database_url for Alembic
- docker-compose.yml: add AUTH_DATABASE_URL, WORKER_DATABASE_URL
- .env.example: add all 4 DB URLs with separate roles
- migration 0085: transfer ownership to crm_migration, fix BYPASSRLS,
enable RLS+FORCE on all tenant tables, drop old policies, create new
fail-closed policies scoped to crm_api+crm_worker, revoke excessive grants,
grant minimal crm_auth access, drop crm_runtime, set default privileges
- tests/test_rls_coverage.py: automated RLS coverage check (13 tests)
- tests/test_cross_tenant_security_v2.py: RLS tests with unprivileged role
2026-07-31 02:05:16 +02:00
032a7e80a8
phase0: fix cross-plugin import, remove app.tenant_id, create cross-tenant v2 tests
...
- Fix report_generator/jobs.py: use DmsContract instead of direct DMS import
- Remove app.tenant_id from set_tenant_context (only app.current_tenant_id)
- Create tests/test_cross_tenant_security_v2.py with real RLS tests using
unprivileged crm_api role (NOSUPERUSER, NOBYPASSRLS)
- Fix existing tests referencing app.tenant_id
- Git baseline tag v-phase0-baseline at 11d6faa
2026-07-31 01:57:51 +02:00
0692fce2e4
fix: RLS fail-closed migration + per-tenant startup code
2026-07-31 01:31:41 +02:00
7fbbe420bd
fix: comprehensive system audit fixes (55+ issues)
...
Check Cross-Plugin Imports / check (push) Has been cancelled
CRITICAL:
- Fix SQL injection in prestart.sh (parameterized query)
- Fix secret key validation (always validate, not just production)
- Fix workspace model partial index bug (func.text -> text)
- Fix HealthResponse schema (add checks field)
- Fix Tenant import in permissions.py (NameError on every auth request)
- Fix README tech stack (React instead of Alpine.js)
- Delete broken test_cross_tenant_security_v2.py
- Add fail-closed RLS migration 0084 (48 tenant tables)
HIGH:
- Add GeneralRateLimitMiddleware for all API routes
- Add file type blocklist for DMS and attachment uploads
- Fix guest auth: Pydantic schema, tenant_slug required, CSRF bypass
- Fix CSRF bypass path matching (in -> endswith)
- Add worker healthcheck in docker-compose.yml
- Add ARQ max_tries=3 for job retries
- Fix 28 bare pass in mail services (-> logger.debug)
- Fix print() -> logger in main.py and ai_assistant
- Fix duplicate email handling (catch IntegrityError -> 409)
- Add session revocation (invalidate_all_user_sessions)
- Add resource limits to all containers
- Fix CORS default (localhost -> production domain)
- Fix SameSite=Lax -> Strict
- Fix Redis password visibility in healthcheck
- Fix npm vulnerabilities (19 -> 9)
- Fix Sidebar OOM (wildcard lucide import -> curated ICON_MAP)
MEDIUM:
- Localize ErrorBoundary to German
- Wire Mail.tsx save/delete filter to API
- Document system_notif plugin (no routes needed)
- Fix datetime.utcnow() -> datetime.now(UTC)
- Pin litellm version (>=1.0,<2.0)
- Move CSRF token from sessionStorage to in-memory
- Fix restore_backup error handling and transaction
- Fix Dms.tsx useEffect cleanup
- Add skip-to-content link for accessibility
- Add selectinload imports to 3 services
- Add .env.example missing variables
- Fix AppShell/TopBar/Sidebar test mocks
NEW TESTS:
- test_guest_auth.py (6 tests)
- test_user_service.py (8 tests)
- test_backup_service.py (5 tests)
NEW SCHEMAS:
- saved_filter, saved_view, user_preference, workspace, entity_policy
Tests: 22/22 PASSED
2026-07-31 00:58:05 +02:00
f7c60069d5
fix: increase mail page_size limit to 10000 for grouping all mails
Check Cross-Plugin Imports / check (push) Has been cancelled
2026-07-30 18:45:31 +02:00
a9a9476e9f
fix: contact_folder_service db.refresh after commit + AI stream own DB session in generator
Check Cross-Plugin Imports / check (push) Has been cancelled
2026-07-30 12:49:53 +02:00
acea622a0f
fix: AI loop prevention (no tools on last iteration), calendar button first, mail filter in toolbar, AI folder rename query invalidation
Check Cross-Plugin Imports / check (push) Has been cancelled
2026-07-30 11:33:45 +02:00
124846ae3b
fix: workflow route remove is_system_admin param not accepted by service
2026-07-30 11:16:56 +02:00
2cd3f30f82
fix: workflow 422 validation + report 500 DB data fetching + AI stream tenant context
Check Cross-Plugin Imports / check (push) Has been cancelled
2026-07-30 10:54:28 +02:00
3f2f594847
fix: AI stream route missing set_tenant_context causing RLS INSERT failure
Check Cross-Plugin Imports / check (push) Has been cancelled
2026-07-30 10:45:34 +02:00
2836d6083e
fix: add async_session_maker alias in db/__init__.py for plugin imports
2026-07-30 02:48:21 +02:00
88bcbfa9a8
fix: add app/core/redis.py shim re-exporting get_redis from auth
2026-07-30 02:01:28 +02:00
Agent Zero