e7ae0ad5ce
feat(C.5): Modularer Import/Export — Shared Helpers, Preview/Mapping, Background Jobs, Partial-Failure
...
C5-BASE: app/services/import_export_helpers.py (NEU, 352 Zeilen)
- parse_csv/json/xlsx, write_csv/json/xlsx, map_fields, suggest_mapping
- validate_row, build_error_report, build_import_result, detect_format
C5-PREVIEW: POST /import/preview + POST /import/validate
- Preview gibt erste 10 Zeilen + Spalten + Mapping-Vorschlag
- Validate gibt Fehler-Report ohne Import
C5-JOB: app/services/import_export_jobs.py (NEU, 165 Zeilen)
- ARQ Background Job für Files >1000 Zeilen
- Job-Status: pending/processing/completed/partial_success/failed
- GET /import/status/{job_id} — Status + Progress + Fehler-Report
- Partial-Failure: try/except pro Zeile, fehlerhafte gesammelt, erfolgreiche committet
C5-CONTACT+C5-COMPANY: Handler auf Shared Helpers umgestellt
C5-UI: ImportWizard.tsx (5 Steps: Upload→Preview→Validation→Review→Result)
C5-TEST: 45 Tests in test_import_export.py — alle grün
C5-DOC: Plugin-Dev-Guide Kapitel 31 (Import/Export Handler)
2026-08-13 22:43:33 +02:00
b5546ea7bd
feat(B.13-B.17+B.16): Error-Handling-Infra, Observability, Graceful Shutdown, Cost-Cap, API Versioning
...
B.13 Error-Handling-Infrastruktur:
- ErrorCategory Enum (TRANSIENT/PERMANENT/PARTIAL), ApiError erweitert
- Einheitliches Error-Response-Format: {code, detail, field, trace_id, retryable, category}
- 3 FastAPI Exception-Handler (ApiError, HTTPException, unhandled)
- classify_exception() Helper, 6 neue Error-Codes
- 28 Tests in test_error_handling.py
B.14 Observability & trace_id-Korrelation:
- trace_id pro Request (UUID4 short) in structlog contextvars
- X-Trace-Id Response-Header
- Sensitive Fields structlog processor
- llm_complete()/llm_embed() akzeptieren trace_id kwarg
- 12 Tests in test_observability.py
B.15 Graceful Shutdown & Connection Draining:
- _shutdown_event + _inflight_requests Tracking in main.py
- drain_all_connections() in ws_helpers.py
- Worker on_shutdown pausiert WorkflowInstances (status=paused)
- 8 Tests in test_graceful_shutdown.py
B.16 API Versioning Strategie:
- Plugin-Dev-Guide Kapitel 30: URL-basiertes Versioning, Breaking Change Prozess
B.17 Cost Overrun Protection:
- llm_monthly_budget_usd + llm_hard_cutoff Settings
- _check_tenant_budget() vor jedem LLM-Call
- _track_tenant_cost() in Redis (INCRBYFLOAT)
- _check_cost_alerts() bei 50%/80%/100% -> post_system_message()
- 20 Tests in test_cost_protection.py
Total: 68 neue Tests, alle grün. Keine Regressionen.
2026-08-13 21:33:14 +02:00
78963f2ca9
feat(B-TRIG): Trigger-Kern konsolidiert — generischer Dispatcher, UI-Event-Typ, 4 Trigger-Typen
...
Check Cross-Plugin Imports / check (push) Has been cancelled
B-TRIG-GEN: app/core/trigger_dispatcher.py (NEU) — generischer Event→Automation Dispatcher
- Wildcard EventBus Subscribe → matcht AutomationDefinition mit trigger_type=event
- Keine hardcodierte Event-Liste mehr — alle Outbox Events können Automations triggern
- Registriert in main.py lifespan + worker.py on_startup
B-TRIG-UI: UI-Event Trigger-Typ
- trigger_type Pattern um „ui" erweitert (event/schedule/manual/ui)
- UI-Events laufen ephemeral über EventBus → TriggerDispatcher → run_automation
- UI-Events NIEMALS in Outbox (is_ui_event() Guard)
B-TRIG-CRON: Cron/Heartbeat verifiziert — bestehender Pfad funktioniert
B-TRIG-MAN: Manual-Trigger verifiziert — bestehender Pfad funktioniert
B-TRIG-TEST: 10 Tests in test_trigger_core.py — alle grün
- Domain Event, UI Event (ephemeral), Cron, Manual, alle nutzen selben Execution-Kern
B-TRIG-DOC: Plugin-Dev-Guide Kapitel 10 mit Implementierungsdetails erweitert
2026-08-13 21:04:46 +02:00
e9164979b5
feat(B-RED): Zentraler Redis Pool — cache.py, monitoring.py, worker.py auf get_redis() umgestellt
...
B-RED: 5 direkte aioredis.from_url() Konstruktoren auf get_redis() umgestellt
- cache.py: get_cache() delegiert auf get_redis(), _cache_redis Singleton entfernt
- monitoring.py: check_redis() und check_worker() nutzen get_redis()
- worker.py: _acquire_cron_lock() und _release_cron_lock() nutzen get_redis()
- 0 verbleibende direkte aioredis.from_url() außerhalb auth.py
B-RED-TEST: 8 Tests in test_redis_pool.py — alle grün
- Singleton, get_cache delegation, SET/GET, parallel, reset, no-direct-from_url checks
2026-08-13 16:25:17 +02:00
74936b3972
Phase 5 (v2): Processing-Recovery, Retention-Cleanup, Replay-Delivery-Reset
...
- recover_stuck_events: Reset processing events stuck >120s back to pending
- cleanup_published_events: Delete published events older than 30 days
- Replay now resets outbox_deliveries for clean retry
- Worker: hourly retention cleanup cron job
- API: /recover-stuck and /cleanup-published endpoints
- process_outbox_batch: auto-recovery at start of each tenant iteration
- 23/23 tests passing (5 new tests)
2026-08-02 23:47:29 +02:00
94847ea515
fix: PluginModel.is_active → PluginModel.active (worker crash fix)
2026-07-31 23:12:05 +02:00
cea21ff576
fix: Gate 5 — worker event handlers and per-tenant outbox processing
...
Worker fixes:
- registry.initialize uses get_migration_engine() for DDL (not worker_engine)
- Worker session uses get_worker_session_factory() (crm_worker, not crm_api)
- Event handlers only registered for active plugins (is_active check)
- Outbox processing per-tenant with set_config(app.current_tenant_id)
- process_outbox_job uses get_worker_session_factory() and loads tenant_ids
- Removed unused get_engine import
Outbox fixes:
- process_outbox_batch iterates over tenants, sets RLS context per tenant
- _process_single_outbox_event extracted for clarity
- Events claimed per-tenant (RLS-compatible, no BYPASSRLS needed)
- Commit after each tenant to release locks
Gate 5 requirements met:
- Plugin event handlers registered for active plugins only
- No plugin routers registered in worker
- Outbox events without handlers marked as no_handlers
- Failed consumers trigger retry with exponential backoff
- Processing is idempotent (consumer_inbox check)
- Every worker DB access sets app.current_tenant_id
- Worker cannot read/write other tenant data (RLS enforced)
2026-07-31 23:09:25 +02:00
31408670e6
gate4: register app.core.jobs in worker for send_password_reset_email
2026-07-31 11:32:47 +02:00
1deb852ff3
gate: worker skips plugin activation, only registers event handlers
Check Cross-Plugin Imports / check (push) Has been cancelled
2026-07-31 09:20:54 +02:00
ab61c81d2b
gate: worker flush after plugin activation to detect swallowed RLS errors
2026-07-31 09:19:45 +02:00
ec0cf6f588
gate: worker resilient to RLS errors during plugin activation, use crm_worker engine
2026-07-31 09:12:42 +02:00
5ce85f4324
gate: fix worker on_startup to set tenant context per-tenant for plugin activation
2026-07-31 09:09:23 +02:00
7fbbe420bd
fix: comprehensive system audit fixes (55+ issues)
...
Check Cross-Plugin Imports / check (push) Has been cancelled
CRITICAL:
- Fix SQL injection in prestart.sh (parameterized query)
- Fix secret key validation (always validate, not just production)
- Fix workspace model partial index bug (func.text -> text)
- Fix HealthResponse schema (add checks field)
- Fix Tenant import in permissions.py (NameError on every auth request)
- Fix README tech stack (React instead of Alpine.js)
- Delete broken test_cross_tenant_security_v2.py
- Add fail-closed RLS migration 0084 (48 tenant tables)
HIGH:
- Add GeneralRateLimitMiddleware for all API routes
- Add file type blocklist for DMS and attachment uploads
- Fix guest auth: Pydantic schema, tenant_slug required, CSRF bypass
- Fix CSRF bypass path matching (in -> endswith)
- Add worker healthcheck in docker-compose.yml
- Add ARQ max_tries=3 for job retries
- Fix 28 bare pass in mail services (-> logger.debug)
- Fix print() -> logger in main.py and ai_assistant
- Fix duplicate email handling (catch IntegrityError -> 409)
- Add session revocation (invalidate_all_user_sessions)
- Add resource limits to all containers
- Fix CORS default (localhost -> production domain)
- Fix SameSite=Lax -> Strict
- Fix Redis password visibility in healthcheck
- Fix npm vulnerabilities (19 -> 9)
- Fix Sidebar OOM (wildcard lucide import -> curated ICON_MAP)
MEDIUM:
- Localize ErrorBoundary to German
- Wire Mail.tsx save/delete filter to API
- Document system_notif plugin (no routes needed)
- Fix datetime.utcnow() -> datetime.now(UTC)
- Pin litellm version (>=1.0,<2.0)
- Move CSRF token from sessionStorage to in-memory
- Fix restore_backup error handling and transaction
- Fix Dms.tsx useEffect cleanup
- Add skip-to-content link for accessibility
- Add selectinload imports to 3 services
- Add .env.example missing variables
- Fix AppShell/TopBar/Sidebar test mocks
NEW TESTS:
- test_guest_auth.py (6 tests)
- test_user_service.py (8 tests)
- test_backup_service.py (5 tests)
NEW SCHEMAS:
- saved_filter, saved_view, user_preference, workspace, entity_policy
Tests: 22/22 PASSED
2026-07-31 00:58:05 +02:00
ece3cdf75a
feat: report ALL errors to Forgejo — backend 4xx/5xx, unhandled exceptions, worker job failures
...
- main.py: RequestLoggingMiddleware reports 4xx/5xx responses and unhandled exceptions to Forgejo
- worker.py: Plugin activation failures and outbox job failures reported to Forgejo
- 401/403 are NOT reported (expected auth/permission behavior)
- All other errors (422, 404, 500, network, worker) ARE reported
2026-07-27 00:36:37 +02:00
5ec1fc9b05
Phase 1: Fix all critical release blockers (B1-B10)
...
B1: Remove duplicate get_redis() — singleton no longer overwritten
B2: Plugin routes now enforce activation status via require_active_plugin()
B3: Fix UploadFile ForwardRef error — remove functools.wraps from wrap_plugin_route
B4: DMS upload uses true streaming via save_stream() instead of RAM accumulation
B5: Worker on_startup registers plugin event handlers + webhook dispatcher
B6: Implement send_password_reset_email job, remove raw token logging
B7: Webhook SSRF protection (IP validation, no redirects), secret removed from response
B8: RLS repair migration 0044 + separate crm_runtime DB user (NOSUPERUSER, NOBYPASSRLS)
B9: Fix .env.docker.example AUTH_SECRET → SECRET_KEY
B10: Remove Redis default password, remove exposed DB/Redis ports
Also: add frontend_url to config, add SMTP settings to .env.docker.example,
update prestart.sh to use MIGRATION_DATABASE_URL for alembic.
2026-07-26 20:45:42 +02:00
388fbdd109
Fix: ARQ cron second schedule must be set of ints, not string
2026-07-25 22:07:56 +02:00
727d86614e
Security fixes: P0-P2 complete (22 fixes)
...
P0 (7): Auth-bypass removed, migrations fixed, plugin-upload disabled, RLS FORCE+WITH CHECK, plugin double-registration fixed, persistent volume, domain removed
P1 (11): User/tenant model, Redis centralized, worker separated, transactional outbox, XSS fixed, DMS chunked streaming, permissions unified, password reset, metrics secured, config/docs fixed, cross-tenant FK
P2 (4): Contact model normalized, cross-imports reduced 94%, commands+state machines for contacts/dms/mail/calendar, SPA path-traversal
8 new migrations, 99 unit tests, 13 commands, 8 contracts, 72 files changed
2026-07-25 21:03:46 +02:00
aaa7406929
perf: Fix all 7 code analysis issues
...
HIGH (Performance):
- Replace 8 sync file operations with aiofiles in async context (storage, mail,
report_generator, dms_bridge, ai_assistant)
- Frontend bundle splitting: manualChunks for react-vendor, ui-components, tanstack,
markdown, icons, utils, i18n (ui chunk 936K → ~19K)
MEDIUM (Architecture):
- Worker circular deps: Replace direct plugin imports with job_registry.py pattern
(register_job/get_all_jobs, importlib-based lazy loading)
- App-wide ErrorBoundary: New ErrorBoundary.tsx component, wrapped in AppShell
and all standalone routes
LOW (Code Quality):
- N+1 query fix: selectinload(Contact.contact_persons) in list_contacts()
- O(n²) dedup fix: SQL GROUP BY for email/phone duplicates, Dict-based name grouping
- Response format standardization: 7 routes converted from plain arrays to
{items: [...], total: N} format
2026-07-25 09:19:32 +02:00
2c9e74776e
feat(5.21): Tasks Plugin — activities with status, priority, due dates, ARQ reminders
...
- New builtin plugin app/plugins/builtins/tasks/ with full CRUD
- Task model with TenantMixin (title, description, status, priority, due_date, assigned_to, contact_id)
- Routes: GET/POST /tasks, GET/PATCH/DELETE /tasks/{id}, POST /tasks/{id}/assign, POST /tasks/{id}/status
- All routes RBAC-protected (tasks:read, tasks:write, tasks:delete)
- ARQ cron job tasks_due_reminder (daily 8:00) sends notifications for due tasks
- Migration 0001_initial.sql creates tasks table with indexes
- Frontend: Tasks.tsx page with list, filter, create/edit modal, detail modal
- Frontend: api/tasks.ts with React Query hooks
- Route /tasks in routes/index.tsx, sidebar entry via plugin manifest
- i18n keys for nav.tasks and tasks.* in de.json and en.json
- Tests: test_tasks.py (11 tests) + Tasks.test.tsx (3 tests)
- Registered TasksPlugin in conftest.py and worker.py
2026-07-23 23:41:34 +02:00
5dc6f29ac1
Phase 3.5: Automation & Agents Plugin
...
Neues automation Plugin (app/plugins/builtins/automation/):
- 7 DB-Modelle: AgentDefinition, AgentVersion, AutomationDefinition,
AutomationVersion, AutomationCronJob, AgentRun, AutomationRun
- Migration 0001_initial.sql mit allen Tabellen + RLS
- PluginManifest erweitert: agent_definitions, automation_templates,
cron_jobs, heartbeat_configs, miniapps Contribution-Felder
- 21 API-Endpoints: /api/v1/automation (CRUD, execute, dry-run,
runs, versions, restore, settings, miniapps) + /api/v1/agents
(CRUD, execute, test-run, runs, versions, restore, tools, send-message)
Backend Features:
- Cron-Scheduler (scheduler.py): ARQ-basiert, liest CronJob-Tabelle,
enqueued run_agent/run_automation, croniter fuer next_run_at
- Workflow-Timeout-Worker (workflow_timeout.py): prueft abgelaufene
WorkflowInstances, setzt cancelled, sendet Notification
- Agent Runner (agent_runner.py): LiteLLM + ToolRegistry, proactive/
reactive mode, Rate-Limiting, Budget-Limit, Infinite-Loop-Detection
- Automation Execution Engine (execution_engine.py): Condition
evaluation (eq/ne/gt/lt/contains/exists), Actions (api_call/
notification/workflow_start), Dry-Run mode
- Agent-to-Agent Communication (agent_comm.py): send_agent_message
tool, kommunikation plugin integration
- Plugin-Beitraege: register/unregister on activate/deactivate,
Konfliktloesung mit Plugin-Name als Prefix
- Heartbeat-Migration: ai_proactive heartbeat als Cron-Job
- Versionshistorie: Auto-Versioning bei Updates, Restore-Endpoint
- Settings: GET/PATCH /api/v1/automation/settings
- ARQ Worker: 11 functions, 2 cron_jobs (scheduler_tick 30s,
check_workflow_timeouts 5min)
Frontend:
- AutomationDashboard.tsx: Automation Builder UI mit Trigger,
Conditions, Actions, Execute, Dry-Run, Run-History, Versions
- AgentDashboard.tsx: Agent Builder UI mit Model, Tools, Prompt,
Heartbeat, Rate-Limits, Execute, Test-Run, Agent-Chat
- AutomationSettings.tsx: Settings + MiniApp-Builder
- automation.ts: 24 React Query Hooks
- automation.ts types: TypeScript Interfaces
- routes/index.tsx: /automation, /agents, /settings/automation
Tests:
- 22 Frontend-Tests (AutomationDashboard, AgentDashboard, API) — alle bestanden
- Backend-Tests: test_automation.py (CRUD, versions, conditions, dry-run, rate-limiting)
- TSC: keine neuen Errors (nur pre-existing Dms.tsx)
- croniter dependency installiert
2026-07-23 20:00:37 +02:00
879106c4eb
Phase 1C: Frontend unified contact UI
2026-07-23 17:17:32 +02:00
d80feb2a3a
fix: register search providers in ARQ worker on_startup
2026-07-19 19:55:14 +02:00
f1be1d280b
fix: ARQ worker use direct function references instead of dotted paths
2026-07-19 19:46:05 +02:00
7970495918
fix: ARQ worker functions must be dotted module paths not objects
2026-07-19 19:39:41 +02:00
f1f3ce26b3
feat: add ARQ background worker to container
...
- Create app/core/worker.py with WorkerSettings for ARQ
- Register all job functions: reindex, index_*, embedding_batch, deep_analysis
- Update prestart.sh to start ARQ worker in background before uvicorn
- Worker runs with max_jobs=10, job_timeout=300s
2026-07-19 19:36:48 +02:00
Agent Zero