diff --git a/.a0/worklog.md b/.a0/worklog.md index 34c6efa..9c1a8cc 100644 --- a/.a0/worklog.md +++ b/.a0/worklog.md @@ -1,237 +1,120 @@ +# LeoCRM — Worklog -## P1-4 — Transactional Outbox — COMPLETE ✅ -**Date**: 2026-07-25 19:17 -**Tests**: 6/6 outbox tests pass -**Migration**: 0040_outbox.py (down_revision=0039_contact_normalize) +## 2026-08-04 — Security Fix Plan Phase 4: Krisensicherheit +**Commit:** a26405f +**Tests:** 30/30 resilience tests pass -### Files Created (4 new) -- alembic/versions/0040_outbox.py — event_outbox table with indexes -- app/core/outbox.py — enqueue_outbox_event() + process_outbox_batch() with retry/backoff -- app/models/outbox.py — SQLAlchemy ORM model -- tests/test_outbox.py — 6 tests (enqueue, publish, retry, max_attempts, batch_size, empty) +### Implementiert: +- app/core/resilience.py: CircuitBreaker (CLOSED/OPEN/HALF_OPEN), retry_db, redis_call_with_fallback, InMemoryRateLimiter, CircuitBreakerMiddleware +- app/core/auth.py: get_session_data() mit DB-Fallback (sessions table) bei Redis-Ausfall +- app/core/permissions.py: get_cached_permissions() mit DB-Fallback bei Redis-Ausfall +- app/core/rate_limit.py: check_rate_limit() mit InMemoryRateLimiter Fallback +- app/core/middleware.py: CSRF-Validierung nutzt get_session_data() (Redis+DB), best-effort TTL +- app/core/db/__init__.py: get_db() mit retry_db() für transiente Verbindungsfehler +- app/deps.py: refresh_session_ttl() in try/except für Redis-Ausfall +- app/main.py: CircuitBreakerMiddleware registriert (503 bei offenem DB-Circuit) +- app/config.py: Resilience-Settings (thresholds, cooldown, retries) +- tests/test_resilience.py: 30 Tests -### Files Modified (4) -- app/core/event_bus.py — added publish_with_results(); docstring note about outbox for domain events -- app/core/worker.py — process_outbox_job cron (every 5s, Redis distributed lock via _wrap_cron_with_lock) -- app/services/contact_service.py — contact.created, lead.created, contact.updated → enqueue_outbox_event -- tests/conftest.py — import EventOutbox model; add event_outbox to TRUNCATE list +### Verifikation: +- 30/30 Resilience-Tests ✅ +- Health Check: 200 ✅ +- Login: 200 ✅ +- Deploy über Coolify erfolgreich -### Verification -- py_compile: ALL OK -- alembic heads: single head 0040_outbox -- pytest tests/test_outbox.py: 6/6 PASSED -- test_contacts.py: 5 failed (pre-existing 403 RBAC issue, confirmed via git stash) +--- -## T03 — Plugin System Framework — COMPLETE ✅ -**Date**: 2026-06-29 01:20 -**Commit**: 7a5a48f (pushed to Forgejo) -**Tests**: 47/47 T03 tests pass, 103/103 full suite pass -**Coverage**: 85.92% for plugin modules (target: 85% ✅) -**Migration**: 0003_plugin_system.py applied (plugins + plugin_migrations tables) +## 2026-08-04 — Security Fix Plan Phase 5.1-5.4: Architektur-Lücken (Teil 1) +**Commit:** cfb4c5a -### Files Created (12 new) -- app/plugins/__init__.py, manifest.py, base.py, registry.py, migration_runner.py -- app/plugins/builtins/__init__.py, test_sample.py, migrations/0001_test_plugin.sql, migrations/0001_bad_migration.sql -- app/models/plugin.py, app/schemas/plugin.py, app/services/plugin_service.py, app/routes/plugins.py -- alembic/versions/0003_plugin_system.py -- tests/test_plugins.py (47 tests, 14 ACs + 33 unit tests) +### 5.1 Public Plugin Endpoints: +- PluginRouteDef.is_public Field in manifest.py +- main.py: Public Routes ohne Auth-Dependency mounten +- permissions/public_routes.py: token-basierte Share-Link Zugriff (info, verify, download) -### Files Modified (8) -- app/main.py (plugins router + registry init in lifespan) -- app/models/__init__.py, app/routes/__init__.py, app/schemas/__init__.py, app/services/__init__.py -- tests/conftest.py (plugin tables in TRUNCATE list) +### 5.2 PWA: +- VitePWA in vite.config.ts konfiguriert (autoUpdate, workbox, runtime caching) +- manifest.json mit Icons, theme-color, apple-mobile-web-app meta tags +- Build generiert sw.js + workbox (90 precache entries) -### Bugs Fixed by Subagent -1. Unterminated f-string in registry.py -2. Migration runner DB connection visibility (now uses session's own connection) -3. Route unregistration by path match (FastAPI wraps routes differently) -4. Dollar-quote SQL splitting (flush after closing $$) -5. AC11 assertion type (dict vs string for HTTPException detail) +### 5.3 Contacts Embedding: +- Vector(768) embedding Column zum Contact model hinzugefügt +- Migration 0002_embeddings.sql existiert bereits (HNSW Index) -### Verification (Orchestrator Independent) -- pytest tests/test_plugins.py -v: 47/47 PASS -- pytest tests/ -v: 103/103 PASS (zero regressions) -- Coverage: 85.92% (manifest 100%, base 88%, registry 88%, migration_runner 79%) -- Migration 0003 applied via alembic upgrade head -- No forbidden patterns found -- Pushed to Forgejo: 6bf0746..7a5a48f +### 5.4 Search Coverage: +- 5 neue Search Provider: task, contactperson, tag, conversation, user +- Total: 10 Search Provider (war 5) +- provider_registry.py aktualisiert -### Next: T07a (Frontend SPA Shell) ∥ T09 (KI-Copilot API) — parallel delegation +### Verifikation: +- py_compile: 17 Dateien OK ✅ +- tsc --noEmit: clean ✅ +- Frontend build: success (sw.js generiert) ✅ +- Health Check: 200 ✅, Login: 200 ✅ -## T09 — KI-Copilot API + Hybrid Workflow Engine Backend — COMPLETE ✅ -**Date**: 2026-06-29 02:46 -**Commit**: 14bd4e3 (pushed to Forgejo) -**Tests**: 238/238 full suite pass (30 AC + 105 coverage + 103 existing) -**Coverage**: 84.12% for T09 modules (target: 80% ✅) -**Migration**: 0004_ai_workflows.py applied (5 tables with RLS) +--- -### Files Created (24 new) -- app/models/ai_conversation.py, app/models/workflow.py -- app/schemas/ai_copilot.py, app/schemas/workflow.py -- app/ai/__init__.py, app/ai/llm_client.py, app/ai/action_mapper.py -- app/services/ai_copilot_service.py (~500 lines), app/services/workflow_service.py (~675 lines) -- app/routes/ai_copilot.py, app/routes/workflows.py -- app/workflows/__init__.py, app/workflows/engine.py -- app/workflows/code/__init__.py, app/workflows/code/onboarding.py -- alembic/versions/0004_ai_workflows.py -- tests/test_ai_copilot.py (67 tests), tests/test_workflows.py (68 tests) -- test_report.md +## 2026-08-04 — Security Fix Plan Phase 5.5-5.9: Architektur-Lücken (Teil 2) +**Commit:** 000c969 +**Files:** 36 files changed, +3118 lines -### Files Modified (7) -- app/models/__init__.py, app/routes/__init__.py, app/schemas/__init__.py, app/services/__init__.py -- app/main.py (added ai_copilot + workflows routers) -- tests/conftest.py (added new tables to TRUNCATE + model imports) -- app/core/event_bus.py (added workflow event handler registration) +### 5.5 Plugin-Marketplace: +- Neues Plugin: marketplace/ (models, routes, services, schemas, config) +- MarketplaceListing model (global, keine tenant_id) +- Ed25519 Signatur-Verifikation via PluginSignature +- Endpoints: list, detail, install, verify, categories -### Bugs Fixed -1. MissingGreenlet on async lazy-load of updated_at/created_at — fixed with _safe_iso() and _get_attr() helpers -2. _message_to_dict in ai_copilot_service.py — patched by orchestrator (m.created_at.isoformat() → _safe_iso(_get_attr(m, "created_at"))) +### 5.6 Agent Memory (persistent): +- Neues Plugin: agent_memory/ (models, routes, services, schemas) +- AgentMemory model mit embedding vector(768) + HNSW index +- store_memory() mit auto-embedding +- retrieve_relevant_memories() mit pgvector cosine similarity -### Coverage Breakdown -- app/workflows/engine.py: 0% → 90.00% -- app/services/ai_copilot_service.py: 38.89% → 98.61% -- app/ai/action_mapper.py: 43.44% → 96.72% -- app/ai/llm_client.py: 64.62% → 81.54% -- app/services/workflow_service.py: 62.54% → 75.95% -- app/routes/workflows.py: 59.48% → 62.93% -- app/routes/ai_copilot.py: 65% → 65.00% -- **Overall: 45.37% → 84.12%** ✅ +### 5.7 GraphRAG: +- Neues Plugin: graph_rag/ (models, routes, services, provider, schemas) +- EntityRelationship model (source/target type+id, relationship_type, metadata) +- BFS Graph-Traversal (bidirektional, konfigurierbare Tiefe) +- GraphRAGSearchProvider im unified_search registriert -### Verification (Orchestrator Independent) -- pytest tests/: 238/238 PASS (zero regressions) -- Migration 0004 applied via alembic upgrade head -- RLS policies on all 5 new tables (ai_conversations, ai_messages, workflows, workflow_instances, workflow_step_history) -- No forbidden patterns (.test TLD, SET LOCAL, raise HTTPException in middleware, POST without status_code) -- POST action endpoints (query/execute/advance/cancel) correctly use 200 default -- POST creation endpoints (workflows, instances) correctly use 201 -- Pushed to Forgejo: 7a5a48f..14bd4e3 +### 5.8 Subagents / Multi-Agent: +- AgentCoordinator Klasse (create_subtask, wait_for_subtask, aggregate, cancel) +- AgentSubtask model + migration 0002_agent_subtasks.sql +- 6 neue API Endpoints für Subtask-Management +- Tools in AI tool registry registriert -### Next: T07a (Frontend SPA Shell — React 18) +### 5.9 External Agent API: +- external_api.py: POST /run, GET /status, POST /stream (SSE) +- Bearer API Token Authentifizierung +- Rate Limiting: 10 req/min per token -## 2026-06-29 08:03 — T07a Complete -- **Task**: T07a — Frontend Core SPA (Shell, Auth, Routing, i18n, UI Library, Accessibility) -- **Commit**: 22976ab (pushed to Forgejo) -- **Tests**: 111/111 passing (20 test files) -- **tsc**: 0 errors -- **Build**: Success (471KB JS, 24KB CSS gzipped) -- **Files**: 66 files, 8598 insertions -- **Fixes applied by orchestrator**: - - Login form aria-label for role=form accessibility - - Avatar img alt="" to prevent duplicate role=img - - Avatar test null-safety with non-null assertion - - index.css border-border → border-secondary-200 (Tailwind class missing) - - .gitignore created to exclude node_modules/dist - - Remote URL fixed from agent-zero to Forgejo leocrm repo -- **Subagent**: implementation_engineer (hit context cap at ~90%, orchestrator completed remaining fixes) +### Verifikation: +- py_compile: alle neuen Dateien OK ✅ +- Health Check: 200 ✅, Login: 200 ✅ +- 3 neue Plugins in main.py registriert -## 2026-06-29 11:05 — T07b Complete -- **Task**: T07b — Frontend Feature Pages -- **Commit**: 700b7a7 (47 files, +4088 lines) -- **Pushed**: Forgejo remote, HEAD=700b7a7 -- **Verification**: 141 tests pass, build success, tsc clean -- **Deliverables**: 11 feature pages, 3 page updates, 13 routes, 12 test files, i18n updates, 7 shared components, 16 API hooks -- **Subagents used**: 3 (implementation_engineer x2, a0-orchestrator-git x1) +--- -## 2026-06-29 20:50 — T04 Complete -- **Task**: T04 — DMS Plugin Backend (Folders, Files, Preview, OnlyOffice, Share Links) -- **Commit**: fdb41da (14 files, +3760 lines) -- **Pushed**: Forgejo remote, HEAD=fdb41da -- **Verification**: 106 DMS tests pass (27 AC + 38 error + 41 coverage), 97.90% coverage, 412 total tests pass (full regression), 0 ruff errors -- **Deliverables**: DMS plugin dir (6 files), 3 test files, conftest fixture sharing, pyproject.toml coverage config fix (concurrency=greenlet) -- **Subagents used**: 2 (implementation_engineer x2 — initial + coverage improvement) -- **Key finding**: coverage.py needed `concurrency = ["greenlet"]` for Python 3.13 async tracking +## 2026-08-04 — Cleanup +**Commit:** 157e454, aaf2784 -## 2026-06-29 14:05 — T11 Complete -- **Task**: T11 — Tags Plugin + Permissions Plugin + Entity Links Backend -- **Commit**: 5d18507 (26 files, +2863 lines) -- **Pushed**: Forgejo remote, HEAD=5d18507 -- **Verification**: 68 tests pass, coverage 66.61% (dead code gaps explained) -- **Deliverables**: 3 plugin dirs (tags, permissions, entity_links), 3 test files, migration_runner fix, builtins registration, conftest updates -- **Subagents used**: 3 (implementation_engineer x3 — initial, fixes, coverage improvement) +### Gelöscht: +- 34 alte Plan-Dateien (SANIERUNGS_FORTSCHRITT.md, UMBAU_PLAN.md, FIX-PLAN.md, etc.) +- dump.rdb, templates/ (Jinja2 HTML), frontend/test_report.md +- .a0/current_status.md und next_steps.md aktualisiert -## 2026-06-30 01:15 — T05 Complete -- **Task**: T05 — Calendar Plugin Backend (Appointments, Tasks, Kanban, ICS, Resources, Recurrence) -- **Commit**: 7fbeeda (14 files, +3674 lines) -- **Pushed**: Forgejo remote, HEAD=7fbeeda -- **Verification**: 69 calendar tests pass (33 AC + 36 recurrence unit), 86.87% coverage, 481 total tests pass (full regression), 0 ruff errors -- **Deliverables**: Calendar plugin dir (8 files: __init__.py, plugin.py, routes.py, models.py, schemas.py, recurrence.py, ics_utils.py, migrations/0001_initial.sql), 2 test files (test_calendar.py 1075 lines, test_recurrence_unit.py), conftest.py calendar fixtures, builtins/__init__.py registration -- **Subagents used**: 2 (implementation_engineer x2 — initial implementation + 8 bug fixes) -- **Key fixes**: MissingGreenlet (db.refresh after flush), CSV export route ordering, ICS token commit, recurrence midnight boundary, datetime.UTC deprecation +--- -## 2026-06-30 13:50 — T06: Test Fixes Complete -- **11 test failures resolved** across all test suites -- Input.tsx: added required={required} native attribute -- Card.tsx: added ...rest spread for data-testid forwarding -- CompanyForm.tsx + ContactForm.tsx: added noValidate to bypass native HTML5 validation in tests -- Test files fixed: CompaniesList, CompanyDetail, CompanyForm, ContactsList, SettingsRoles -- ARIA spec: aria-sort value corrected to 'ascending' -- **Results:** 112/112 tests pass, tsc clean, vite build successful -- **Commit:** e28d11f +## Ältere Einträge (archiviert) -## 2026-07-01 15:41 — T06: Mail Plugin Backend Complete -- **Mail Plugin implementiert:** 8 neue Dateien, 4667 Zeilen -- **14 Models:** mail_accounts, mail_folders, mails, attachments, labels, rules, templates, signatures, vacation_sent_log, seen_by, delegates, send_permissions, pgp_keys, contact_pgp_keys -- **Features:** IMAP sync, SMTP send/reply/forward, threading, templates, rules, vacation (dedup), PGP, shared mailboxes, delegates, send permissions, HTML sanitization, FTS search, contact linking, calendar event creation -- **Tests:** 46/46 pass, 74.56% coverage -- **Regression:** 527/527 pass (0 failures) -- **Ruff:** 0 errors, format clean -- **Commit:** f646c59 -- **Risks:** Coverage 74.56% (target 80%), ILIKE fallback instead of tsvector, ARQ worker not wired - -## 2026-07-01 16:54 — T08a: Frontend DMS + Tags + Permissions UI Complete -- **18 neue Dateien, 6 modified** — 3368 Zeilen -- **DMS:** File browser (folder tree + file grid), upload dropzone, preview modal, share dialog, bulk actions, trash view -- **Tags:** TagPicker, TagCloud, BulkTagDialog — integriert in CompanyDetail + ContactDetail -- **Permissions:** Share dialog, public share links, permission display -- **API clients:** dms.ts, tags.ts, permissions.ts -- **Routes:** /dms, /dms/trash -- **i18n:** de.json + en.json translations -- **Tests:** 33/33 new tests pass, full regression 276/276 pass -- **tsc:** 0 errors, **vite build:** 252 modules, 3.31s -- **Commit:** 0962f3a - -## 2026-07-01 20:44 — T08c: Frontend Mail UI + Global Search UI Complete -- **16 neue Dateien, 5 modified** — 4313 Zeilen -- **Mail UI:** 3-pane layout (folder tree + mail list + reading pane), compose modal (bold/italic/link/template), reply/forward, shared mailbox selector, attachment download, create-event-from-mail -- **Mail Settings:** 6 tabs (accounts, signatures, rules, labels, vacation, PGP) -- **Global Search:** Tabs for companies/contacts/mails/files/events -- **API client:** mail.ts (all endpoints) -- **Routes:** /mail, /mail/settings -- **i18n:** de.json + en.json translations -- **Tests:** 44/44 new tests pass, full regression 318/318 pass -- **tsc:** 0 errors, **vite build:** 267 modules, 5.19s -- **Commit:** 0070fb3 - -## 2026-07-01 23:01 — T10: Monitoring, Performance, Documentation & Environment Config Complete -- **Monitoring:** Prometheus metrics (http_requests_total, db_pool_connections, arq_jobs_total), structured JSON logging via structlog, extended health checks (DB, Redis, storage, worker) -- **Metrics endpoint:** GET /api/v1/metrics (admin-only, text/plain Prometheus format, 403 for non-admin) -- **Health endpoint:** Extended with database, redis, storage, worker checks — status healthy/degraded -- **Performance:** Streaming CSV export for contacts and companies (StreamingResponse with own DB session), page_size max 100 enforced (422 for >100) -- **Scripts:** seed_perf_data.py (--count N), check_indexes.py -- **Documentation:** README.md updated (prod setup, API section, admin-guide link, env profiles), docs/admin-guide.md created, docs/api-overview.md created -- **Config:** .env.example updated with SMTP, storage, secret_key vars; config.py extended with SMTP/storage/secret_key settings -- **Dependencies:** prometheus-client, structlog added to requirements.txt -- **Tests:** 38/38 pass (test_monitoring.py 17, test_performance.py 15, test_health.py 6) in 24.24s -- **Ruff:** All checks passed -- **Docs check:** README.md, docs/admin-guide.md, docs/api-overview.md all present - -## 2026-07-01 23:15 — T10: Monitoring, Performance, Documentation Complete -- **8 new files, 8 modified** — 2250 lines -- **Monitoring:** Extended health (DB+Redis+Storage+Worker), Prometheus metrics (admin-only), structured JSON logging (structlog) -- **Performance:** page_size max 100 enforced, streaming CSV export, seed_perf_data.py script -- **Docs:** admin-guide.md, api-overview.md, README updated, .env.example updated -- **Tests:** 38 new tests pass, full regression 564/564 pass -- **Ruff:** all checks passed -- **Commit:** 69e91fd - -## 🎉 PHASE 3 COMPLETE — ALL 14 TASKS DONE - -## 2026-07-25 19:07 — P2-1: Unified Contact Model normalisieren — COMPLETE -- **6 files changed** (5 modified + 1 new migration) -- **Migration 0039_contact_normalize.py**: surfix→suffix rename, Float→Numeric(5,2) for 6 discount columns with CHECK constraints (0-100), JSON→JSONB for contacts.custom and contactpersons.custom, partial unique indexes on (tenant_id, code) and (tenant_id, accounting_code) -- **Model**: surfix→suffix, Float→Numeric(5,2), JSON→JSONB, UniqueConstraint added, Decimal import -- **Schema**: surfix→suffix (3x), float→Decimal (18x), Decimal import -- **Services**: contact_service.py (3x surfix→suffix), dedup_service.py (1x surfix→suffix) -- **Frontend**: unifiedContacts.ts surfix→suffix in UnifiedContact interface -- **Checks**: py_compile OK, alembic heads → 0039_contact_normalize (single head), comprehensive grep confirms zero surfix in source code -- **Tests**: 1 passed, 5 failed (pre-existing 403/404 errors unrelated to P2-1) +### 2026-07-25 — P1-4: Transactional Outbox — COMPLETE +### 2026-06-29 — T03: Plugin System Framework — COMPLETE +### 2026-06-29 — T09: KI-Copilot API + Workflow Engine — COMPLETE +### 2026-06-29 — T07a: Frontend Core SPA — COMPLETE +### 2026-06-29 — T07b: Frontend Feature Pages — COMPLETE +### 2026-06-29 — T04: DMS Plugin Backend — COMPLETE +### 2026-06-29 — T11: Tags + Permissions + Entity Links — COMPLETE +### 2026-06-30 — T05: Calendar Plugin Backend — COMPLETE +### 2026-06-30 — T06: Test Fixes — COMPLETE +### 2026-07-01 — T06: Mail Plugin Backend — COMPLETE +### 2026-07-01 — T08a: Frontend DMS + Tags + Permissions UI — COMPLETE +### 2026-07-01 — T08c: Frontend Mail + Global Search UI — COMPLETE