"""FastAPI application - LeoCRM backend.""" from __future__ import annotations import asyncio import importlib import logging import os import time import traceback import uuid as _uuid from contextlib import asynccontextmanager import structlog from fastapi import Depends, FastAPI, HTTPException, Request from fastapi.middleware.cors import CORSMiddleware from fastapi.responses import FileResponse, JSONResponse from fastapi.staticfiles import StaticFiles from starlette.middleware.base import BaseHTTPMiddleware logger = logging.getLogger(__name__) from app.config import get_settings # noqa: E402 from app.core.db import close_engine, get_engine # noqa: E402 from app.core.error_codes import ApiError, build_error_response # noqa: E402 from app.core.middleware import CSRFMiddleware, SecurityHeadersMiddleware # noqa: E402 from app.core.monitoring import record_error, record_request # noqa: E402 from app.core.rate_limit import GeneralRateLimitMiddleware # noqa: E402 from app.core.resilience import CircuitBreakerMiddleware # noqa: E402 from app.core.service_container import get_container # noqa: E402 from app.plugins.registry import get_registry # noqa: E402 from app.routes import ( # noqa: E402 addresses, ai_copilot, api_tokens, approvals, attachments, audit, auth, backups, bank_accounts, contact_folder_permissions, contact_folders, contacts, currencies, custom_field_definitions, custom_fields, dashboard, entity_history, entity_permissions, errors, groups, guests, health, import_export, metrics, notifications, outbox, owner_transfer, permission_templates, plugins, # delegations, # ⏸ Parked — not integrated into resolve_permissions() policies, roles, saved_filters, saved_views, sequences, system_settings, taxes, tenants, user_preferences, users, webhooks, workflows, workspaces, ) # ── Graceful shutdown signal ───────────────────────────────────────────────── # Set during lifespan shutdown so middleware and handlers can stop accepting work. _shutdown_event = asyncio.Event() # Track in-flight requests for graceful draining _inflight_requests: set[asyncio.Task] = set() async def _drain_inflight(timeout_per_request: float = 25.0) -> None: """Wait for all in-flight request tasks to complete.""" if not _inflight_requests: return logger.info(f"Waiting for {len(_inflight_requests)} in-flight requests to complete") # Give tasks a chance to finish; cancel remaining after timeout done, pending = await asyncio.wait( _inflight_requests, timeout=timeout_per_request, ) if pending: logger.warning(f"Cancelling {len(pending)} in-flight requests that exceeded grace period") for task in pending: task.cancel() await asyncio.gather(*pending, return_exceptions=True) def _generate_trace_id() -> str: """Generate a short trace ID (first 8 chars of UUID4).""" return _uuid.uuid4().hex[:8] def _get_trace_id() -> str | None: """Get the current trace_id from structlog contextvars (best-effort).""" try: ctx = structlog.contextvars.get_contextvars() return ctx.get("trace_id") except Exception: return None class RequestLoggingMiddleware(BaseHTTPMiddleware): """Structured logging + Prometheus metrics + trace_id for every HTTP request.""" async def dispatch(self, request: Request, call_next): start_time = time.perf_counter() method = request.method path = request.url.path # Generate trace_id and bind to structlog contextvars for this request trace_id = _generate_trace_id() structlog.contextvars.clear_contextvars() structlog.contextvars.bind_contextvars(trace_id=trace_id) # Extract tenant_id from session cookie if available (best-effort) tenant_id = None try: response = await call_next(request) except Exception as exc: duration_ms = (time.perf_counter() - start_time) * 1000 tb_str = traceback.format_exc() record_error( event="unhandled_exception", method=method, path=path, status_code=500, error=str(exc), traceback_str=tb_str, tenant_id=tenant_id, ) # Report to Forgejo error reporter try: from app.plugins.builtins.contracts import get_contract reporter_contract = get_contract("forgejo_error_reporter") if reporter_contract is not None: await reporter_contract.report_error_to_forgejo({ "message": f"[Backend] {method} {path}: {exc}", "stack": tb_str, "url": str(request.url), "context": {"method": method, "path": path, "source": "backend_middleware", "trace_id": trace_id}, }) except Exception: pass # Never let error reporting break the request raise duration_ms = (time.perf_counter() - start_time) * 1000 status_code = response.status_code # Add trace_id to response header response.headers["X-Trace-Id"] = trace_id # Report 4xx and 5xx errors to Forgejo (except 401/403 which are expected) if status_code >= 400 and status_code not in (401, 403): try: from app.plugins.builtins.contracts import get_contract reporter_contract = get_contract("forgejo_error_reporter") if reporter_contract is not None: await reporter_contract.report_error_to_forgejo({ "message": f"[Backend] {method} {path} → {status_code}", "url": str(request.url), "context": {"method": method, "path": path, "status": status_code, "source": "backend_response", "trace_id": trace_id}, }) except Exception: pass # Never let error reporting break the response # Try to get tenant_id from response headers or request state # (set by auth middleware/dependency — best-effort, never log credentials) record_request( method=method, path=path, status_code=status_code, duration_ms=duration_ms, tenant_id=tenant_id, ) # Clear contextvars after request completes structlog.contextvars.clear_contextvars() return response @asynccontextmanager async def lifespan(app: FastAPI): """Application lifespan: startup and shutdown.""" # Initialize global Redis client (singleton) from app.core.auth import close_redis, init_redis from app.core.jobs import close_job_pool, init_job_pool await init_redis() await init_job_pool() # Initialize service container container = get_container() await container.initialize() # Initialize plugin registry and discover built-in plugins registry = get_registry() from app.core.db import get_migration_engine registry.initialize(get_migration_engine(), app) registry.discover_builtins() # Install discovered builtin plugins and activate only those marked active in DB from sqlalchemy import select as sa_select from sqlalchemy.ext.asyncio import async_sessionmaker from app.core.event_bus import get_event_bus from app.models.plugin import Plugin as PluginModel event_bus = get_event_bus() async_session = async_sessionmaker(get_engine(), expire_on_commit=False) # Load all tenant IDs for per-tenant plugin activation (RLS fail-closed requires tenant context) from app.core.db import set_tenant_context from app.models.tenant import Tenant as TenantModel async with async_session() as db: tenant_result = await db.execute(sa_select(TenantModel.id)) all_tenant_ids = [row[0] for row in tenant_result] logger.info(f"Loaded {len(all_tenant_ids)} tenants for plugin activation") # Install plugin records and run migrations (global, no tenant context needed) async with async_session() as db: for name in registry.resolve_load_order(): plugin = registry.get_plugin(name) if plugin is None: continue # Check if plugin already in DB result = await db.execute( sa_select(PluginModel).where(PluginModel.name == name) ) plugin_record = result.scalar_one_or_none() if plugin_record is None: plugin_record = PluginModel( name=name, display_name=plugin.manifest.display_name, version=plugin.manifest.version, status="installed", active=plugin.manifest.is_core, is_core=plugin.manifest.is_core, ) db.add(plugin_record) await db.flush() logger.info(f"Created plugin record: {name} (core={plugin.manifest.is_core})") # Run migrations if not yet applied — use MIGRATION engine (crm_migration) for DDL if plugin.manifest.migrations: try: from app.core.db import get_migration_session_factory mig_session_factory = get_migration_session_factory() async with mig_session_factory() as mig_db: await registry.migration_runner.run_all_migrations( mig_db, name, plugin.manifest.migrations ) await mig_db.commit() except Exception as exc: logger.error(f"Migration FAILED for {name}: {exc}") if plugin_record.active: logger.error(f"Deactivating plugin {name} due to migration failure") plugin_record.active = False plugin_record.status = "migration_failed" continue # Only activate plugins that are marked active in DB if not plugin_record.active: logger.info(f"Plugin {name} is inactive — skipping activation") continue # Activate plugin with a FRESH session per plugin to avoid RLS state leakage # RLS fail-closed requires app.current_tenant_id for tenant-table writes. # Plugin activation may fail on duplicate cron job inserts — this is harmless # since cron jobs already exist from previous startups. plugin_activated = False for tenant_id in all_tenant_ids: try: async with async_session() as plugin_db: await set_tenant_context(plugin_db, tenant_id) await plugin.on_activate(plugin_db, container, event_bus) await plugin_db.flush() await plugin_db.commit() plugin_activated = True except Exception as exc: logger.warning(f"[STARTUP] Plugin {name} activation issue for tenant {tenant_id}: {exc}") break if plugin_activated: plugin_record.status = "active" logger.info(f"[STARTUP] Activated plugin: {name}") try: await db.commit() except Exception as exc: logger.warning(f"[STARTUP] Commit failed after plugin activation: {exc}") await db.rollback() # Initialize permission registry with active plugin names from app.core.permission_registry import init_permission_registry, register_plugin_permissions active_plugin_names: set[str] = set() async with async_session() as db: result = await db.execute( sa_select(PluginModel).where(PluginModel.active == True) # noqa: E712 ) for record in result.scalars().all(): active_plugin_names.add(record.name) plugin = registry.get_plugin(record.name) if plugin and plugin.manifest.permissions: register_plugin_permissions(record.name, plugin.manifest.permissions) init_permission_registry(active_plugin_names) logger.info("Permission registry initialized with %d active plugins", len(active_plugin_names)) # Register webhook dispatcher on the event bus from app.core.webhook_dispatcher import register_webhook_event_handlers register_webhook_event_handlers(event_bus) logger.info("Webhook event handlers registered") # Register trigger dispatcher — generic event→automation bridge from app.core.trigger_dispatcher import register_trigger_dispatcher register_trigger_dispatcher(event_bus) logger.info("Trigger dispatcher registered") # Entity restore + history hooks are registered by plugins in on_activate(), # including Contacts (via ContactsPlugin). No Core special case here. # Register entity models from active plugins (P0-3 fix) from app.services.entity_permission_service import register_entity_model for name in active_plugin_names: plugin = registry.get_plugin(name) if plugin: for entity_type, model_class in plugin.get_entity_models().items(): register_entity_model(entity_type, model_class) logger.info("Entity models registered for %d active plugins", len(active_plugin_names)) # Register field definitions from active plugins only from app.core.permission_registry import get_permission_registry for name in active_plugin_names: plugin = registry.get_plugin(name) if plugin: field_defs = plugin.get_field_definitions() if field_defs: get_permission_registry().register_field_definitions(name, field_defs) logger.info("Field definitions registered for %d active plugins", len(active_plugin_names)) # Seed default data (EUR currency, 19%/7% tax rates) for all tenants # ⚠️ Use migration engine (crm_migration, BYPASSRLS) — RLS on currencies/taxes # blocks inserts from crm_api role without tenant context. from app.core.db import get_migration_session_factory from app.core.seeds import seed_default_data mig_session_factory = get_migration_session_factory() async with mig_session_factory() as db: try: await seed_default_data(db) await db.commit() logger.info("Default data seeding completed") except Exception as exc: logger.warning(f"Default data seeding failed: {exc}") await db.rollback() yield # ── Graceful shutdown ──────────────────────────────────────────────────── # Signal that we're shutting down — no new requests should be accepted. _shutdown_event.set() logger.info("Graceful shutdown initiated — draining in-flight requests") # Drain WebSocket connections (notify clients to reconnect) try: from app.core.ws_helpers import drain_all_connections await drain_all_connections(grace_period_seconds=5) except Exception as exc: logger.warning(f"WS drain failed during shutdown: {exc}") # Give in-flight requests time to complete (max 30s) try: await asyncio.wait_for(_drain_inflight(), timeout=30.0) except TimeoutError: logger.warning("Graceful shutdown: 30s timeout reached, forcing shutdown") # Close global Redis and ARQ pool await close_job_pool() await close_redis() await close_engine() logger.info("Graceful shutdown complete") def create_app() -> FastAPI: """Create and configure the FastAPI application.""" settings = get_settings() app = FastAPI( title="LeoCRM", version="1.0.0", lifespan=lifespan, description=( "LeoCRM — Self-hosted CRM system for small sales teams.\n\n" "## Authentication\n" "All endpoints (except `/api/v1/auth/login` and `/api/v1/health`) require " "a valid session cookie. Obtain a session by calling `POST /api/v1/auth/login`.\n\n" "## Multi-Tenancy\n" "All data is tenant-scoped. The tenant context is set automatically from the " "authenticated session.\n\n" "## Plugins\n" "LeoCRM uses a plugin architecture. Core routes are always available. " "Plugin routes (DMS, Mail, Calendar, Automation, etc.) are registered when " "the corresponding plugin is active." ), openapi_tags=[ {"name": "health", "description": "Health check endpoints — no auth required."}, {"name": "metrics", "description": "Prometheus metrics endpoint."}, {"name": "auth", "description": "Authentication: login, logout, session, password reset."}, {"name": "users", "description": "User management: CRUD, user-tenant assignments."}, {"name": "roles", "description": "Role management and permission definitions."}, {"name": "groups", "description": "User groups for contact assignment and filtering."}, {"name": "tenants", "description": "Tenant management and tenant switching."}, {"name": "notifications", "description": "User notifications and notification preferences."}, {"name": "contacts", "description": "Contact CRUD, contact persons, FTS search, export, soft-delete."}, {"name": "contact-folders", "description": "Contact folder management for organization."}, {"name": "entity-history", "description": "Audit trail and entity change history."}, {"name": "import-export", "description": "Bulk import and export of contacts and data."}, {"name": "plugins", "description": "Plugin management: list, install, activate, deactivate."}, {"name": "ai-copilot", "description": "AI copilot: chat, suggestions, conversation history."}, {"name": "workflows", "description": "Workflow definitions, instances, and execution."}, {"name": "user-preferences", "description": "Per-user preference settings."}, {"name": "currencies", "description": "Currency management for multi-currency support."}, {"name": "taxes", "description": "Tax rate management (VAT, sales tax)."}, {"name": "sequences", "description": "Number sequence management for invoices, quotes, etc."}, {"name": "system-settings", "description": "Tenant-level system settings (company info, theme)."}, {"name": "attachments", "description": "File attachments for contacts, companies, and entities."}, {"name": "addresses", "description": "Address management for contacts and companies."}, {"name": "bank_accounts", "description": "Bank account management for tenant."}, {"name": "audit", "description": "Audit log queries and compliance reporting."}, {"name": "automation", "description": "Automation engine: agents, automations, cron jobs, execution logs."}, {"name": "agents", "description": "AI agent definitions and agent runner endpoints."}, {"name": "dms", "description": "Document Management System: files, folders, sources, sharing."}, {"name": "mail", "description": "Email integration: IMAP accounts, folders, messages, send."}, {"name": "calendar", "description": "Calendar management: appointments, resources, recurrence."}, {"name": "search", "description": "Unified search across contacts, documents, emails, etc."}, {"name": "reports", "description": "Report generator: templates, rendering, scheduled reports."}, {"name": "entity-links", "description": "Entity linking: connect contacts to DMS documents and other entities."}, {"name": "kommunikation", "description": "Unified messaging: conversations, participants, messages."}, {"name": "ai-proactive", "description": "Proactive AI: insights, alerts, and recommendations."}, {"name": "ai-assistant", "description": "AI assistant: chat completions, tool calling, context awareness."}, {"name": "tags", "description": "Tag management: create, assign, search tags across entities."}, {"name": "permissions", "description": "Permission management: roles, field-level permissions, sharing."}, {"name": "public-share", "description": "Public sharing endpoints — no auth required, token-based access."}, ], ) app.add_middleware( CORSMiddleware, allow_origins=settings.cors_origin_list, allow_credentials=True, allow_methods=["GET", "POST", "PATCH", "PUT", "DELETE", "OPTIONS"], allow_headers=["Authorization", "Content-Type", "X-CSRF-Token", "X-Tenant-ID"], max_age=3600, ) app.add_middleware(CSRFMiddleware) app.add_middleware(SecurityHeadersMiddleware) app.add_middleware(GeneralRateLimitMiddleware) app.add_middleware(RequestLoggingMiddleware) app.add_middleware(CircuitBreakerMiddleware) # ── Global exception handler — catch ALL unhandled exceptions ── @app.exception_handler(Exception) async def global_exception_handler(request: Request, exc: Exception): trace_id = _get_trace_id() logger.error(f"Unhandled exception: {exc}", exc_info=True, extra={"trace_id": trace_id}) record_error( event="unhandled_exception", method=request.method, path=request.url.path, status_code=500, error=str(exc), ) body = build_error_response( code="internal_error", detail="Internal server error", trace_id=trace_id, ) resp = JSONResponse(status_code=500, content=body) if trace_id: resp.headers["X-Trace-Id"] = trace_id return resp # ── HTTPException handler — unified format ── @app.exception_handler(HTTPException) async def http_exception_handler(request: Request, exc: HTTPException): trace_id = _get_trace_id() # Map common HTTP status codes to error codes status_to_code = { 404: "not_found", 403: "forbidden", 422: "unprocessable", 429: "rate_limited", 501: "not_implemented", 502: "bad_gateway", 503: "service_unavailable", 504: "service_timeout", } code = status_to_code.get(exc.status_code, "internal_error" if exc.status_code >= 500 else "validation_error") body = build_error_response( code=code, detail=str(exc.detail) if exc.detail else None, trace_id=trace_id, ) resp = JSONResponse(status_code=exc.status_code, content=body) if trace_id: resp.headers["X-Trace-Id"] = trace_id return resp # ── ApiError handler — structured error responses with category/retryable ── @app.exception_handler(ApiError) async def api_error_handler(request: Request, exc: ApiError): trace_id = _get_trace_id() body = exc.to_response(trace_id=trace_id) resp = JSONResponse(status_code=exc.status, content=body) if trace_id: resp.headers["X-Trace-Id"] = trace_id return resp app.include_router(health.router) app.include_router(metrics.router) app.include_router(auth.router) app.include_router(users.router) app.include_router(roles.router) app.include_router(groups.router) app.include_router(tenants.router) app.include_router(notifications.router) from app.routes.companies import router as companies_router app.include_router(companies_router) app.include_router(contacts.router) app.include_router(contact_folders.router) app.include_router(contact_folder_permissions.router) app.include_router(entity_permissions.router) app.include_router(dashboard.router) app.include_router(entity_history.router) app.include_router(import_export.router) app.include_router(plugins.router) app.include_router(ai_copilot.router) app.include_router(workflows.router) app.include_router(user_preferences.router) app.include_router(currencies.router) app.include_router(taxes.router) app.include_router(sequences.router) app.include_router(system_settings.router) app.include_router(attachments.router) app.include_router(addresses.router) app.include_router(bank_accounts.router) app.include_router(audit.router) app.include_router(backups.router) app.include_router(owner_transfer.router) app.include_router(custom_field_definitions.router) app.include_router(custom_fields.router) app.include_router(saved_filters.router) app.include_router(saved_views.router) app.include_router(webhooks.router) app.include_router(permission_templates.router) # app.include_router(delegations.router) # ⏸ Parked — not integrated into resolve_permissions() app.include_router(policies.router) app.include_router(errors.router) app.include_router(guests.router) # ⚠️ Guest-System umgebaut — Guests sind jetzt reguläre User mit role=guest app.include_router(workspaces.router) app.include_router(outbox.router) app.include_router(api_tokens.router) app.include_router(approvals.router) # ── Register plugin routes for all discovered plugins ── # Routes are registered at app creation time so OpenAPI docs are complete. # Activation status is enforced per-request via require_active_plugin(). # Plugin modules are discovered dynamically via the registry — no hardcoded list. from app.deps import require_active_plugin from app.plugins.registry import get_registry _route_registry = get_registry() # Ensure builtins are discovered before registering routes. # discover_builtins() is idempotent — safe to call even if lifespan hasn't run yet. if not _route_registry.list_discovered(): _route_registry.discover_builtins() for plugin_name in _route_registry.list_discovered(): plugin = _route_registry.get_plugin(plugin_name) if plugin is None or not plugin.manifest.routes: continue try: for route_def in plugin.manifest.routes: try: router_module = importlib.import_module(route_def.module) router = getattr(router_module, route_def.router_attr) # Check if this route definition is public (no auth required) is_public = getattr(route_def, "is_public", False) if is_public: # Public routes: no auth dependency, no plugin check app.include_router(router) logger.info(f"Registered PUBLIC routes for {plugin_name}: {route_def.module}") continue plugin_dep = Depends(require_active_plugin(plugin_name)) # Use include_router with dependencies to avoid mutating # the shared module-level router object (which tests reuse) app.include_router(router, dependencies=[plugin_dep]) except Exception as exc: logger.error(f"Failed to register route {route_def.module}.{route_def.router_attr}: {exc}") except Exception as exc: logger.error(f"Failed to register plugin routes for {plugin_name}: {exc}") # ── Serve frontend static files (SPA) ────────────────────────────── # Mount built frontend assets (JS, CSS, images) frontend_dist = os.path.join(os.path.dirname(__file__), "..", "frontend", "dist") if os.path.isdir(frontend_dist): # Serve static assets at /assets assets_path = os.path.join(frontend_dist, "assets") if os.path.isdir(assets_path): app.mount("/assets", StaticFiles(directory=assets_path), name="assets") # SPA catch-all: serve index.html for all non-API routes @app.get("/{full_path:path}") async def spa_spa(full_path: str): """Serve index.html for all non-API routes (SPA fallback).""" # Don't intercept API routes if full_path.startswith(("api/", "docs", "openapi", "redoc")): raise HTTPException(status_code=404, detail="Not Found") # Block path traversal and system file access blocked_prefixes = ("var/log/", "error/", "error_log", "var/", "etc/", "proc/", "sys/") if full_path.startswith(blocked_prefixes) or ".." in full_path: raise HTTPException(status_code=404, detail="Not Found") index_path = os.path.join(frontend_dist, "index.html") if os.path.isfile(index_path): # noqa: ASYNC240 return FileResponse(index_path) raise HTTPException(status_code=404, detail="Frontend not built") return app app = create_app()