feat(B.13-B.17+B.16): Error-Handling-Infra, Observability, Graceful Shutdown, Cost-Cap, API Versioning

B.13 Error-Handling-Infrastruktur:
- ErrorCategory Enum (TRANSIENT/PERMANENT/PARTIAL), ApiError erweitert
- Einheitliches Error-Response-Format: {code, detail, field, trace_id, retryable, category}
- 3 FastAPI Exception-Handler (ApiError, HTTPException, unhandled)
- classify_exception() Helper, 6 neue Error-Codes
- 28 Tests in test_error_handling.py

B.14 Observability & trace_id-Korrelation:
- trace_id pro Request (UUID4 short) in structlog contextvars
- X-Trace-Id Response-Header
- Sensitive Fields structlog processor
- llm_complete()/llm_embed() akzeptieren trace_id kwarg
- 12 Tests in test_observability.py

B.15 Graceful Shutdown & Connection Draining:
- _shutdown_event + _inflight_requests Tracking in main.py
- drain_all_connections() in ws_helpers.py
- Worker on_shutdown pausiert WorkflowInstances (status=paused)
- 8 Tests in test_graceful_shutdown.py

B.16 API Versioning Strategie:
- Plugin-Dev-Guide Kapitel 30: URL-basiertes Versioning, Breaking Change Prozess

B.17 Cost Overrun Protection:
- llm_monthly_budget_usd + llm_hard_cutoff Settings
- _check_tenant_budget() vor jedem LLM-Call
- _track_tenant_cost() in Redis (INCRBYFLOAT)
- _check_cost_alerts() bei 50%/80%/100% -> post_system_message()
- 20 Tests in test_cost_protection.py

Total: 68 neue Tests, alle grün. Keine Regressionen.
This commit is contained in:
Agent Zero
2026-08-13 21:33:14 +02:00
parent 1baa9481a2
commit b5546ea7bd
13 changed files with 1461 additions and 31 deletions
+28 -1
View File
@@ -176,8 +176,35 @@ async def on_startup(ctx: dict[str, Any]) -> None:
async def on_shutdown(ctx: dict[str, Any]) -> None:
"""Called when worker shuts down."""
"""Called when worker shuts down.
Pauses any running WorkflowRun instances so they can be resumed after
restart, then closes Redis.
"""
logger.info("ARQ worker shutting down...")
# Pause running workflow instances so they can be resumed after restart
try:
from app.core.db import get_worker_session_factory
from sqlalchemy import select as sa_select
from app.models.workflow import WorkflowInstance
session_factory = get_worker_session_factory()
async with session_factory() as db:
result = await db.execute(
sa_select(WorkflowInstance).where(
WorkflowInstance.status == "running"
)
)
running = result.scalars().all()
if running:
for wf in running:
wf.status = "paused"
await db.commit()
logger.info(f"Paused {len(running)} running workflow(s) for graceful shutdown")
except Exception as exc:
logger.warning(f"Failed to pause running workflows during shutdown: {exc}")
from app.core.auth import close_redis
await close_redis()