feat(B.13-B.17+B.16): Error-Handling-Infra, Observability, Graceful Shutdown, Cost-Cap, API Versioning
B.13 Error-Handling-Infrastruktur:
- ErrorCategory Enum (TRANSIENT/PERMANENT/PARTIAL), ApiError erweitert
- Einheitliches Error-Response-Format: {code, detail, field, trace_id, retryable, category}
- 3 FastAPI Exception-Handler (ApiError, HTTPException, unhandled)
- classify_exception() Helper, 6 neue Error-Codes
- 28 Tests in test_error_handling.py
B.14 Observability & trace_id-Korrelation:
- trace_id pro Request (UUID4 short) in structlog contextvars
- X-Trace-Id Response-Header
- Sensitive Fields structlog processor
- llm_complete()/llm_embed() akzeptieren trace_id kwarg
- 12 Tests in test_observability.py
B.15 Graceful Shutdown & Connection Draining:
- _shutdown_event + _inflight_requests Tracking in main.py
- drain_all_connections() in ws_helpers.py
- Worker on_shutdown pausiert WorkflowInstances (status=paused)
- 8 Tests in test_graceful_shutdown.py
B.16 API Versioning Strategie:
- Plugin-Dev-Guide Kapitel 30: URL-basiertes Versioning, Breaking Change Prozess
B.17 Cost Overrun Protection:
- llm_monthly_budget_usd + llm_hard_cutoff Settings
- _check_tenant_budget() vor jedem LLM-Call
- _track_tenant_cost() in Redis (INCRBYFLOAT)
- _check_cost_alerts() bei 50%/80%/100% -> post_system_message()
- 20 Tests in test_cost_protection.py
Total: 68 neue Tests, alle grün. Keine Regressionen.
This commit is contained in:
+28
-1
@@ -176,8 +176,35 @@ async def on_startup(ctx: dict[str, Any]) -> None:
|
||||
|
||||
|
||||
async def on_shutdown(ctx: dict[str, Any]) -> None:
|
||||
"""Called when worker shuts down."""
|
||||
"""Called when worker shuts down.
|
||||
|
||||
Pauses any running WorkflowRun instances so they can be resumed after
|
||||
restart, then closes Redis.
|
||||
"""
|
||||
logger.info("ARQ worker shutting down...")
|
||||
|
||||
# Pause running workflow instances so they can be resumed after restart
|
||||
try:
|
||||
from app.core.db import get_worker_session_factory
|
||||
from sqlalchemy import select as sa_select
|
||||
from app.models.workflow import WorkflowInstance
|
||||
|
||||
session_factory = get_worker_session_factory()
|
||||
async with session_factory() as db:
|
||||
result = await db.execute(
|
||||
sa_select(WorkflowInstance).where(
|
||||
WorkflowInstance.status == "running"
|
||||
)
|
||||
)
|
||||
running = result.scalars().all()
|
||||
if running:
|
||||
for wf in running:
|
||||
wf.status = "paused"
|
||||
await db.commit()
|
||||
logger.info(f"Paused {len(running)} running workflow(s) for graceful shutdown")
|
||||
except Exception as exc:
|
||||
logger.warning(f"Failed to pause running workflows during shutdown: {exc}")
|
||||
|
||||
from app.core.auth import close_redis
|
||||
await close_redis()
|
||||
|
||||
|
||||
Reference in New Issue
Block a user