feat(B.13-B.17+B.16): Error-Handling-Infra, Observability, Graceful Shutdown, Cost-Cap, API Versioning

B.13 Error-Handling-Infrastruktur:
- ErrorCategory Enum (TRANSIENT/PERMANENT/PARTIAL), ApiError erweitert
- Einheitliches Error-Response-Format: {code, detail, field, trace_id, retryable, category}
- 3 FastAPI Exception-Handler (ApiError, HTTPException, unhandled)
- classify_exception() Helper, 6 neue Error-Codes
- 28 Tests in test_error_handling.py

B.14 Observability & trace_id-Korrelation:
- trace_id pro Request (UUID4 short) in structlog contextvars
- X-Trace-Id Response-Header
- Sensitive Fields structlog processor
- llm_complete()/llm_embed() akzeptieren trace_id kwarg
- 12 Tests in test_observability.py

B.15 Graceful Shutdown & Connection Draining:
- _shutdown_event + _inflight_requests Tracking in main.py
- drain_all_connections() in ws_helpers.py
- Worker on_shutdown pausiert WorkflowInstances (status=paused)
- 8 Tests in test_graceful_shutdown.py

B.16 API Versioning Strategie:
- Plugin-Dev-Guide Kapitel 30: URL-basiertes Versioning, Breaking Change Prozess

B.17 Cost Overrun Protection:
- llm_monthly_budget_usd + llm_hard_cutoff Settings
- _check_tenant_budget() vor jedem LLM-Call
- _track_tenant_cost() in Redis (INCRBYFLOAT)
- _check_cost_alerts() bei 50%/80%/100% -> post_system_message()
- 20 Tests in test_cost_protection.py

Total: 68 neue Tests, alle grün. Keine Regressionen.
This commit is contained in:
Agent Zero
2026-08-13 21:33:14 +02:00
parent 1baa9481a2
commit b5546ea7bd
13 changed files with 1461 additions and 31 deletions
+51
View File
@@ -146,6 +146,57 @@ async def send_ws_error(
logger.debug("Failed to send WS error to client")
# ── Global connection registry for drain_all_connections ────────────────────
# Plugin WS endpoints register their connection registries here so that
# drain_all_connections() can close them all during graceful shutdown.
_global_ws_registries: list[dict[str, list[WebSocket]]] = []
def register_ws_registry(registry: dict[str, list[WebSocket]]) -> None:
"""Register a WebSocket connection registry for graceful shutdown."""
if registry not in _global_ws_registries:
_global_ws_registries.append(registry)
async def drain_all_connections(grace_period_seconds: float = 5.0) -> None:
"""Notify all connected WS clients about reconnect and close connections.
Sends a ``reconnect`` hint message to every connected client, waits
for ``grace_period_seconds``, then forcefully closes all sockets.
Called during application graceful shutdown.
"""
total_connections = 0
for registry in _global_ws_registries:
for user_id, conns in list(registry.items()):
for ws in list(conns):
try:
await ws.send_text(json.dumps({
"type": "reconnect",
"reason": "server_shutdown",
"message": "Server is shutting down. Please reconnect shortly.",
}))
total_connections += 1
except Exception:
logger.debug("Failed to send reconnect hint to WS client")
logger.info(f"WS drain: notified {total_connections} connections, waiting {grace_period_seconds}s")
if grace_period_seconds > 0:
await asyncio.sleep(grace_period_seconds)
# Close all connections
for registry in _global_ws_registries:
for user_id, conns in list(registry.items()):
for ws in list(conns):
try:
await ws.close(code=1001, reason="Server shutting down")
except Exception:
logger.debug("WS already closed during drain")
registry.clear()
_global_ws_registries.clear()
logger.info("WS drain: all connections closed")
async def handle_ws_message(
websocket: WebSocket,
message: str,