Phase 4: Circuit Breaker, DB Retry, Redis Graceful Degradation

- app/core/resilience.py: CircuitBreaker (CLOSED/OPEN/HALF_OPEN), retry_db,
  redis_call_with_fallback, InMemoryRateLimiter, CircuitBreakerMiddleware
- app/core/auth.py: get_session_data now falls back to PostgreSQL sessions
  table when Redis is unavailable
- app/core/permissions.py: get_cached_permissions falls back to direct DB
  resolution when Redis circuit is open
- app/core/rate_limit.py: check_rate_limit falls back to in-memory limiter
  when Redis is down; reset_rate_limit clears both Redis and in-memory
- app/core/middleware.py: CSRF validation uses get_session_data (Redis+DB
  fallback); sliding session TTL is best-effort during outage
- app/core/db/__init__.py: get_db() wraps session creation with retry_db
  for transient connection errors; records circuit breaker success/failure
- app/deps.py: refresh_session_ttl wrapped in try/except for Redis outage
- app/main.py: CircuitBreakerMiddleware registered (returns 503 when DB
  circuit is OPEN, skips health/metrics endpoints)
- app/config.py: Added resilience settings (thresholds, cooldown, retries)
- tests/test_resilience.py: 30 tests covering all patterns

30/30 resilience tests pass. No regressions in plugin lifecycle tests.
This commit is contained in:
Agent Zero
2026-08-04 14:34:06 +02:00
parent 247d4165ea
commit a26405f15e
10 changed files with 875 additions and 58 deletions
+2
View File
@@ -23,6 +23,7 @@ from app.core.db import close_engine, get_engine
from app.core.error_codes import ApiError
from app.core.middleware import CSRFMiddleware, SecurityHeadersMiddleware
from app.core.rate_limit import GeneralRateLimitMiddleware
from app.core.resilience import CircuitBreakerMiddleware
from app.core.monitoring import record_error, record_request
from app.core.plugin_error_handler import wrap_plugin_route
from app.core.service_container import get_container
@@ -379,6 +380,7 @@ def create_app() -> FastAPI:
app.add_middleware(SecurityHeadersMiddleware)
app.add_middleware(GeneralRateLimitMiddleware)
app.add_middleware(RequestLoggingMiddleware)
app.add_middleware(CircuitBreakerMiddleware)
# ── Global exception handler — catch ALL unhandled exceptions ──
@app.exception_handler(Exception)