Phase 4: Circuit Breaker, DB Retry, Redis Graceful Degradation

- app/core/resilience.py: CircuitBreaker (CLOSED/OPEN/HALF_OPEN), retry_db,
  redis_call_with_fallback, InMemoryRateLimiter, CircuitBreakerMiddleware
- app/core/auth.py: get_session_data now falls back to PostgreSQL sessions
  table when Redis is unavailable
- app/core/permissions.py: get_cached_permissions falls back to direct DB
  resolution when Redis circuit is open
- app/core/rate_limit.py: check_rate_limit falls back to in-memory limiter
  when Redis is down; reset_rate_limit clears both Redis and in-memory
- app/core/middleware.py: CSRF validation uses get_session_data (Redis+DB
  fallback); sliding session TTL is best-effort during outage
- app/core/db/__init__.py: get_db() wraps session creation with retry_db
  for transient connection errors; records circuit breaker success/failure
- app/deps.py: refresh_session_ttl wrapped in try/except for Redis outage
- app/main.py: CircuitBreakerMiddleware registered (returns 503 when DB
  circuit is OPEN, skips health/metrics endpoints)
- app/config.py: Added resilience settings (thresholds, cooldown, retries)
- tests/test_resilience.py: 30 tests covering all patterns

30/30 resilience tests pass. No regressions in plugin lifecycle tests.
This commit is contained in:
Agent Zero
2026-08-04 14:34:06 +02:00
parent 247d4165ea
commit a26405f15e
10 changed files with 875 additions and 58 deletions
+7
View File
@@ -66,6 +66,13 @@ class Settings(BaseSettings):
# Trusted proxy CIDRs (comma-separated) — only these proxies can set X-Forwarded-For
trusted_proxy_cidrs: str = ""
# Resilience
circuit_breaker_failure_threshold: int = 5
circuit_breaker_window_seconds: int = 30
circuit_breaker_cooldown_seconds: int = 60
db_retry_max_attempts: int = 3
db_retry_base_delay: float = 0.1
# Rate Limiting
rate_limit_login_max: int = 5
rate_limit_login_window: int = 900 # 15 min