Phase 4: Circuit Breaker, DB Retry, Redis Graceful Degradation

- app/core/resilience.py: CircuitBreaker (CLOSED/OPEN/HALF_OPEN), retry_db,
  redis_call_with_fallback, InMemoryRateLimiter, CircuitBreakerMiddleware
- app/core/auth.py: get_session_data now falls back to PostgreSQL sessions
  table when Redis is unavailable
- app/core/permissions.py: get_cached_permissions falls back to direct DB
  resolution when Redis circuit is open
- app/core/rate_limit.py: check_rate_limit falls back to in-memory limiter
  when Redis is down; reset_rate_limit clears both Redis and in-memory
- app/core/middleware.py: CSRF validation uses get_session_data (Redis+DB
  fallback); sliding session TTL is best-effort during outage
- app/core/db/__init__.py: get_db() wraps session creation with retry_db
  for transient connection errors; records circuit breaker success/failure
- app/deps.py: refresh_session_ttl wrapped in try/except for Redis outage
- app/main.py: CircuitBreakerMiddleware registered (returns 503 when DB
  circuit is OPEN, skips health/metrics endpoints)
- app/config.py: Added resilience settings (thresholds, cooldown, retries)
- tests/test_resilience.py: 30 tests covering all patterns

30/30 resilience tests pass. No regressions in plugin lifecycle tests.
This commit is contained in:
Agent Zero
2026-08-04 14:34:06 +02:00
parent 247d4165ea
commit a26405f15e
10 changed files with 875 additions and 58 deletions
+5 -1
View File
@@ -103,7 +103,11 @@ async def get_current_user(
)
# Sliding session: extend TTL on each authenticated request
await refresh_session_ttl(redis, session_id)
# Best-effort during Redis outage — session still valid from DB fallback
try:
await refresh_session_ttl(redis, session_id)
except Exception:
logger.debug("refresh_session_ttl failed (Redis may be down) — continuing")
if not session_data.get("is_active", True):
raise HTTPException(