Phase 4: Circuit Breaker, DB Retry, Redis Graceful Degradation

- app/core/resilience.py: CircuitBreaker (CLOSED/OPEN/HALF_OPEN), retry_db,
  redis_call_with_fallback, InMemoryRateLimiter, CircuitBreakerMiddleware
- app/core/auth.py: get_session_data now falls back to PostgreSQL sessions
  table when Redis is unavailable
- app/core/permissions.py: get_cached_permissions falls back to direct DB
  resolution when Redis circuit is open
- app/core/rate_limit.py: check_rate_limit falls back to in-memory limiter
  when Redis is down; reset_rate_limit clears both Redis and in-memory
- app/core/middleware.py: CSRF validation uses get_session_data (Redis+DB
  fallback); sliding session TTL is best-effort during outage
- app/core/db/__init__.py: get_db() wraps session creation with retry_db
  for transient connection errors; records circuit breaker success/failure
- app/deps.py: refresh_session_ttl wrapped in try/except for Redis outage
- app/main.py: CircuitBreakerMiddleware registered (returns 503 when DB
  circuit is OPEN, skips health/metrics endpoints)
- app/config.py: Added resilience settings (thresholds, cooldown, retries)
- tests/test_resilience.py: 30 tests covering all patterns

30/30 resilience tests pass. No regressions in plugin lifecycle tests.
This commit is contained in:
Agent Zero
2026-08-04 14:34:06 +02:00
parent 247d4165ea
commit a26405f15e
10 changed files with 875 additions and 58 deletions
+18 -8
View File
@@ -222,15 +222,25 @@ async def get_db() -> AsyncGenerator[AsyncSession, None]:
"""FastAPI dependency: yield an async database session (crm_api role).
Used for normal API requests with tenant context set via RLS.
Includes retry logic for transient connection errors.
"""
factory = get_session_factory()
async with factory() as session:
try:
yield session
await session.commit()
except Exception:
await session.rollback()
raise
from app.core.resilience import get_circuit, retry_db
async def _get_session():
factory = get_session_factory()
return factory()
session = await retry_db(_get_session)
try:
yield session
await session.commit()
await get_circuit("db").record_success()
except Exception:
await session.rollback()
await get_circuit("db").record_failure()
raise
finally:
await session.close()
async def get_auth_db() -> AsyncGenerator[AsyncSession, None]: