Phase 4: Circuit Breaker, DB Retry, Redis Graceful Degradation
- app/core/resilience.py: CircuitBreaker (CLOSED/OPEN/HALF_OPEN), retry_db, redis_call_with_fallback, InMemoryRateLimiter, CircuitBreakerMiddleware - app/core/auth.py: get_session_data now falls back to PostgreSQL sessions table when Redis is unavailable - app/core/permissions.py: get_cached_permissions falls back to direct DB resolution when Redis circuit is open - app/core/rate_limit.py: check_rate_limit falls back to in-memory limiter when Redis is down; reset_rate_limit clears both Redis and in-memory - app/core/middleware.py: CSRF validation uses get_session_data (Redis+DB fallback); sliding session TTL is best-effort during outage - app/core/db/__init__.py: get_db() wraps session creation with retry_db for transient connection errors; records circuit breaker success/failure - app/deps.py: refresh_session_ttl wrapped in try/except for Redis outage - app/main.py: CircuitBreakerMiddleware registered (returns 503 when DB circuit is OPEN, skips health/metrics endpoints) - app/config.py: Added resilience settings (thresholds, cooldown, retries) - tests/test_resilience.py: 30 tests covering all patterns 30/30 resilience tests pass. No regressions in plugin lifecycle tests.
This commit is contained in:
+18
-8
@@ -222,15 +222,25 @@ async def get_db() -> AsyncGenerator[AsyncSession, None]:
|
||||
"""FastAPI dependency: yield an async database session (crm_api role).
|
||||
|
||||
Used for normal API requests with tenant context set via RLS.
|
||||
Includes retry logic for transient connection errors.
|
||||
"""
|
||||
factory = get_session_factory()
|
||||
async with factory() as session:
|
||||
try:
|
||||
yield session
|
||||
await session.commit()
|
||||
except Exception:
|
||||
await session.rollback()
|
||||
raise
|
||||
from app.core.resilience import get_circuit, retry_db
|
||||
|
||||
async def _get_session():
|
||||
factory = get_session_factory()
|
||||
return factory()
|
||||
|
||||
session = await retry_db(_get_session)
|
||||
try:
|
||||
yield session
|
||||
await session.commit()
|
||||
await get_circuit("db").record_success()
|
||||
except Exception:
|
||||
await session.rollback()
|
||||
await get_circuit("db").record_failure()
|
||||
raise
|
||||
finally:
|
||||
await session.close()
|
||||
|
||||
|
||||
async def get_auth_db() -> AsyncGenerator[AsyncSession, None]:
|
||||
|
||||
Reference in New Issue
Block a user