Phase 4: Circuit Breaker, DB Retry, Redis Graceful Degradation
- app/core/resilience.py: CircuitBreaker (CLOSED/OPEN/HALF_OPEN), retry_db, redis_call_with_fallback, InMemoryRateLimiter, CircuitBreakerMiddleware - app/core/auth.py: get_session_data now falls back to PostgreSQL sessions table when Redis is unavailable - app/core/permissions.py: get_cached_permissions falls back to direct DB resolution when Redis circuit is open - app/core/rate_limit.py: check_rate_limit falls back to in-memory limiter when Redis is down; reset_rate_limit clears both Redis and in-memory - app/core/middleware.py: CSRF validation uses get_session_data (Redis+DB fallback); sliding session TTL is best-effort during outage - app/core/db/__init__.py: get_db() wraps session creation with retry_db for transient connection errors; records circuit breaker success/failure - app/deps.py: refresh_session_ttl wrapped in try/except for Redis outage - app/main.py: CircuitBreakerMiddleware registered (returns 503 when DB circuit is OPEN, skips health/metrics endpoints) - app/config.py: Added resilience settings (thresholds, cooldown, retries) - tests/test_resilience.py: 30 tests covering all patterns 30/30 resilience tests pass. No regressions in plugin lifecycle tests.
This commit is contained in:
@@ -23,6 +23,7 @@ from app.core.db import close_engine, get_engine
|
||||
from app.core.error_codes import ApiError
|
||||
from app.core.middleware import CSRFMiddleware, SecurityHeadersMiddleware
|
||||
from app.core.rate_limit import GeneralRateLimitMiddleware
|
||||
from app.core.resilience import CircuitBreakerMiddleware
|
||||
from app.core.monitoring import record_error, record_request
|
||||
from app.core.plugin_error_handler import wrap_plugin_route
|
||||
from app.core.service_container import get_container
|
||||
@@ -379,6 +380,7 @@ def create_app() -> FastAPI:
|
||||
app.add_middleware(SecurityHeadersMiddleware)
|
||||
app.add_middleware(GeneralRateLimitMiddleware)
|
||||
app.add_middleware(RequestLoggingMiddleware)
|
||||
app.add_middleware(CircuitBreakerMiddleware)
|
||||
|
||||
# ── Global exception handler — catch ALL unhandled exceptions ──
|
||||
@app.exception_handler(Exception)
|
||||
|
||||
Reference in New Issue
Block a user