T10: Monitoring, Performance, Documentation & Environment Config — 38 tests, ruff clean, docs OK
- Extended health endpoint: DB+Redis+Storage+Worker checks with degraded status - Prometheus metrics endpoint: admin-only, text/plain format - Metrics: http_requests_total, db_pool_connections, arq_jobs_total - Structured JSON logging (structlog): timestamp, level, method, path, status, duration_ms, tenant_id - Performance: page_size max 100 enforced (422), streaming CSV export (StreamingResponse) - Scripts: seed_perf_data.py, check_indexes.py - Docs: admin-guide.md (Deploy, Backup, Restore, Env-Vars, Troubleshooting), api-overview.md - README updated: prod setup, API section, env profiles, admin-guide link - .env.example: added SECRET_KEY, STORAGE_PATH, SMTP_* vars - 38 new tests, full regression 564/564 pass (0 failures) - Ruff: all checks passed
This commit is contained in:
leocrm-bot
@@ -0,0 +1,380 @@
|
||||
# LeoCRM Admin Guide
|
||||
|
||||
> Operations manual for LeoCRM administrators: deployment, backup, restore, environment configuration, and troubleshooting.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Deployment](#deployment)
|
||||
2. [Environment Configuration](#environment-configuration)
|
||||
3. [Environment Profiles](#environment-profiles)
|
||||
4. [Backup](#backup)
|
||||
5. [Restore](#restore)
|
||||
6. [Monitoring](#monitoring)
|
||||
7. [Troubleshooting](#troubleshooting)
|
||||
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Docker 24+ and Docker Compose v2
|
||||
- PostgreSQL 15+ (or use the included Docker container)
|
||||
- Redis 7+ (or use the included Docker container)
|
||||
- A Coolify instance (for managed deployment) or a VPS with Docker
|
||||
|
||||
### Docker Compose Deployment (Production)
|
||||
|
||||
1. **Clone the repository:**
|
||||
|
||||
```bash
|
||||
git clone <repo-url> leocrm
|
||||
cd leocrm
|
||||
```
|
||||
|
||||
2. **Copy and configure environment:**
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
# Edit .env — set DATABASE_URL, REDIS_URL, SECRET_KEY, CORS_ORIGINS
|
||||
nano .env
|
||||
```
|
||||
|
||||
3. **Start services:**
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
4. **Run database migrations:**
|
||||
|
||||
```bash
|
||||
docker compose exec api alembic upgrade head
|
||||
```
|
||||
|
||||
5. **Create the first admin user:**
|
||||
|
||||
```bash
|
||||
docker compose exec api python -c \
|
||||
"from app.services.auth_service import bootstrap_first_user; import asyncio; asyncio.run(bootstrap_first_user('admin@example.com', 'SecurePassword123!', 'Admin'))"
|
||||
```
|
||||
|
||||
6. **Verify health:**
|
||||
|
||||
```bash
|
||||
curl http://localhost:8000/api/v1/health
|
||||
```
|
||||
|
||||
### Coolify Deployment
|
||||
|
||||
See [COOLIFY_SETUP.md](../COOLIFY_SETUP.md) for detailed Coolify deployment instructions.
|
||||
|
||||
### Manual Deployment (without Docker)
|
||||
|
||||
1. Install Python 3.11+ and PostgreSQL 15+
|
||||
2. Create a virtual environment: `python3 -m venv .venv && source .venv/bin/activate`
|
||||
3. Install dependencies: `pip install -r requirements.txt`
|
||||
4. Configure `.env` (see [Environment Configuration](#environment-configuration))
|
||||
5. Run migrations: `alembic upgrade head`
|
||||
6. Start the server: `uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 2`
|
||||
7. Start the ARQ worker: `arq app.core.jobs.WorkerSettings`
|
||||
|
||||
---
|
||||
|
||||
## Environment Configuration
|
||||
|
||||
All configuration is managed via environment variables (loaded from `.env`).
|
||||
|
||||
### Required Variables
|
||||
|
||||
| Variable | Description | Example |
|
||||
|---|---|---|
|
||||
| `DATABASE_URL` | PostgreSQL async connection URL | `postgresql+asyncpg://user:pass@host:5432/dbname` |
|
||||
| `REDIS_URL` | Redis connection URL | `redis://localhost:6379/0` |
|
||||
| `SECRET_KEY` | Secret key for signing (≥32 chars) | Use `python3 -c "import secrets; print(secrets.token_urlsafe(48))"` |
|
||||
| `CORS_ORIGINS` | Comma-separated allowed origins (no wildcards) | `https://crm.example.com` |
|
||||
|
||||
### Database Configuration
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `DB_POOL_SIZE` | `10` | Connection pool size |
|
||||
| `DB_MAX_OVERFLOW` | `20` | Max overflow connections |
|
||||
| `DB_ECHO` | `false` | Echo SQL statements (debug only) |
|
||||
|
||||
### Redis Configuration
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `REDIS_URL` | `redis://localhost:6379/0` | Redis connection URL |
|
||||
| `SESSION_TTL_SECONDS` | `28800` | Session lifetime (8 hours) |
|
||||
|
||||
### SMTP / Email Configuration
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `SMTP_HOST` | `localhost` | SMTP server hostname |
|
||||
| `SMTP_PORT` | `587` | SMTP server port |
|
||||
| `SMTP_USERNAME` | _(none)_ | SMTP username (if auth required) |
|
||||
| `SMTP_PASSWORD` | _(none)_ | SMTP password (if auth required) |
|
||||
| `SMTP_FROM_EMAIL` | `noreply@leocrm.local` | From email address |
|
||||
| `SMTP_USE_TLS` | `true` | Use TLS for SMTP connection |
|
||||
|
||||
### Storage Configuration
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `STORAGE_PATH` | `/tmp` | File storage path (DMS, uploads) |
|
||||
|
||||
### Security Configuration
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `BCRYPT_ROUNDS` | `12` | Password hashing cost factor |
|
||||
| `SESSION_COOKIE_SECURE` | `false` | Set `true` in production (HTTPS only) |
|
||||
| `SESSION_COOKIE_SAMESITE` | `strict` | SameSite cookie policy |
|
||||
| `SESSION_COOKIE_HTTPONLY` | `true` | HttpOnly cookie flag |
|
||||
| `PASSWORD_RESET_EXPIRY_HOURS` | `1` | Password reset token lifetime |
|
||||
|
||||
### Rate Limiting
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `RATE_LIMIT_LOGIN_MAX` | `5` | Max login attempts per window |
|
||||
| `RATE_LIMIT_LOGIN_WINDOW` | `900` | Login window (seconds, 15 min) |
|
||||
| `RATE_LIMIT_GENERAL_MAX` | `60` | General API rate limit |
|
||||
| `RATE_LIMIT_GENERAL_WINDOW` | `60` | General window (seconds, 1 min) |
|
||||
|
||||
---
|
||||
|
||||
## Environment Profiles
|
||||
|
||||
LeoCRM supports three environment profiles via the `ENVIRONMENT` variable.
|
||||
|
||||
### Development (`ENVIRONMENT=development`)
|
||||
|
||||
- **Database**: Local PostgreSQL or Docker container
|
||||
- **Redis**: Local instance
|
||||
- **Logging**: DEBUG level (verbose)
|
||||
- **CORS**: `http://localhost:5173,http://localhost:3000`
|
||||
- **Session cookie secure**: `false`
|
||||
- **DB Echo**: `false` (set `true` for SQL debugging)
|
||||
- **Auto-reload**: `uvicorn app.main:app --reload --port 8000`
|
||||
|
||||
### Testing (`ENVIRONMENT=testing`)
|
||||
|
||||
- **Database**: `leocrm_test` database (separate from dev/prod)
|
||||
- **Redis**: Local instance (flushed between tests)
|
||||
- **Logging**: WARNING level (minimal output)
|
||||
- **Test fixtures**: Auto-seeded tenants, users, and test data
|
||||
- **Run tests**: `pytest -v --tb=short`
|
||||
|
||||
### Production (`ENVIRONMENT=production`)
|
||||
|
||||
- **Database**: Managed PostgreSQL (e.g., Coolify managed DB)
|
||||
- **Redis**: Separate Redis container or managed service
|
||||
- **Logging**: INFO level (structured JSON via structlog)
|
||||
- **CORS**: Production domain only (e.g., `https://crm.example.com`)
|
||||
- **Session cookie secure**: `true` (HTTPS only)
|
||||
- **DB Echo**: `false`
|
||||
- **Workers**: `uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 2`
|
||||
- **ARQ Worker**: `arq app.core.jobs.WorkerSettings`
|
||||
- **Secret key**: Must be a secure random string (≥32 chars)
|
||||
|
||||
---
|
||||
|
||||
## Backup
|
||||
|
||||
### Database Backup
|
||||
|
||||
```bash
|
||||
# Full database dump (recommended daily)
|
||||
pg_dump -U leocrm -h localhost leocrm > backup_$(date +%Y%m%d).sql
|
||||
|
||||
# Compressed backup
|
||||
pg_dump -U leocrm -h localhost leocrm | gzip > backup_$(date +%Y%m%d).sql.gz
|
||||
```
|
||||
|
||||
### Automated Backup (Cron)
|
||||
|
||||
Add to crontab for daily backup at 2 AM:
|
||||
|
||||
```cron
|
||||
0 2 * * * pg_dump -U leocrm -h localhost leocrm | gzip > /backups/leocrm_$(date +\%Y\%m\%d).sql.gz
|
||||
```
|
||||
|
||||
### File Storage Backup
|
||||
|
||||
```bash
|
||||
# Backup the storage directory
|
||||
tar -czf storage_$(date +%Y%m%d).tar.gz /data/storage/
|
||||
```
|
||||
|
||||
### Redis Backup
|
||||
|
||||
```bash
|
||||
# Save Redis snapshot
|
||||
redis-cli SAVE
|
||||
cp /var/lib/redis/dump.rdb /backups/redis_$(date +%Y%m%d).rdb
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Restore
|
||||
|
||||
### Database Restore
|
||||
|
||||
```bash
|
||||
# Restore from SQL dump
|
||||
psql -U leocrm -h localhost leocrm < backup_20260101.sql
|
||||
|
||||
# Restore from compressed backup
|
||||
gunzip -c backup_20260101.sql.gz | psql -U leocrm -h localhost leocrm
|
||||
```
|
||||
|
||||
### File Storage Restore
|
||||
|
||||
```bash
|
||||
# Restore storage directory
|
||||
tar -xzf storage_20260101.tar.gz -C /data/
|
||||
```
|
||||
|
||||
### Redis Restore
|
||||
|
||||
```bash
|
||||
# Stop Redis, replace dump file, start Redis
|
||||
systemctl stop redis
|
||||
cp /backups/redis_20260101.rdb /var/lib/redis/dump.rdb
|
||||
systemctl start redis
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Health Endpoint
|
||||
|
||||
```bash
|
||||
curl http://localhost:8000/api/v1/health
|
||||
```
|
||||
|
||||
Returns JSON with overall status and individual checks:
|
||||
- `database` — PostgreSQL connectivity
|
||||
- `redis` — Redis connectivity
|
||||
- `storage` — Storage directory writability
|
||||
- `worker` — ARQ worker queue status
|
||||
|
||||
Status values: `healthy` (all checks up) or `degraded` (one or more checks down).
|
||||
|
||||
### Prometheus Metrics
|
||||
|
||||
```bash
|
||||
# Requires admin authentication
|
||||
curl -H "Cookie: leocrm_session=<session_id>" http://localhost:8000/api/v1/metrics
|
||||
```
|
||||
|
||||
Available metrics:
|
||||
- `leocrm_http_requests_total` — Total HTTP requests (by method, path, status)
|
||||
- `leocrm_http_request_duration_seconds` — Request duration histogram
|
||||
- `leocrm_db_pool_connections` — Database connection pool size
|
||||
- `leocrm_arq_jobs_total` — Total ARQ background jobs (by function, status)
|
||||
|
||||
### Structured Logging
|
||||
|
||||
LeoCRM uses `structlog` for structured JSON logging. Logs include:
|
||||
- `timestamp` — ISO timestamp
|
||||
- `level` — Log level (INFO, ERROR, etc.)
|
||||
- `event` — Event name (e.g., `api_request`)
|
||||
- `method` — HTTP method
|
||||
- `path` — Request path
|
||||
- `status` — HTTP status code
|
||||
- `duration_ms` — Request duration in milliseconds
|
||||
- `tenant_id` — Tenant identifier (when available)
|
||||
|
||||
Error logs additionally include:
|
||||
- `error` — Error message
|
||||
- `traceback` — Full stacktrace
|
||||
|
||||
### Performance Testing
|
||||
|
||||
```bash
|
||||
# Seed 200k contacts for performance testing
|
||||
python scripts/seed_perf_data.py --count 200000
|
||||
|
||||
# Verify database indexes
|
||||
python scripts/check_indexes.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Database Connection Issues
|
||||
|
||||
**Symptom**: Health check reports `database: down`
|
||||
|
||||
1. Verify PostgreSQL is running: `systemctl status postgresql`
|
||||
2. Check connection string in `.env`: `DATABASE_URL=postgresql+asyncpg://...`
|
||||
3. Test connection: `psql -U leocrm -h localhost leocrm`
|
||||
4. Check pool settings: `DB_POOL_SIZE` and `DB_MAX_OVERFLOW`
|
||||
|
||||
### Redis Connection Issues
|
||||
|
||||
**Symptom**: Health check reports `redis: down`, sessions not persisting
|
||||
|
||||
1. Verify Redis is running: `systemctl status redis` or `redis-cli ping`
|
||||
2. Check `REDIS_URL` in `.env`
|
||||
3. Check Redis logs for errors
|
||||
|
||||
### Storage Issues
|
||||
|
||||
**Symptom**: Health check reports `storage: down`, file uploads failing
|
||||
|
||||
1. Verify storage path exists: `ls -la $STORAGE_PATH`
|
||||
2. Check write permissions: `touch $STORAGE_PATH/test && rm $STORAGE_PATH/test`
|
||||
3. Update `STORAGE_PATH` in `.env` if needed
|
||||
|
||||
### Worker Issues
|
||||
|
||||
**Symptom**: Background jobs not processing, health check reports `worker: down`
|
||||
|
||||
1. Verify ARQ worker is running: `ps aux | grep arq`
|
||||
2. Start worker: `arq app.core.jobs.WorkerSettings`
|
||||
3. Check Redis queue: `redis-cli LLEN arq:queue`
|
||||
|
||||
### Performance Issues
|
||||
|
||||
**Symptom**: Slow API responses (>500ms)
|
||||
|
||||
1. Run `python scripts/check_indexes.py` — verify all indexes exist
|
||||
2. Check database pool size: increase `DB_POOL_SIZE` if needed
|
||||
3. Seed test data: `python scripts/seed_perf_data.py --count 10000`
|
||||
4. Check Prometheus metrics at `/api/v1/metrics` for slow endpoints
|
||||
5. Enable SQL echo (temporarily): set `DB_ECHO=true` in `.env`
|
||||
|
||||
### Authentication Issues
|
||||
|
||||
**Symptom**: Login fails, 401 errors
|
||||
|
||||
1. Verify user exists and is active
|
||||
2. Check session cookie name: `SESSION_COOKIE_NAME` in `.env`
|
||||
3. Verify `SESSION_COOKIE_SECURE` is `false` in development (no HTTPS)
|
||||
4. Check Redis for session data: `redis-cli KEYS session:*`
|
||||
|
||||
### CORS Errors
|
||||
|
||||
**Symptom**: Browser console shows CORS errors
|
||||
|
||||
1. Check `CORS_ORIGINS` in `.env` — must include frontend URL
|
||||
2. No wildcards allowed — explicit origins only
|
||||
3. Restart server after changing `.env`
|
||||
|
||||
### Migration Issues
|
||||
|
||||
**Symptom**: `alembic upgrade head` fails
|
||||
|
||||
1. Check database connectivity
|
||||
2. Verify Alembic config: `alembic.ini`
|
||||
3. Review migration files in `alembic/versions/`
|
||||
4. Check current revision: `alembic current`
|
||||
5. Reset (DESTRUCTIVE): `alembic downgrade base && alembic upgrade head`
|
||||
Reference in New Issue
Block a user