leocrm/docs/admin-guide.md

# LeoCRM Admin Guide

> Operations manual for LeoCRM administrators: deployment, backup, restore, environment configuration, and troubleshooting.

## Table of Contents

1. [Deployment](#deployment)
2. [Environment Configuration](#environment-configuration)
3. [Environment Profiles](#environment-profiles)
4. [Backup](#backup)
5. [Restore](#restore)
6. [Monitoring](#monitoring)
7. [Troubleshooting](#troubleshooting)

---

## Deployment

### Prerequisites

- Docker 24+ and Docker Compose v2
- PostgreSQL 15+ (or use the included Docker container)
- Redis 7+ (or use the included Docker container)
- A Coolify instance (for managed deployment) or a VPS with Docker

### Docker Compose Deployment (Production)

1. **Clone the repository:**

   ```bash
   git clone <repo-url> leocrm
   cd leocrm
   ```

2. **Copy and configure environment:**

   ```bash
   cp .env.example .env
   # Edit .env — set DATABASE_URL, REDIS_URL, SECRET_KEY, CORS_ORIGINS
   nano .env
   ```

3. **Start services:**

   ```bash
   docker compose up -d
   ```

4. **Run database migrations:**

   ```bash
   docker compose exec api alembic upgrade head
   ```

5. **Create the first admin user:**

   ```bash
   docker compose exec api python -c \
     "from app.services.auth_service import bootstrap_first_user; import asyncio; asyncio.run(bootstrap_first_user('admin@example.com', 'SecurePassword123!', 'Admin'))"
   ```

6. **Verify health:**

   ```bash
   curl http://localhost:8000/api/v1/health
   ```

### Coolify Deployment

See [COOLIFY_SETUP.md](../COOLIFY_SETUP.md) for detailed Coolify deployment instructions.

### Manual Deployment (without Docker)

1. Install Python 3.11+ and PostgreSQL 15+
2. Create a virtual environment: `python3 -m venv .venv && source .venv/bin/activate`
3. Install dependencies: `pip install -r requirements.txt`
4. Configure `.env` (see [Environment Configuration](#environment-configuration))
5. Run migrations: `alembic upgrade head`
6. Start the server: `uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 2`
7. Start the ARQ worker: `arq app.core.jobs.WorkerSettings`

---

## Environment Configuration

All configuration is managed via environment variables (loaded from `.env`).

### Required Variables

| Variable | Description | Example |
|---|---|---|
| `DATABASE_URL` | PostgreSQL async connection URL | `postgresql+asyncpg://user:pass@host:5432/dbname` |
| `REDIS_URL` | Redis connection URL | `redis://localhost:6379/0` |
| `SECRET_KEY` | Secret key for signing (≥32 chars) | Use `python3 -c "import secrets; print(secrets.token_urlsafe(48))"` |
| `CORS_ORIGINS` | Comma-separated allowed origins (no wildcards) | `https://crm.example.com` |

### Database Configuration

| Variable | Default | Description |
|---|---|---|
| `DB_POOL_SIZE` | `10` | Connection pool size |
| `DB_MAX_OVERFLOW` | `20` | Max overflow connections |
| `DB_ECHO` | `false` | Echo SQL statements (debug only) |

### Redis Configuration

| Variable | Default | Description |
|---|---|---|
| `REDIS_URL` | `redis://localhost:6379/0` | Redis connection URL |
| `SESSION_TTL_SECONDS` | `28800` | Session lifetime (8 hours) |

### SMTP / Email Configuration

| Variable | Default | Description |
|---|---|---|
| `SMTP_HOST` | `localhost` | SMTP server hostname |
| `SMTP_PORT` | `587` | SMTP server port |
| `SMTP_USERNAME` | _(none)_ | SMTP username (if auth required) |
| `SMTP_PASSWORD` | _(none)_ | SMTP password (if auth required) |
| `SMTP_FROM_EMAIL` | `noreply@leocrm.local` | From email address |
| `SMTP_USE_TLS` | `true` | Use TLS for SMTP connection |

### Storage Configuration

| Variable | Default | Description |
|---|---|---|
| `STORAGE_PATH` | `/tmp` | File storage path (DMS, uploads) |

### Security Configuration

| Variable | Default | Description |
|---|---|---|
| `BCRYPT_ROUNDS` | `12` | Password hashing cost factor |
| `SESSION_COOKIE_SECURE` | `false` | Set `true` in production (HTTPS only) |
| `SESSION_COOKIE_SAMESITE` | `strict` | SameSite cookie policy |
| `SESSION_COOKIE_HTTPONLY` | `true` | HttpOnly cookie flag |
| `PASSWORD_RESET_EXPIRY_HOURS` | `1` | Password reset token lifetime |

### Rate Limiting

| Variable | Default | Description |
|---|---|---|
| `RATE_LIMIT_LOGIN_MAX` | `5` | Max login attempts per window |
| `RATE_LIMIT_LOGIN_WINDOW` | `900` | Login window (seconds, 15 min) |
| `RATE_LIMIT_GENERAL_MAX` | `60` | General API rate limit |
| `RATE_LIMIT_GENERAL_WINDOW` | `60` | General window (seconds, 1 min) |

---

## Environment Profiles

LeoCRM supports three environment profiles via the `ENVIRONMENT` variable.

### Development (`ENVIRONMENT=development`)

- **Database**: Local PostgreSQL or Docker container
- **Redis**: Local instance
- **Logging**: DEBUG level (verbose)
- **CORS**: `http://localhost:5173,http://localhost:3000`
- **Session cookie secure**: `false`
- **DB Echo**: `false` (set `true` for SQL debugging)
- **Auto-reload**: `uvicorn app.main:app --reload --port 8000`

### Testing (`ENVIRONMENT=testing`)

- **Database**: `leocrm_test` database (separate from dev/prod)
- **Redis**: Local instance (flushed between tests)
- **Logging**: WARNING level (minimal output)
- **Test fixtures**: Auto-seeded tenants, users, and test data
- **Run tests**: `pytest -v --tb=short`

### Production (`ENVIRONMENT=production`)

- **Database**: Managed PostgreSQL (e.g., Coolify managed DB)
- **Redis**: Separate Redis container or managed service
- **Logging**: INFO level (structured JSON via structlog)
- **CORS**: Production domain only (e.g., `https://crm.example.com`)
- **Session cookie secure**: `true` (HTTPS only)
- **DB Echo**: `false`
- **Workers**: `uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 2`
- **ARQ Worker**: `arq app.core.jobs.WorkerSettings`
- **Secret key**: Must be a secure random string (≥32 chars)

---

## Backup

### Database Backup

```bash
# Full database dump (recommended daily)
pg_dump -U leocrm -h localhost leocrm > backup_$(date +%Y%m%d).sql

# Compressed backup
pg_dump -U leocrm -h localhost leocrm | gzip > backup_$(date +%Y%m%d).sql.gz
```

### Automated Backup (Cron)

Add to crontab for daily backup at 2 AM:

```cron
0 2 * * * pg_dump -U leocrm -h localhost leocrm | gzip > /backups/leocrm_$(date +\%Y\%m\%d).sql.gz
```

### File Storage Backup

```bash
# Backup the storage directory
tar -czf storage_$(date +%Y%m%d).tar.gz /data/storage/
```

### Redis Backup

```bash
# Save Redis snapshot
redis-cli SAVE
cp /var/lib/redis/dump.rdb /backups/redis_$(date +%Y%m%d).rdb
```

---

## Restore

### Database Restore

```bash
# Restore from SQL dump
psql -U leocrm -h localhost leocrm < backup_20260101.sql

# Restore from compressed backup
gunzip -c backup_20260101.sql.gz | psql -U leocrm -h localhost leocrm
```

### File Storage Restore

```bash
# Restore storage directory
tar -xzf storage_20260101.tar.gz -C /data/
```

### Redis Restore

```bash
# Stop Redis, replace dump file, start Redis
systemctl stop redis
cp /backups/redis_20260101.rdb /var/lib/redis/dump.rdb
systemctl start redis
```

---

## Monitoring

### Health Endpoint

```bash
curl http://localhost:8000/api/v1/health
```

Returns JSON with overall status and individual checks:
- `database` — PostgreSQL connectivity
- `redis` — Redis connectivity
- `storage` — Storage directory writability
- `worker` — ARQ worker queue status

Status values: `healthy` (all checks up) or `degraded` (one or more checks down).

### Prometheus Metrics

```bash
# Requires admin authentication
curl -H "Cookie: leocrm_session=<session_id>" http://localhost:8000/api/v1/metrics
```

Available metrics:
- `leocrm_http_requests_total` — Total HTTP requests (by method, path, status)
- `leocrm_http_request_duration_seconds` — Request duration histogram
- `leocrm_db_pool_connections` — Database connection pool size
- `leocrm_arq_jobs_total` — Total ARQ background jobs (by function, status)

### Structured Logging

LeoCRM uses `structlog` for structured JSON logging. Logs include:
- `timestamp` — ISO timestamp
- `level` — Log level (INFO, ERROR, etc.)
- `event` — Event name (e.g., `api_request`)
- `method` — HTTP method
- `path` — Request path
- `status` — HTTP status code
- `duration_ms` — Request duration in milliseconds
- `tenant_id` — Tenant identifier (when available)

Error logs additionally include:
- `error` — Error message
- `traceback` — Full stacktrace

### Performance Testing

```bash
# Seed 200k contacts for performance testing
python scripts/seed_perf_data.py --count 200000

# Verify database indexes
python scripts/check_indexes.py
```

---

## Troubleshooting

### Database Connection Issues

**Symptom**: Health check reports `database: down`

1. Verify PostgreSQL is running: `systemctl status postgresql`
2. Check connection string in `.env`: `DATABASE_URL=postgresql+asyncpg://...`
3. Test connection: `psql -U leocrm -h localhost leocrm`
4. Check pool settings: `DB_POOL_SIZE` and `DB_MAX_OVERFLOW`

### Redis Connection Issues

**Symptom**: Health check reports `redis: down`, sessions not persisting

1. Verify Redis is running: `systemctl status redis` or `redis-cli ping`
2. Check `REDIS_URL` in `.env`
3. Check Redis logs for errors

### Storage Issues

**Symptom**: Health check reports `storage: down`, file uploads failing

1. Verify storage path exists: `ls -la $STORAGE_PATH`
2. Check write permissions: `touch $STORAGE_PATH/test && rm $STORAGE_PATH/test`
3. Update `STORAGE_PATH` in `.env` if needed

### Worker Issues

**Symptom**: Background jobs not processing, health check reports `worker: down`

1. Verify ARQ worker is running: `ps aux | grep arq`
2. Start worker: `arq app.core.jobs.WorkerSettings`
3. Check Redis queue: `redis-cli LLEN arq:queue`

### Performance Issues

**Symptom**: Slow API responses (>500ms)

1. Run `python scripts/check_indexes.py` — verify all indexes exist
2. Check database pool size: increase `DB_POOL_SIZE` if needed
3. Seed test data: `python scripts/seed_perf_data.py --count 10000`
4. Check Prometheus metrics at `/api/v1/metrics` for slow endpoints
5. Enable SQL echo (temporarily): set `DB_ECHO=true` in `.env`

### Authentication Issues

**Symptom**: Login fails, 401 errors

1. Verify user exists and is active
2. Check session cookie name: `SESSION_COOKIE_NAME` in `.env`
3. Verify `SESSION_COOKIE_SECURE` is `false` in development (no HTTPS)
4. Check Redis for session data: `redis-cli KEYS session:*`

### CORS Errors

**Symptom**: Browser console shows CORS errors

1. Check `CORS_ORIGINS` in `.env` — must include frontend URL
2. No wildcards allowed — explicit origins only
3. Restart server after changing `.env`

### Migration Issues

**Symptom**: `alembic upgrade head` fails

1. Check database connectivity
2. Verify Alembic config: `alembic.ini`
3. Review migration files in `alembic/versions/`
4. Check current revision: `alembic current`
5. Reset (DESTRUCTIVE): `alembic downgrade base && alembic upgrade head`