T10: Monitoring, Performance, Documentation & Environment Config — 38 tests, ruff clean, docs OK
- Extended health endpoint: DB+Redis+Storage+Worker checks with degraded status - Prometheus metrics endpoint: admin-only, text/plain format - Metrics: http_requests_total, db_pool_connections, arq_jobs_total - Structured JSON logging (structlog): timestamp, level, method, path, status, duration_ms, tenant_id - Performance: page_size max 100 enforced (422), streaming CSV export (StreamingResponse) - Scripts: seed_perf_data.py, check_indexes.py - Docs: admin-guide.md (Deploy, Backup, Restore, Env-Vars, Troubleshooting), api-overview.md - README updated: prod setup, API section, env profiles, admin-guide link - .env.example: added SECRET_KEY, STORAGE_PATH, SMTP_* vars - 38 new tests, full regression 564/564 pass (0 failures) - Ruff: all checks passed
This commit is contained in:
leocrm-bot
@@ -1,15 +1,15 @@
|
||||
# CRM System v1.0
|
||||
# LeoCRM v1.0
|
||||
|
||||
> Self-hosted CRM for small sales teams (5–25 sales reps).
|
||||
> Stack: FastAPI + SQLAlchemy (async) + Alembic + Pydantic v2 + SQLite/PostgreSQL + Alpine.js + Tailwind + Docker + Coolify
|
||||
> Stack: FastAPI + SQLAlchemy (async) + PostgreSQL + Redis + Alpine.js + Tailwind + Docker + Coolify
|
||||
|
||||
## Quick Start (Development)
|
||||
|
||||
### 1. Clone and Setup
|
||||
|
||||
```bash
|
||||
git clone <repo-url> crm-system
|
||||
cd crm-system
|
||||
git clone <repo-url> leocrm
|
||||
cd leocrm
|
||||
|
||||
# Create virtual environment
|
||||
python3 -m venv .venv
|
||||
@@ -22,13 +22,13 @@ pip install -r requirements.txt -r requirements-dev.txt
|
||||
### 2. Configure Environment
|
||||
|
||||
```bash
|
||||
# Copy template
|
||||
cp .env.example .env
|
||||
|
||||
# Generate a secure AUTH_SECRET (min 32 chars)
|
||||
python3 -c "import secrets; print('AUTH_SECRET=' + secrets.token_urlsafe(48))" >> .env
|
||||
# Generate a secure SECRET_KEY (min 32 chars)
|
||||
python3 -c "import secrets; print('SECRET_KEY=' + secrets.token_urlsafe(48))" >> .env
|
||||
|
||||
# Edit .env and set AUTH_SECRET (remove the placeholder line first)
|
||||
# Edit .env and set DATABASE_URL, REDIS_URL, SECRET_KEY
|
||||
nano .env
|
||||
```
|
||||
|
||||
### 3. Initialize Database
|
||||
@@ -36,9 +36,6 @@ python3 -c "import secrets; print('AUTH_SECRET=' + secrets.token_urlsafe(48))" >
|
||||
```bash
|
||||
# Apply migrations
|
||||
alembic upgrade head
|
||||
|
||||
# (Optional) Create migration after model changes
|
||||
# alembic revision --autogenerate -m "description"
|
||||
```
|
||||
|
||||
### 4. Run Server
|
||||
@@ -47,96 +44,200 @@ alembic upgrade head
|
||||
# Development with auto-reload
|
||||
uvicorn app.main:app --reload --port 8000
|
||||
|
||||
# Production-like
|
||||
uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 2
|
||||
# Start ARQ worker (for background jobs)
|
||||
arq app.core.jobs.WorkerSettings
|
||||
```
|
||||
|
||||
Open:
|
||||
- API: http://localhost:8000
|
||||
- Swagger UI: http://localhost:8000/docs
|
||||
- ReDoc: http://localhost:8000/redoc
|
||||
- Health: http://localhost:8000/health
|
||||
- Health: http://localhost:8000/api/v1/health
|
||||
- Metrics: http://localhost:8000/api/v1/metrics (admin-only)
|
||||
|
||||
### 5. Bootstrap First User
|
||||
## Production Setup
|
||||
|
||||
### Docker Compose
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/api/v1/auth/register \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"email": "admin@example.com",
|
||||
"password": "secure-password-123",
|
||||
"name": "First Admin"
|
||||
}'
|
||||
cp .env.example .env
|
||||
# Edit .env — set DATABASE_URL, REDIS_URL, SECRET_KEY, CORS_ORIGINS
|
||||
# Set ENVIRONMENT=production, SESSION_COOKIE_SECURE=true
|
||||
docker compose up -d
|
||||
|
||||
# Run migrations
|
||||
docker compose exec api alembic upgrade head
|
||||
```
|
||||
|
||||
This creates the first user + a default org. After that, registration is disabled (use admin invite flow in v1.1).
|
||||
### Manual (without Docker)
|
||||
|
||||
## Project Structure
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
alembic upgrade head
|
||||
|
||||
# Start API server (2 workers)
|
||||
uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 2
|
||||
|
||||
# Start ARQ worker (separate process)
|
||||
arq app.core.jobs.WorkerSettings
|
||||
```
|
||||
crm-system/
|
||||
├── app/ # Application package
|
||||
│ ├── main.py # FastAPI entry point
|
||||
│ ├── core/ # Core modules (config, db, security, deps)
|
||||
│ ├── models/ # SQLAlchemy models
|
||||
│ ├── schemas/ # Pydantic schemas (request/response)
|
||||
│ ├── services/ # Business logic layer
|
||||
│ ├── api/v1/ # API routers (versioned)
|
||||
│ └── webui/ # Static frontend (Phase 4c)
|
||||
├── alembic/ # Database migrations
|
||||
│ ├── env.py # Async migration environment
|
||||
│ └── versions/ # Migration scripts
|
||||
├── tests/ # Test suite (pytest + pytest-asyncio)
|
||||
├── requirements.txt # Production dependencies
|
||||
├── requirements-dev.txt # Test/lint dependencies
|
||||
├── pyproject.toml # Tool configuration
|
||||
├── alembic.ini # Alembic configuration
|
||||
├── .env.example # Environment template
|
||||
└── README.md
|
||||
|
||||
See [docs/admin-guide.md](docs/admin-guide.md) for detailed deployment, backup, and troubleshooting instructions.
|
||||
|
||||
## API
|
||||
|
||||
### Key Endpoints
|
||||
|
||||
| Endpoint | Method | Auth | Description |
|
||||
|---|---|---|---|
|
||||
| `/api/v1/health` | GET | No | Health check (DB, Redis, storage, worker) |
|
||||
| `/api/v1/metrics` | GET | Admin | Prometheus metrics (text/plain) |
|
||||
| `/api/v1/auth/login` | POST | No | Login |
|
||||
| `/api/v1/contacts` | GET | Yes | List contacts (paginated, max page_size=100) |
|
||||
| `/api/v1/contacts/export` | GET | Yes | Stream contacts as CSV |
|
||||
| `/api/v1/companies` | GET | Yes | List companies (paginated, max page_size=100) |
|
||||
| `/api/v1/companies/export` | GET | Yes | Stream companies as CSV |
|
||||
|
||||
### Pagination
|
||||
|
||||
All list endpoints support pagination with `page` and `page_size` parameters.
|
||||
`page_size` is capped at **100** — values >100 return HTTP 422.
|
||||
|
||||
### CSV Export
|
||||
|
||||
Contacts and companies support streaming CSV export via `/export?format=csv`.
|
||||
Uses `StreamingResponse` — does not buffer the entire file in memory.
|
||||
|
||||
### Swagger UI
|
||||
|
||||
Interactive API documentation: http://localhost:8000/docs
|
||||
|
||||
See [docs/api-overview.md](docs/api-overview.md) for the full endpoint summary.
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Health Check
|
||||
|
||||
```bash
|
||||
curl http://localhost:8000/api/v1/health
|
||||
```
|
||||
|
||||
Returns JSON with overall status (`healthy`/`degraded`) and individual checks for
|
||||
`database`, `redis`, `storage`, and `worker`.
|
||||
|
||||
### Prometheus Metrics
|
||||
|
||||
```bash
|
||||
# Requires admin authentication
|
||||
curl -b "leocrm_session=<session>" http://localhost:8000/api/v1/metrics
|
||||
```
|
||||
|
||||
Available metrics:
|
||||
- `leocrm_http_requests_total` — Total HTTP requests
|
||||
- `leocrm_http_request_duration_seconds` — Request duration histogram
|
||||
- `leocrm_db_pool_connections` — Database connection pool size
|
||||
- `leocrm_arq_jobs_total` — Total ARQ background jobs
|
||||
|
||||
### Structured Logging
|
||||
|
||||
LeoCRM uses `structlog` for structured JSON logging. All API requests are logged with:
|
||||
`timestamp`, `level`, `event`, `method`, `path`, `status`, `duration_ms`, `tenant_id`.
|
||||
|
||||
## Environment Profiles
|
||||
|
||||
| Profile | `ENVIRONMENT` | Use Case |
|
||||
|---|---|---|
|
||||
| Development | `development` | Local dev (auto-reload, verbose logging) |
|
||||
| Testing | `testing` | Test suite (separate test DB, minimal logging) |
|
||||
| Production | `production` | Docker/Coolify deployment (JSON logging, secure cookies) |
|
||||
|
||||
See [docs/admin-guide.md](docs/admin-guide.md#environment-profiles) for profile details.
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
pytest -v --tb=short
|
||||
|
||||
# Run specific test suites
|
||||
pytest tests/test_monitoring.py tests/test_performance.py tests/test_health.py -v
|
||||
|
||||
# Run with coverage
|
||||
pytest --cov=app --cov-report=term-missing
|
||||
|
||||
# Run specific test file
|
||||
pytest tests/test_auth.py -v
|
||||
|
||||
# Stop on first failure (for debugging)
|
||||
pytest -x
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `AUTH_SECRET` | ✅ | – | JWT signing secret (≥32 chars). Hard-fail if missing. |
|
||||
| `DATABASE_URL` | ❌ | `sqlite+aiosqlite:///./dev.db` | Async DB URL (aiosqlite or asyncpg) |
|
||||
| `JWT_ALGORITHM` | ❌ | `HS256` | JWT algorithm |
|
||||
| `JWT_EXPIRY_HOURS` | ❌ | `24` | Token lifetime |
|
||||
| `BCRYPT_ROUNDS` | ❌ | `12` | Password hashing cost |
|
||||
| `CORS_ORIGINS` | ❌ | `http://localhost:5500,http://localhost:8000` | Allowed origins (comma-separated, NO wildcards) |
|
||||
| `ENVIRONMENT` | ❌ | `development` | `development` or `production` |
|
||||
| `LOG_LEVEL` | ❌ | `INFO` | Python log level |
|
||||
See [.env.example](.env.example) for all variables and [docs/admin-guide.md](docs/admin-guide.md#environment-configuration) for detailed descriptions.
|
||||
|
||||
## Architecture Decisions (ADR)
|
||||
### Key Variables
|
||||
|
||||
- **JWT Library**: `python-jose[cryptography]==3.3.0` (pattern reuse from wochenplaner)
|
||||
- **Database**: SQLite (aiosqlite) for dev, PostgreSQL (asyncpg) for prod
|
||||
- **Auth**: Stateless JWT in localStorage, bcrypt password hashing (12 rounds)
|
||||
- **Security**: CORS whitelist (no wildcard), CSP middleware, no default admin user
|
||||
- **Async**: All routers/services/DB operations are async (SQLAlchemy 2.0 + aiosqlite)
|
||||
| Variable | Required | Description |
|
||||
|---|---|---|
|
||||
| `DATABASE_URL` | ✅ | PostgreSQL async connection URL |
|
||||
| `REDIS_URL` | ✅ | Redis connection URL |
|
||||
| `SECRET_KEY` | ✅ | Secret key for signing (≥32 chars in prod) |
|
||||
| `CORS_ORIGINS` | ✅ | Comma-separated allowed origins (no wildcards) |
|
||||
| `ENVIRONMENT` | ❌ | `development` \| `testing` \| `production` |
|
||||
| `STORAGE_PATH` | ❌ | File storage path (default: `/tmp`) |
|
||||
| `SMTP_HOST` | ❌ | SMTP server hostname |
|
||||
|
||||
See `/a0/.a0/02-architecture.md` Section 13 for full lockdown decisions.
|
||||
## Performance Testing
|
||||
|
||||
## Deployment
|
||||
```bash
|
||||
# Seed 200k contacts for performance testing
|
||||
python scripts/seed_perf_data.py --count 200000
|
||||
|
||||
See `/a0/.a0/03-task-graph.json` Phase 4d for Docker + Coolify setup (out of scope for Phase 4a).
|
||||
# Verify database indexes
|
||||
python scripts/check_indexes.py
|
||||
|
||||
# Test pagination performance
|
||||
# GET /api/v1/contacts?page=1&page_size=25 — should be <500ms with 200k records
|
||||
```
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
leocrm/
|
||||
├── app/
|
||||
│ ├── main.py # FastAPI entry point with logging middleware
|
||||
│ ├── config.py # Pydantic settings
|
||||
│ ├── core/
|
||||
│ │ ├── monitoring.py # Prometheus metrics + structured logging + health checks
|
||||
│ │ ├── db.py # Async database engine
|
||||
│ │ ├── middleware.py # CSRF middleware
|
||||
│ │ └── ...
|
||||
│ ├── routes/
|
||||
│ │ ├── health.py # Health endpoint
|
||||
│ │ ├── metrics.py # Prometheus metrics endpoint (admin-only)
|
||||
│ │ ├── contacts.py # Contact CRUD + streaming CSV export
|
||||
│ │ ├── companies.py # Company CRUD + streaming CSV export
|
||||
│ │ └── ...
|
||||
│ ├── models/ # SQLAlchemy models
|
||||
│ ├── schemas/ # Pydantic schemas
|
||||
│ ├── services/ # Business logic
|
||||
│ └── plugins/ # Plugin system
|
||||
├── scripts/
|
||||
│ ├── seed_perf_data.py # Performance test data seeding
|
||||
│ └── check_indexes.py # Database index verification
|
||||
├── tests/ # Test suite (pytest + pytest-asyncio)
|
||||
├── docs/
|
||||
│ ├── admin-guide.md # Admin guide (deploy, backup, restore, troubleshooting)
|
||||
│ └── api-overview.md # API endpoint summary
|
||||
├── alembic/ # Database migrations
|
||||
├── requirements.txt # Production dependencies
|
||||
├── requirements-dev.txt # Test/lint dependencies
|
||||
├── .env.example # Environment template
|
||||
├── docker-compose.yml # Docker Compose
|
||||
└── README.md # This file
|
||||
```
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Admin Guide](docs/admin-guide.md) — Deployment, backup, restore, env vars, troubleshooting
|
||||
- [API Overview](docs/api-overview.md) — Full endpoint reference
|
||||
- [Coolify Setup](COOLIFY_SETUP.md) — Coolify deployment instructions
|
||||
- [Swagger UI](http://localhost:8000/docs) — Interactive API docs (auto-generated)
|
||||
|
||||
## License
|
||||
|
||||
|
||||
Reference in New Issue
Block a user