sprint20-23: tests + documentation + guest access + infrastructure + migrations 0059
This commit is contained in:
Agent Zero
@@ -0,0 +1,408 @@
|
||||
# Infrastructure Guide
|
||||
|
||||
> **Version:** 1.0
|
||||
> **Date:** 2026-07-29
|
||||
> **Applies to:** System administrators and DevOps engineers
|
||||
|
||||
---
|
||||
|
||||
## 1. PgBouncer Setup
|
||||
|
||||
PgBouncer is a lightweight connection pooler for PostgreSQL. It reduces the overhead of establishing new database connections by reusing existing ones.
|
||||
|
||||
### Why PgBouncer?
|
||||
|
||||
- **Connection pooling** — Reduces PostgreSQL connection overhead
|
||||
- **Resource efficiency** — Handles thousands of client connections with minimal resources
|
||||
- **Transaction pooling** — Best for stateless applications like FastAPI
|
||||
- **Session pooling** — For stateful connections
|
||||
- **Statement pooling** — For specific use cases
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
# Debian/Ubuntu
|
||||
apt-get update && apt-get install -y pgbouncer
|
||||
|
||||
# Verify installation
|
||||
pgbouncer --version
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
Create `/etc/pgbouncer/pgbouncer.ini`:
|
||||
|
||||
```ini
|
||||
[databases]
|
||||
leocrm = host=localhost port=5432 dbname=leocrm
|
||||
leocrm_test = host=localhost port=5432 dbname=leocrm_test
|
||||
|
||||
[pgbouncer]
|
||||
listen_addr = 0.0.0.0
|
||||
listen_port = 6432
|
||||
unix_socket_dir = /var/run/pgbouncer
|
||||
|
||||
# Authentication
|
||||
# Use md5 for password-based auth
|
||||
# Use trust for local development
|
||||
auth_type = md5
|
||||
auth_file = /etc/pgbouncer/userlist.txt
|
||||
|
||||
# Pool settings
|
||||
pool_mode = transaction
|
||||
default_pool_size = 25
|
||||
max_client_conn = 200
|
||||
max_db_connections = 50
|
||||
|
||||
# Timeouts
|
||||
server_idle_timeout = 600
|
||||
server_lifetime = 3600
|
||||
client_idle_timeout = 1800
|
||||
query_timeout = 30
|
||||
|
||||
# Logging
|
||||
log_connections = 1
|
||||
log_disconnections = 1
|
||||
log_pooler_errors = 1
|
||||
stats_period = 60
|
||||
|
||||
# Security
|
||||
# Only allow connections from localhost and Docker network
|
||||
listen_backlog = 128
|
||||
```
|
||||
|
||||
### User List
|
||||
|
||||
Create `/etc/pgbouncer/userlist.txt`:
|
||||
|
||||
```
|
||||
"leocrm" "md5<password_hash>"
|
||||
"postgres" "md5<password_hash>"
|
||||
```
|
||||
|
||||
Generate the md5 hash:
|
||||
```bash
|
||||
# Format: md5 + md5(password + username)
|
||||
echo -n "md5" && echo -n "your_passwordleocrm" | md5sum | cut -d' ' -f1
|
||||
```
|
||||
|
||||
### Running PgBouncer
|
||||
|
||||
```bash
|
||||
# Start PgBouncer
|
||||
pgbouncer -d /etc/pgbouncer/pgbouncer.ini
|
||||
|
||||
# Check status
|
||||
pgbouncer -d /etc/pgbouncer/pgbouncer.ini -R
|
||||
|
||||
# Reload configuration
|
||||
kill -HUP $(cat /var/run/pgbouncer/pgbouncer.pid)
|
||||
|
||||
# Stop PgBouncer
|
||||
kill -INT $(cat /var/run/pgbouncer/pgbouncer.pid)
|
||||
```
|
||||
|
||||
### Docker Compose Integration
|
||||
|
||||
Add to `docker-compose.yml`:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
pgbouncer:
|
||||
image: bitnami/pgbouncer:latest
|
||||
container_name: leocrm-pgbouncer
|
||||
ports:
|
||||
- "6432:6432"
|
||||
environment:
|
||||
- POSTGRESQL_HOST=crm-postgres
|
||||
- POSTGRESQL_PORT=5432
|
||||
- POSTGRESQL_USERNAME=leocrm
|
||||
- POSTGRESQL_PASSWORD=${POSTGRES_PASSWORD}
|
||||
- POSTGRESQL_DATABASE=crm_db
|
||||
- PGBOUNCER_POOL_MODE=transaction
|
||||
- PGBOUNCER_DEFAULT_POOL_SIZE=25
|
||||
- PGBOUNCER_MAX_CLIENT_CONN=200
|
||||
depends_on:
|
||||
- crm-postgres
|
||||
restart: unless-stopped
|
||||
```
|
||||
|
||||
### Application Configuration
|
||||
|
||||
Update the database URL to use PgBouncer:
|
||||
|
||||
```python
|
||||
# Before (direct connection)
|
||||
DATABASE_URL = "postgresql+asyncpg://leocrm:password@crm-postgres:5432/crm_db"
|
||||
|
||||
# After (via PgBouncer)
|
||||
DATABASE_URL = "postgresql+asyncpg://leocrm:password@leocrm-pgbouncer:6432/crm_db"
|
||||
```
|
||||
|
||||
### Monitoring
|
||||
|
||||
```bash
|
||||
# Show pool statistics
|
||||
echo "SHOW STATS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer
|
||||
|
||||
# Show active pools
|
||||
echo "SHOW POOLS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer
|
||||
|
||||
# Show clients
|
||||
echo "SHOW CLIENTS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer
|
||||
|
||||
# Show servers
|
||||
echo "SHOW SERVERS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer
|
||||
```
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
| Issue | Cause | Solution |
|
||||
|-------|-------|----------|
|
||||
| Connection refused | PgBouncer not running | Check `pgbouncer -d` status |
|
||||
| Auth failed | Wrong password in userlist | Regenerate md5 hash |
|
||||
| Pool exhausted | Too many connections | Increase `default_pool_size` |
|
||||
| Slow queries | Query timeout | Check `query_timeout` setting |
|
||||
| Connection timeout | PostgreSQL overload | Check PostgreSQL connections |
|
||||
|
||||
---
|
||||
|
||||
## 2. Audit Log Partitioning
|
||||
|
||||
The `audit_log` table can grow very large over time. PostgreSQL table partitioning helps manage this by splitting the table into smaller, more manageable pieces.
|
||||
|
||||
### Why Partition?
|
||||
|
||||
- **Faster queries** — Queries only scan relevant partitions
|
||||
- **Easier maintenance** — Drop old partitions instead of DELETE
|
||||
- **Better vacuum** — Each partition is vacuumed independently
|
||||
- **Improved performance** — Smaller indexes per partition
|
||||
|
||||
### Partitioning Strategy
|
||||
|
||||
We use **monthly range partitioning** on the `created_at` column:
|
||||
|
||||
```sql
|
||||
-- Each partition covers one month
|
||||
-- Partition name: audit_log_YYYY_MM
|
||||
-- Example: audit_log_2026_01, audit_log_2026_02, ...
|
||||
```
|
||||
|
||||
### Creating the Partitioned Table
|
||||
|
||||
```sql
|
||||
-- Create the partitioned table
|
||||
CREATE TABLE audit_log_partitioned (
|
||||
id UUID NOT NULL DEFAULT gen_random_uuid(),
|
||||
tenant_id UUID,
|
||||
user_id UUID,
|
||||
action VARCHAR(100) NOT NULL,
|
||||
entity_type VARCHAR(50),
|
||||
entity_id UUID,
|
||||
changes JSONB,
|
||||
ip_address VARCHAR(45),
|
||||
user_agent TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
PRIMARY KEY (id, created_at)
|
||||
) PARTITION BY RANGE (created_at);
|
||||
|
||||
-- Create monthly partitions
|
||||
CREATE TABLE audit_log_2026_01 PARTITION OF audit_log_partitioned
|
||||
FOR VALUES FROM ('2026-01-01') TO ('2026-02-01');
|
||||
|
||||
CREATE TABLE audit_log_2026_02 PARTITION OF audit_log_partitioned
|
||||
FOR VALUES FROM ('2026-02-01') TO ('2026-03-01');
|
||||
|
||||
CREATE TABLE audit_log_2026_03 PARTITION OF audit_log_partitioned
|
||||
FOR VALUES FROM ('2026-03-01') TO ('2026-04-01');
|
||||
|
||||
-- Add indexes on each partition
|
||||
CREATE INDEX idx_audit_log_2026_01_tenant ON audit_log_2026_01 (tenant_id);
|
||||
CREATE INDEX idx_audit_log_2026_01_action ON audit_log_2026_01 (action);
|
||||
CREATE INDEX idx_audit_log_2026_01_entity ON audit_log_2026_01 (entity_type, entity_id);
|
||||
CREATE INDEX idx_audit_log_2026_01_created ON audit_log_2026_01 (created_at DESC);
|
||||
|
||||
CREATE INDEX idx_audit_log_2026_02_tenant ON audit_log_2026_02 (tenant_id);
|
||||
CREATE INDEX idx_audit_log_2026_02_action ON audit_log_2026_02 (action);
|
||||
CREATE INDEX idx_audit_log_2026_02_entity ON audit_log_2026_02 (entity_type, entity_id);
|
||||
CREATE INDEX idx_audit_log_2026_02_created ON audit_log_2026_02 (created_at DESC);
|
||||
|
||||
CREATE INDEX idx_audit_log_2026_03_tenant ON audit_log_2026_03 (tenant_id);
|
||||
CREATE INDEX idx_audit_log_2026_03_action ON audit_log_2026_03 (action);
|
||||
CREATE INDEX idx_audit_log_2026_03_entity ON audit_log_2026_03 (entity_type, entity_id);
|
||||
CREATE INDEX idx_audit_log_2026_03_created ON audit_log_2026_03 (created_at DESC);
|
||||
```
|
||||
|
||||
### Migrating Existing Data
|
||||
|
||||
```sql
|
||||
-- Step 1: Create the partitioned table
|
||||
-- (see script above)
|
||||
|
||||
-- Step 2: Insert existing data
|
||||
INSERT INTO audit_log_partitioned (
|
||||
id, tenant_id, user_id, action, entity_type,
|
||||
entity_id, changes, ip_address, user_agent, created_at
|
||||
)
|
||||
SELECT id, tenant_id, user_id, action, entity_type,
|
||||
entity_id, changes, ip_address, user_agent, created_at
|
||||
FROM audit_log;
|
||||
|
||||
-- Step 3: Verify data integrity
|
||||
SELECT COUNT(*) FROM audit_log_partitioned;
|
||||
SELECT COUNT(*) FROM audit_log;
|
||||
|
||||
-- Step 4: Rename tables
|
||||
ALTER TABLE audit_log RENAME TO audit_log_old;
|
||||
ALTER TABLE audit_log_partitioned RENAME TO audit_log;
|
||||
|
||||
-- Step 5: Update sequences and indexes
|
||||
-- (handled by the partitioned table definition)
|
||||
|
||||
-- Step 6: Drop old table after verification
|
||||
-- DROP TABLE audit_log_old;
|
||||
```
|
||||
|
||||
### Automating Partition Creation
|
||||
|
||||
Use the `setup_audit_partitioning.sql` script to automate partition management:
|
||||
|
||||
```bash
|
||||
# Run the setup script
|
||||
psql -h localhost -U leocrm -d crm_db -f scripts/setup_audit_partitioning.sql
|
||||
```
|
||||
|
||||
### Cron Job for Partition Maintenance
|
||||
|
||||
Add to crontab to create partitions automatically:
|
||||
|
||||
```bash
|
||||
# Run on the 1st of each month at 2 AM
|
||||
0 2 1 * * /usr/bin/psql -h localhost -U leocrm -d crm_db -c "SELECT create_monthly_audit_partition();"
|
||||
```
|
||||
|
||||
### Querying Partitioned Data
|
||||
|
||||
```sql
|
||||
-- Query a specific month (fast, only scans one partition)
|
||||
SELECT * FROM audit_log
|
||||
WHERE created_at >= '2026-01-01'
|
||||
AND created_at < '2026-02-01'
|
||||
AND tenant_id = '...';
|
||||
|
||||
-- Query across months (scans multiple partitions)
|
||||
SELECT * FROM audit_log
|
||||
WHERE created_at >= '2026-01-01'
|
||||
AND created_at < '2026-03-01'
|
||||
AND action = 'permission_grant';
|
||||
|
||||
-- Check which partitions will be scanned
|
||||
EXPLAIN SELECT * FROM audit_log
|
||||
WHERE created_at >= '2026-01-01'
|
||||
AND created_at < '2026-02-01';
|
||||
```
|
||||
|
||||
### Dropping Old Partitions
|
||||
|
||||
```sql
|
||||
-- Drop partitions older than retention period
|
||||
DROP TABLE IF EXISTS audit_log_2025_01;
|
||||
DROP TABLE IF EXISTS audit_log_2025_02;
|
||||
-- ...
|
||||
|
||||
-- Or use a function
|
||||
SELECT drop_old_audit_partitions(12); -- Keep last 12 months
|
||||
```
|
||||
|
||||
### Performance Considerations
|
||||
|
||||
- **Index each partition** — Don't rely on parent table indexes
|
||||
- **Use `created_at` in WHERE** — Always filter by date for partition pruning
|
||||
- **Monitor partition count** — Too many partitions can slow planning
|
||||
- **Archive old partitions** — Consider moving to cheaper storage
|
||||
- **Vacuum partitions** — Each partition needs independent vacuum
|
||||
|
||||
### Monitoring Partition Health
|
||||
|
||||
```sql
|
||||
-- Check partition sizes
|
||||
SELECT
|
||||
relname AS partition_name,
|
||||
pg_size_pretty(pg_total_relation_size(relid)) AS total_size
|
||||
FROM pg_catalog.pg_statio_user_tables
|
||||
WHERE relname LIKE 'audit_log_%'
|
||||
ORDER BY relname;
|
||||
|
||||
-- Check row counts per partition
|
||||
SELECT
|
||||
relname AS partition_name,
|
||||
n_live_tup AS row_count
|
||||
FROM pg_catalog.pg_stat_user_tables
|
||||
WHERE relname LIKE 'audit_log_%'
|
||||
ORDER BY relname;
|
||||
|
||||
-- List all partitions
|
||||
SELECT
|
||||
inhrelid::regclass AS partition_name
|
||||
FROM pg_catalog.pg_inherits
|
||||
WHERE inhparent = 'audit_log'::regclass
|
||||
ORDER BY partition_name;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Backup and Recovery
|
||||
|
||||
### Database Backup
|
||||
|
||||
```bash
|
||||
# Full backup
|
||||
pg_dump -h localhost -U leocrm -d crm_db -F c -f /backups/crm_db_$(date +%Y%m%d).dump
|
||||
|
||||
# Backup with compression
|
||||
pg_dump -h localhost -U leocrm -d crm_db -F c -Z 9 -f /backups/crm_db_$(date +%Y%m%d).dump.gz
|
||||
|
||||
# Backup specific schema only
|
||||
pg_dump -h localhost -U leocrm -d crm_db -n public -F c -f /backups/crm_db_schema_$(date +%Y%m%d).dump
|
||||
```
|
||||
|
||||
### Database Restore
|
||||
|
||||
```bash
|
||||
# Restore full backup
|
||||
pg_restore -h localhost -U leocrm -d crm_db -c /backups/crm_db_20260701.dump
|
||||
|
||||
# Restore with parallel workers (faster)
|
||||
pg_restore -h localhost -U leocrm -d crm_db -j 4 -c /backups/crm_db_20260701.dump
|
||||
```
|
||||
|
||||
### Automated Backup Script
|
||||
|
||||
See `scripts/backup.py` for the automated backup solution.
|
||||
|
||||
---
|
||||
|
||||
## 4. Monitoring and Alerts
|
||||
|
||||
### Key Metrics
|
||||
|
||||
| Metric | Target | Alert Threshold |
|
||||
|--------|--------|----------------|
|
||||
| Database connections | < 50 | > 80% of max |
|
||||
| Query response time | < 100ms | > 500ms |
|
||||
| Cache hit ratio | > 95% | < 90% |
|
||||
| Partition size | < 10GB | > 50GB |
|
||||
| PgBouncer pool usage | < 80% | > 90% |
|
||||
|
||||
### Health Checks
|
||||
|
||||
```bash
|
||||
# Check PgBouncer status
|
||||
echo "SHOW STATS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer | grep -E "total_|avg_"
|
||||
|
||||
# Check partition health
|
||||
psql -h localhost -U leocrm -d crm_db -c "SELECT count(*) FROM audit_log WHERE created_at < NOW() - INTERVAL '3 months';"
|
||||
|
||||
# Check database size
|
||||
psql -h localhost -U leocrm -d crm_db -c "SELECT pg_size_pretty(pg_database_size('crm_db'));"
|
||||
```
|
||||
Reference in New Issue
Block a user