# Infrastructure Guide

> **Version:** 1.0  
> **Date:** 2026-07-29  
> **Applies to:** System administrators and DevOps engineers

---

## 1. PgBouncer Setup

PgBouncer is a lightweight connection pooler for PostgreSQL. It reduces the overhead of establishing new database connections by reusing existing ones.

### Why PgBouncer?

- **Connection pooling** — Reduces PostgreSQL connection overhead
- **Resource efficiency** — Handles thousands of client connections with minimal resources
- **Transaction pooling** — Best for stateless applications like FastAPI
- **Session pooling** — For stateful connections
- **Statement pooling** — For specific use cases

### Installation

```bash
# Debian/Ubuntu
apt-get update && apt-get install -y pgbouncer

# Verify installation
pgbouncer --version
```

### Configuration

Create `/etc/pgbouncer/pgbouncer.ini`:

```ini
[databases]
leocrm = host=localhost port=5432 dbname=leocrm
leocrm_test = host=localhost port=5432 dbname=leocrm_test

[pgbouncer]
listen_addr = 0.0.0.0
listen_port = 6432
unix_socket_dir = /var/run/pgbouncer

# Authentication
# Use md5 for password-based auth
# Use trust for local development
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt

# Pool settings
pool_mode = transaction
default_pool_size = 25
max_client_conn = 200
max_db_connections = 50

# Timeouts
server_idle_timeout = 600
server_lifetime = 3600
client_idle_timeout = 1800
query_timeout = 30

# Logging
log_connections = 1
log_disconnections = 1
log_pooler_errors = 1
stats_period = 60

# Security
# Only allow connections from localhost and Docker network
listen_backlog = 128
```

### User List

Create `/etc/pgbouncer/userlist.txt`:

```
"leocrm" "md5<password_hash>"
"postgres" "md5<password_hash>"
```

Generate the md5 hash:
```bash
# Format: md5 + md5(password + username)
echo -n "md5" && echo -n "your_passwordleocrm" | md5sum | cut -d' ' -f1
```

### Running PgBouncer

```bash
# Start PgBouncer
pgbouncer -d /etc/pgbouncer/pgbouncer.ini

# Check status
pgbouncer -d /etc/pgbouncer/pgbouncer.ini -R

# Reload configuration
kill -HUP $(cat /var/run/pgbouncer/pgbouncer.pid)

# Stop PgBouncer
kill -INT $(cat /var/run/pgbouncer/pgbouncer.pid)
```

### Docker Compose Integration

Add to `docker-compose.yml`:

```yaml
services:
  pgbouncer:
    image: bitnami/pgbouncer:latest
    container_name: leocrm-pgbouncer
    ports:
      - "6432:6432"
    environment:
      - POSTGRESQL_HOST=crm-postgres
      - POSTGRESQL_PORT=5432
      - POSTGRESQL_USERNAME=leocrm
      - POSTGRESQL_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRESQL_DATABASE=crm_db
      - PGBOUNCER_POOL_MODE=transaction
      - PGBOUNCER_DEFAULT_POOL_SIZE=25
      - PGBOUNCER_MAX_CLIENT_CONN=200
    depends_on:
      - crm-postgres
    restart: unless-stopped
```

### Application Configuration

Update the database URL to use PgBouncer:

```python
# Before (direct connection)
DATABASE_URL = "postgresql+asyncpg://leocrm:password@crm-postgres:5432/crm_db"

# After (via PgBouncer)
DATABASE_URL = "postgresql+asyncpg://leocrm:password@leocrm-pgbouncer:6432/crm_db"
```

### Monitoring

```bash
# Show pool statistics
echo "SHOW STATS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer

# Show active pools
echo "SHOW POOLS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer

# Show clients
echo "SHOW CLIENTS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer

# Show servers
echo "SHOW SERVERS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer
```

### Troubleshooting

| Issue | Cause | Solution |
|-------|-------|----------|
| Connection refused | PgBouncer not running | Check `pgbouncer -d` status |
| Auth failed | Wrong password in userlist | Regenerate md5 hash |
| Pool exhausted | Too many connections | Increase `default_pool_size` |
| Slow queries | Query timeout | Check `query_timeout` setting |
| Connection timeout | PostgreSQL overload | Check PostgreSQL connections |

---

## 2. Audit Log Partitioning

The `audit_log` table can grow very large over time. PostgreSQL table partitioning helps manage this by splitting the table into smaller, more manageable pieces.

### Why Partition?

- **Faster queries** — Queries only scan relevant partitions
- **Easier maintenance** — Drop old partitions instead of DELETE
- **Better vacuum** — Each partition is vacuumed independently
- **Improved performance** — Smaller indexes per partition

### Partitioning Strategy

We use **monthly range partitioning** on the `created_at` column:

```sql
-- Each partition covers one month
-- Partition name: audit_log_YYYY_MM
-- Example: audit_log_2026_01, audit_log_2026_02, ...
```

### Creating the Partitioned Table

```sql
-- Create the partitioned table
CREATE TABLE audit_log_partitioned (
    id UUID NOT NULL DEFAULT gen_random_uuid(),
    tenant_id UUID,
    user_id UUID,
    action VARCHAR(100) NOT NULL,
    entity_type VARCHAR(50),
    entity_id UUID,
    changes JSONB,
    ip_address VARCHAR(45),
    user_agent TEXT,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    PRIMARY KEY (id, created_at)
) PARTITION BY RANGE (created_at);

-- Create monthly partitions
CREATE TABLE audit_log_2026_01 PARTITION OF audit_log_partitioned
    FOR VALUES FROM ('2026-01-01') TO ('2026-02-01');

CREATE TABLE audit_log_2026_02 PARTITION OF audit_log_partitioned
    FOR VALUES FROM ('2026-02-01') TO ('2026-03-01');

CREATE TABLE audit_log_2026_03 PARTITION OF audit_log_partitioned
    FOR VALUES FROM ('2026-03-01') TO ('2026-04-01');

-- Add indexes on each partition
CREATE INDEX idx_audit_log_2026_01_tenant ON audit_log_2026_01 (tenant_id);
CREATE INDEX idx_audit_log_2026_01_action ON audit_log_2026_01 (action);
CREATE INDEX idx_audit_log_2026_01_entity ON audit_log_2026_01 (entity_type, entity_id);
CREATE INDEX idx_audit_log_2026_01_created ON audit_log_2026_01 (created_at DESC);

CREATE INDEX idx_audit_log_2026_02_tenant ON audit_log_2026_02 (tenant_id);
CREATE INDEX idx_audit_log_2026_02_action ON audit_log_2026_02 (action);
CREATE INDEX idx_audit_log_2026_02_entity ON audit_log_2026_02 (entity_type, entity_id);
CREATE INDEX idx_audit_log_2026_02_created ON audit_log_2026_02 (created_at DESC);

CREATE INDEX idx_audit_log_2026_03_tenant ON audit_log_2026_03 (tenant_id);
CREATE INDEX idx_audit_log_2026_03_action ON audit_log_2026_03 (action);
CREATE INDEX idx_audit_log_2026_03_entity ON audit_log_2026_03 (entity_type, entity_id);
CREATE INDEX idx_audit_log_2026_03_created ON audit_log_2026_03 (created_at DESC);
```

### Migrating Existing Data

```sql
-- Step 1: Create the partitioned table
-- (see script above)

-- Step 2: Insert existing data
INSERT INTO audit_log_partitioned (
    id, tenant_id, user_id, action, entity_type,
    entity_id, changes, ip_address, user_agent, created_at
)
SELECT id, tenant_id, user_id, action, entity_type,
       entity_id, changes, ip_address, user_agent, created_at
FROM audit_log;

-- Step 3: Verify data integrity
SELECT COUNT(*) FROM audit_log_partitioned;
SELECT COUNT(*) FROM audit_log;

-- Step 4: Rename tables
ALTER TABLE audit_log RENAME TO audit_log_old;
ALTER TABLE audit_log_partitioned RENAME TO audit_log;

-- Step 5: Update sequences and indexes
-- (handled by the partitioned table definition)

-- Step 6: Drop old table after verification
-- DROP TABLE audit_log_old;
```

### Automating Partition Creation

Use the `setup_audit_partitioning.sql` script to automate partition management:

```bash
# Run the setup script
psql -h localhost -U leocrm -d crm_db -f scripts/setup_audit_partitioning.sql
```

### Cron Job for Partition Maintenance

Add to crontab to create partitions automatically:

```bash
# Run on the 1st of each month at 2 AM
0 2 1 * * /usr/bin/psql -h localhost -U leocrm -d crm_db -c "SELECT create_monthly_audit_partition();"
```

### Querying Partitioned Data

```sql
-- Query a specific month (fast, only scans one partition)
SELECT * FROM audit_log
WHERE created_at >= '2026-01-01'
  AND created_at < '2026-02-01'
  AND tenant_id = '...';

-- Query across months (scans multiple partitions)
SELECT * FROM audit_log
WHERE created_at >= '2026-01-01'
  AND created_at < '2026-03-01'
  AND action = 'permission_grant';

-- Check which partitions will be scanned
EXPLAIN SELECT * FROM audit_log
WHERE created_at >= '2026-01-01'
  AND created_at < '2026-02-01';
```

### Dropping Old Partitions

```sql
-- Drop partitions older than retention period
DROP TABLE IF EXISTS audit_log_2025_01;
DROP TABLE IF EXISTS audit_log_2025_02;
-- ...

-- Or use a function
SELECT drop_old_audit_partitions(12);  -- Keep last 12 months
```

### Performance Considerations

- **Index each partition** — Don't rely on parent table indexes
- **Use `created_at` in WHERE** — Always filter by date for partition pruning
- **Monitor partition count** — Too many partitions can slow planning
- **Archive old partitions** — Consider moving to cheaper storage
- **Vacuum partitions** — Each partition needs independent vacuum

### Monitoring Partition Health

```sql
-- Check partition sizes
SELECT
    relname AS partition_name,
    pg_size_pretty(pg_total_relation_size(relid)) AS total_size
FROM pg_catalog.pg_statio_user_tables
WHERE relname LIKE 'audit_log_%'
ORDER BY relname;

-- Check row counts per partition
SELECT
    relname AS partition_name,
    n_live_tup AS row_count
FROM pg_catalog.pg_stat_user_tables
WHERE relname LIKE 'audit_log_%'
ORDER BY relname;

-- List all partitions
SELECT
    inhrelid::regclass AS partition_name
FROM pg_catalog.pg_inherits
WHERE inhparent = 'audit_log'::regclass
ORDER BY partition_name;
```

---

## 3. Backup and Recovery

### Database Backup

```bash
# Full backup
pg_dump -h localhost -U leocrm -d crm_db -F c -f /backups/crm_db_$(date +%Y%m%d).dump

# Backup with compression
pg_dump -h localhost -U leocrm -d crm_db -F c -Z 9 -f /backups/crm_db_$(date +%Y%m%d).dump.gz

# Backup specific schema only
pg_dump -h localhost -U leocrm -d crm_db -n public -F c -f /backups/crm_db_schema_$(date +%Y%m%d).dump
```

### Database Restore

```bash
# Restore full backup
pg_restore -h localhost -U leocrm -d crm_db -c /backups/crm_db_20260701.dump

# Restore with parallel workers (faster)
pg_restore -h localhost -U leocrm -d crm_db -j 4 -c /backups/crm_db_20260701.dump
```

### Automated Backup Script

See `scripts/backup.py` for the automated backup solution.

---

## 4. Monitoring and Alerts

### Key Metrics

| Metric | Target | Alert Threshold |
|--------|--------|----------------|
| Database connections | < 50 | > 80% of max |
| Query response time | < 100ms | > 500ms |
| Cache hit ratio | > 95% | < 90% |
| Partition size | < 10GB | > 50GB |
| PgBouncer pool usage | < 80% | > 90% |

### Health Checks

```bash
# Check PgBouncer status
echo "SHOW STATS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer | grep -E "total_|avg_"

# Check partition health
psql -h localhost -U leocrm -d crm_db -c "SELECT count(*) FROM audit_log WHERE created_at < NOW() - INTERVAL '3 months';"

# Check database size
psql -h localhost -U leocrm -d crm_db -c "SELECT pg_size_pretty(pg_database_size('crm_db'));"
```