leocrm/README.md

# LeoCRM v1.0

> Plugin-basierte KI und Business-Plattform mit 25 Plugins (CRM, Mail, DMS, Chat, AI-Agenten, Workflows, Knowledge, Search, Self-Improvement, Compliance). FastAPI Backend + React/TypeScript Frontend. Deployiert über Coolify auf Hetzner VPS.
> Stack: FastAPI + SQLAlchemy (async) + PostgreSQL 16 (pgvector) + Redis 7 + React 18 + TypeScript + Vite + TanStack Query + Zustand + Tailwind + Docker + Coolify

## Features

### Core Platform
- **Multi-Tenant** — Tenant-Isolation via ORM Auto-Filter + Row Level Security (RLS)
- **Plugin System** — 25 Built-in Plugins, Manifest-basiert, aktivierbar/deaktivierbar
- **Permission System** — ABAC/RBAC mit feingranularen Permissions
- **Audit Log** — Vollständige Audit-Trail, CSV/JSON Export, 365 Tage Retention
- **Entity History** — Undo/Restore für alle Entitäten
- **Soft Delete** — `deleted_at` auf allen Entitäten, Hard-Delete mit `?gdpr=true`
- **Unified Search** — Hybrid-Suche (PostgreSQL FTS + pgvector), KI Query-Understanding
- **System Dashboard** — Admin-only Monitoring (DB, Redis, Worker, Errors, LLM Costs)
- **Backup Automation** — ARQ-gesteuert, einstellbar in Settings, Backup-History
- **Trash Cleanup** — Automatische endgültige Löschung nach 90 Tagen

### 25 Plugins

| # | Plugin | Beschreibung |
|---|--------|-------------|
| 1 | **contacts** | Kontakt-Verwaltung (Personen, Firmen, Ordner, Custom Fields) |
| 2 | **mail** | IMAP/SMTP E-Mail-Integration, PGP, Filter-Regeln, Vacation Responder |
| 3 | **dms** | Document Management System, File Upload, Preview, Sharing, Permissions |
| 4 | **calendar** | Kalender, Termine, Ressourcen-Buchung, ICS Import/Export, Kanban |
| 5 | **tasks** | Unified Task System, Subtasks, Goals, polymorphe Zuweisung |
| 6 | **kommunikation** | Unified Messaging, Chat, Mini-Apps, WebSocket-basiert |
| 7 | **automation** | Automation Builder, Trigger, Agent Runner, Cron-Scheduler |
| 8 | **ai_assistant** | AI Chat Sessions, Provider, Models, Presets, Tools |
| 9 | **ai_proactive** | Proactive AI, Suggestions, SSE Streaming, Settings |
| 10 | **ai_ui_control** | AI-driven UI Control via WebSocket |
| 11 | **agent_memory** | Agent Memory Plugin, eigene Routes |
| 12 | **unified_search** | Hybrid-Suche, Embeddings, RRF Rank Fusion, Facets |
| 13 | **graph_rag** | GraphRAG, Knowledge Graph, Relationship Extraction |
| 14 | **wiki** | Wiki Plugin, Article Versioning, Categories, Entity Links |
| 15 | **report_generator** | Report Templates, Generation, Download |
| 16 | **entity_links** | Entity Linking, File-Entity Connections |
| 17 | **tags** | Tag Management, Bulk-Assign, Entity-Tag Queries |
| 18 | **permissions** | File-level Permissions, Share Links |
| 19 | **mcp_server** | MCP Server, Tool Definitions für AI Agents |
| 20 | **mcp_client** | MCP Client für externe Tool-Integration |
| 21 | **marketplace** | Marketplace Listings |
| 22 | **system_notif** | System Notifications, Alerting via Communication-System |
| 23 | **forgejo_error_reporter** | Forgejo Error Reporting |
| 24 | **knowledge** | LLM-based Knowledge Extraction, Ask-Knowledge, Review Queue |
| 25 | **self_improvement** | Controlled Self-Improvement Loop (Signals, Patterns, Proposals, Impact) |

### AI & Automation
- **Agent System** — ReAct-Loop, Tool-Calls, Skills, Approvals, Monitoring, SSE Streaming
- **Workflow Engine** — 14 Step-Types, Durable Runs, Retry, Idempotency, SSRF-Schutz
- **Decision Guard** — Automated-Decision Guard für High-Risk Actions
- **Approval System** — Human Approval für Agent Actions und Workflow Steps
- **LLM Client** — Zentraler LLM Client, Cost-Tracking, Multi-Provider

## Quick Start (Development)

### 1. Clone and Setup

```bash
git clone <repo-url> leocrm
cd leocrm

# Create virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt -r requirements-dev.txt
```

### 2. Configure Environment

```bash
cp .env.example .env

# Generate a secure SECRET_KEY (min 32 chars)
python3 -c "import secrets; print('SECRET_KEY=' + secrets.token_urlsafe(48))" >> .env

# Edit .env and set DATABASE_URL, REDIS_URL, SECRET_KEY
nano .env
```

### 3. Initialize Database

```bash
# Apply migrations
alembic upgrade head
```

### 4. Run Server

```bash
# Development with auto-reload
uvicorn app.main:app --reload --port 8000

# Start ARQ worker (for background jobs)
arq app.core.jobs.WorkerSettings
```

Open:
- API: http://localhost:8000
- Swagger UI: http://localhost:8000/docs
- ReDoc: http://localhost:8000/redoc
- Health: http://localhost:8000/api/v1/health
- Metrics: http://localhost:8000/api/v1/metrics (admin-only)

## Production Setup

### Docker Compose

```bash
cp .env.docker.example .env.docker
# Edit .env.docker — set DB_PASSWORD, REDIS_PASSWORD, SECRET_KEY, APP_DOMAIN
# Set ENVIRONMENT=production, SESSION_COOKIE_SECURE=true
docker compose --env-file .env.docker up --build -d
```

### Manual (without Docker)

```bash
pip install -r requirements.txt
alembic upgrade head

# Start API server (2 workers)
uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 2

# Start ARQ worker (separate process)
arq app.core.jobs.WorkerSettings
```

See [docs/admin-guide.md](docs/admin-guide.md) for detailed deployment, backup, and troubleshooting instructions.

## API

### Key Endpoints

| Endpoint | Method | Auth | Description |
|---|---|---|---|
| `/health/live` | GET | No | Liveness probe |
| `/health/ready` | GET | No | Readiness probe (DB, Redis, storage, worker) |
| `/api/v1/health` | GET | No | Full health check (DB, Redis, storage, worker) |
| `/api/v1/metrics` | GET | Admin | Prometheus metrics (text/plain) |
| `/api/v1/system/dashboard` | GET | Admin | System dashboard (DB, Redis, worker, errors, LLM costs) |
| `/api/v1/system/alerts` | GET | Admin | Active system alerts |
| `/api/v1/auth/login` | POST | No | Login |
| `/api/v1/contacts` | GET | Yes | List contacts (paginated, max page_size=100) |
| `/api/v1/contacts/export` | GET | Yes | Stream contacts as CSV |
| `/api/v1/companies` | GET | Yes | List companies (paginated, max page_size=100) |
| `/api/v1/companies/export` | GET | Yes | Stream companies as CSV |
| `/api/v1/search` | POST | Yes | Hybrid search (FTS + pgvector) |
| `/api/v1/audit-log` | GET | Admin | Query audit log entries |
| `/api/v1/audit-log/export` | GET | Admin | Export audit log (CSV/JSON) |
| `/api/v1/system-settings/backup-config` | GET/PUT | Admin | Backup configuration |
| `/api/v1/system-settings/backup-now` | POST | Admin | Trigger immediate backup |
| `/api/v1/system-settings/backup-history` | GET | Admin | Backup history (last 10) |

### Pagination

All list endpoints support pagination with `page` and `page_size` parameters.
`page_size` is capped at **100** — values >100 return HTTP 422.

### CSV Export

Contacts and companies support streaming CSV export via `/export?format=csv`.
Uses `StreamingResponse` — does not buffer the entire file in memory.

### Swagger UI

Interactive API documentation: http://localhost:8000/docs

See [docs/api-documentation.md](docs/api-documentation.md) for the full endpoint reference.

## Monitoring

### Health Checks

```bash
# Liveness
curl http://localhost:8000/health/live
# → {"status":"alive"}

# Readiness
curl http://localhost:8000/health/ready
# → {"status":"ready","checks":{"database":"ok","redis":"ok","storage":"ok"}}

# Full health
curl http://localhost:8000/api/v1/health
# → {"status":"healthy","version":"1.0.0","checks":{...}}
```

### Prometheus Metrics

```bash
# Requires admin authentication
curl -b "leocrm_session=<session>" http://localhost:8000/api/v1/metrics
```

Available metrics:
- `leocrm_http_requests_total` — Total HTTP requests
- `leocrm_http_request_duration_seconds` — Request duration histogram
- `leocrm_db_pool_connections` — Database connection pool size
- `leocrm_arq_jobs_total` — Total ARQ background jobs

### System Dashboard

Admin-only dashboard at `/system-dashboard` in the WebUI. Shows:
- System Health, DB Stats, Redis Stats, Worker Queue
- API Stats (total requests, error rate, avg response time)
- Plugin Stats (discovered, active)
- Storage Stats (disk usage, file count)
- Alert Feed (system messages from Communication-System)

### Structured Logging

LeoCRM uses `structlog` for structured JSON logging. All API requests are logged with:
`timestamp`, `level`, `event`, `method`, `path`, `status`, `duration_ms`, `tenant_id`.

## Environment Profiles

| Profile | `ENVIRONMENT` | Use Case |
|---|---|---|
| Development | `development` | Local dev (auto-reload, verbose logging) |
| Testing | `testing` | Test suite (separate test DB, minimal logging) |
| Production | `production` | Docker/Coolify deployment (JSON logging, secure cookies) |

See [docs/admin-guide.md](docs/admin-guide.md#environment-profiles) for profile details.

## Testing

```bash
# Run all tests
pytest -v --tb=short

# Run specific test suites
pytest tests/test_monitoring.py tests/test_performance.py tests/test_health.py -v

# Run with coverage
pytest --cov=app --cov-report=term-missing
```

## Environment Variables

See [.env.example](.env.example) for all variables and [docs/admin-guide.md](docs/admin-guide.md#environment-configuration) for detailed descriptions.

### Key Variables

| Variable | Required | Description |
|---|---|---|
| `DATABASE_URL` | ✅ | PostgreSQL async connection URL |
| `REDIS_URL` | ✅ | Redis connection URL |
| `SECRET_KEY` | ✅ | Secret key for signing (≥32 chars in prod) |
| `CORS_ORIGINS` | ✅ | Comma-separated allowed origins (no wildcards) |
| `ENVIRONMENT` | ❌ | `development` \| `testing` \| `production` |
| `STORAGE_PATH` | ❌ | File storage path (default: `/tmp`) |
| `SMTP_HOST` | ❌ | SMTP server hostname |

## Performance Testing

```bash
# Seed 200k contacts for performance testing
python scripts/seed_perf_data.py --count 200000

# Verify database indexes
python scripts/check_indexes.py

# Test pagination performance
# GET /api/v1/contacts?page=1&page_size=25 — should be <500ms with 200k records
```

## Project Structure

```
leocrm/
├── app/
│   ├── main.py               # FastAPI entry point with logging middleware
│   ├── config.py             # Pydantic settings
│   ├── deps.py               # FastAPI dependencies (auth, permissions)
│   ├── core/
│   │   ├── monitoring.py      # Prometheus metrics + structured logging + health checks
│   │   ├── db.py             # Async database engine
│   │   ├── middleware.py     # CSRF middleware
│   │   ├── worker.py        # ARQ worker settings
│   │   ├── backup_job.py    # Automated backup job
│   │   ├── notifications.py  # System notification dispatch
│   │   └── ...
│   ├── routes/
│   │   ├── health.py         # Health endpoints
│   │   ├── metrics.py        # Prometheus metrics endpoint (admin-only)
│   │   ├── system_dashboard.py # System dashboard (admin-only)
│   │   ├── system_settings.py # System settings + backup config
│   │   ├── audit.py          # Audit log (list, export, retention)
│   │   ├── contacts.py       # Contact CRUD + streaming CSV export
│   │   ├── companies.py      # Company CRUD + streaming CSV export
│   │   ├── workflows.py      # Workflow engine routes
│   │   └── ...
│   ├── models/               # SQLAlchemy models
│   ├── schemas/              # Pydantic schemas
│   ├── services/             # Business logic
│   ├── plugins/              # Plugin system (registry, manifest, base)
│   │   └── builtins/         # 25 built-in plugins
│   ├── workflows/            # Workflow engine
│   └── ai/                   # AI modules
├── scripts/
│   ├── fast-deploy.sh        # Frontend-only / full deploy
│   ├── deploy.py             # Coolify API deployment
│   ├── backup.py             # Backup script (pg_dump + files)
│   ├── restore.py            # Restore script
│   ├── seed_perf_data.py     # Performance test data seeding
│   └── check_indexes.py      # Database index verification
├── tests/                    # Test suite (pytest + pytest-asyncio)
├── docs/
│   ├── admin-guide.md       # Admin guide (deploy, backup, restore, troubleshooting)
│   ├── api-documentation.md  # Full API endpoint reference
│   ├── monitoring.md        # Monitoring & health checks
│   ├── infrastructure.md    # Infrastructure guide
│   ├── deploy-guide.md      # Deploy guide (fast-deploy, Coolify, server info)
│   └── ...
├── alembic/                  # Database migrations (130+ files)
├── frontend/                 # React + TypeScript + Vite + Tailwind
│   └── src/pages/            # SystemDashboard, Contacts, Mail, DMS, Calendar, etc.
├── requirements.txt          # Production dependencies
├── requirements-dev.txt     # Test/lint dependencies
├── .env.example              # Environment template
├── docker-compose.yaml       # Docker Compose (postgres, redis, crm_app, crm_worker)
├── Dockerfile                # Multi-stage build (frontend → builder → runtime)
├── prestart.sh               # Container entrypoint (migrations, seed, uvicorn)
├── worker.sh                 # ARQ worker entrypoint
├── healthcheck.sh            # Container healthcheck
└── README.md                # This file
```

## Documentation

- [Admin Guide](docs/admin-guide.md) — Deployment, backup, restore, env vars, troubleshooting
- [API Documentation](docs/api-documentation.md) — Full endpoint reference (300+ endpoints)
- [Monitoring](docs/monitoring.md) — Health checks, metrics, system dashboard, alerting
- [Infrastructure](docs/infrastructure.md) — Docker, PgBouncer, audit partitioning, backup
- [Deploy Guide](docs/deploy-guide.md) — Fast-deploy, Coolify API, server info
- [Plugin Development](docs/plugin-development-guide.md) — Plugin development guide
- [Security Kernel](docs/security_kernel.md) — ABAC, RLS, session security
- [Permissions](docs/permissions.md) — Permission system documentation
- [Test Strategy](docs/test-strategy.md) — Test conventions and constraints
- [UI Design Guidelines](docs/ui-design-guidelines.md) — UI design rules
- [Swagger UI](http://localhost:8000/docs) — Interactive API docs (auto-generated)

## License

Internal project – proprietary.