sprint20-23: tests + documentation + guest access + infrastructure + migrations 0059

This commit is contained in:
Agent Zero
2026-07-29 02:53:37 +02:00
parent ddf73ee42e
commit 24690fb674
20 changed files with 3898 additions and 2 deletions
+408
View File
@@ -0,0 +1,408 @@
# Infrastructure Guide
> **Version:** 1.0
> **Date:** 2026-07-29
> **Applies to:** System administrators and DevOps engineers
---
## 1. PgBouncer Setup
PgBouncer is a lightweight connection pooler for PostgreSQL. It reduces the overhead of establishing new database connections by reusing existing ones.
### Why PgBouncer?
- **Connection pooling** — Reduces PostgreSQL connection overhead
- **Resource efficiency** — Handles thousands of client connections with minimal resources
- **Transaction pooling** — Best for stateless applications like FastAPI
- **Session pooling** — For stateful connections
- **Statement pooling** — For specific use cases
### Installation
```bash
# Debian/Ubuntu
apt-get update && apt-get install -y pgbouncer
# Verify installation
pgbouncer --version
```
### Configuration
Create `/etc/pgbouncer/pgbouncer.ini`:
```ini
[databases]
leocrm = host=localhost port=5432 dbname=leocrm
leocrm_test = host=localhost port=5432 dbname=leocrm_test
[pgbouncer]
listen_addr = 0.0.0.0
listen_port = 6432
unix_socket_dir = /var/run/pgbouncer
# Authentication
# Use md5 for password-based auth
# Use trust for local development
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt
# Pool settings
pool_mode = transaction
default_pool_size = 25
max_client_conn = 200
max_db_connections = 50
# Timeouts
server_idle_timeout = 600
server_lifetime = 3600
client_idle_timeout = 1800
query_timeout = 30
# Logging
log_connections = 1
log_disconnections = 1
log_pooler_errors = 1
stats_period = 60
# Security
# Only allow connections from localhost and Docker network
listen_backlog = 128
```
### User List
Create `/etc/pgbouncer/userlist.txt`:
```
"leocrm" "md5<password_hash>"
"postgres" "md5<password_hash>"
```
Generate the md5 hash:
```bash
# Format: md5 + md5(password + username)
echo -n "md5" && echo -n "your_passwordleocrm" | md5sum | cut -d' ' -f1
```
### Running PgBouncer
```bash
# Start PgBouncer
pgbouncer -d /etc/pgbouncer/pgbouncer.ini
# Check status
pgbouncer -d /etc/pgbouncer/pgbouncer.ini -R
# Reload configuration
kill -HUP $(cat /var/run/pgbouncer/pgbouncer.pid)
# Stop PgBouncer
kill -INT $(cat /var/run/pgbouncer/pgbouncer.pid)
```
### Docker Compose Integration
Add to `docker-compose.yml`:
```yaml
services:
pgbouncer:
image: bitnami/pgbouncer:latest
container_name: leocrm-pgbouncer
ports:
- "6432:6432"
environment:
- POSTGRESQL_HOST=crm-postgres
- POSTGRESQL_PORT=5432
- POSTGRESQL_USERNAME=leocrm
- POSTGRESQL_PASSWORD=${POSTGRES_PASSWORD}
- POSTGRESQL_DATABASE=crm_db
- PGBOUNCER_POOL_MODE=transaction
- PGBOUNCER_DEFAULT_POOL_SIZE=25
- PGBOUNCER_MAX_CLIENT_CONN=200
depends_on:
- crm-postgres
restart: unless-stopped
```
### Application Configuration
Update the database URL to use PgBouncer:
```python
# Before (direct connection)
DATABASE_URL = "postgresql+asyncpg://leocrm:password@crm-postgres:5432/crm_db"
# After (via PgBouncer)
DATABASE_URL = "postgresql+asyncpg://leocrm:password@leocrm-pgbouncer:6432/crm_db"
```
### Monitoring
```bash
# Show pool statistics
echo "SHOW STATS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer
# Show active pools
echo "SHOW POOLS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer
# Show clients
echo "SHOW CLIENTS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer
# Show servers
echo "SHOW SERVERS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer
```
### Troubleshooting
| Issue | Cause | Solution |
|-------|-------|----------|
| Connection refused | PgBouncer not running | Check `pgbouncer -d` status |
| Auth failed | Wrong password in userlist | Regenerate md5 hash |
| Pool exhausted | Too many connections | Increase `default_pool_size` |
| Slow queries | Query timeout | Check `query_timeout` setting |
| Connection timeout | PostgreSQL overload | Check PostgreSQL connections |
---
## 2. Audit Log Partitioning
The `audit_log` table can grow very large over time. PostgreSQL table partitioning helps manage this by splitting the table into smaller, more manageable pieces.
### Why Partition?
- **Faster queries** — Queries only scan relevant partitions
- **Easier maintenance** — Drop old partitions instead of DELETE
- **Better vacuum** — Each partition is vacuumed independently
- **Improved performance** — Smaller indexes per partition
### Partitioning Strategy
We use **monthly range partitioning** on the `created_at` column:
```sql
-- Each partition covers one month
-- Partition name: audit_log_YYYY_MM
-- Example: audit_log_2026_01, audit_log_2026_02, ...
```
### Creating the Partitioned Table
```sql
-- Create the partitioned table
CREATE TABLE audit_log_partitioned (
id UUID NOT NULL DEFAULT gen_random_uuid(),
tenant_id UUID,
user_id UUID,
action VARCHAR(100) NOT NULL,
entity_type VARCHAR(50),
entity_id UUID,
changes JSONB,
ip_address VARCHAR(45),
user_agent TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
PRIMARY KEY (id, created_at)
) PARTITION BY RANGE (created_at);
-- Create monthly partitions
CREATE TABLE audit_log_2026_01 PARTITION OF audit_log_partitioned
FOR VALUES FROM ('2026-01-01') TO ('2026-02-01');
CREATE TABLE audit_log_2026_02 PARTITION OF audit_log_partitioned
FOR VALUES FROM ('2026-02-01') TO ('2026-03-01');
CREATE TABLE audit_log_2026_03 PARTITION OF audit_log_partitioned
FOR VALUES FROM ('2026-03-01') TO ('2026-04-01');
-- Add indexes on each partition
CREATE INDEX idx_audit_log_2026_01_tenant ON audit_log_2026_01 (tenant_id);
CREATE INDEX idx_audit_log_2026_01_action ON audit_log_2026_01 (action);
CREATE INDEX idx_audit_log_2026_01_entity ON audit_log_2026_01 (entity_type, entity_id);
CREATE INDEX idx_audit_log_2026_01_created ON audit_log_2026_01 (created_at DESC);
CREATE INDEX idx_audit_log_2026_02_tenant ON audit_log_2026_02 (tenant_id);
CREATE INDEX idx_audit_log_2026_02_action ON audit_log_2026_02 (action);
CREATE INDEX idx_audit_log_2026_02_entity ON audit_log_2026_02 (entity_type, entity_id);
CREATE INDEX idx_audit_log_2026_02_created ON audit_log_2026_02 (created_at DESC);
CREATE INDEX idx_audit_log_2026_03_tenant ON audit_log_2026_03 (tenant_id);
CREATE INDEX idx_audit_log_2026_03_action ON audit_log_2026_03 (action);
CREATE INDEX idx_audit_log_2026_03_entity ON audit_log_2026_03 (entity_type, entity_id);
CREATE INDEX idx_audit_log_2026_03_created ON audit_log_2026_03 (created_at DESC);
```
### Migrating Existing Data
```sql
-- Step 1: Create the partitioned table
-- (see script above)
-- Step 2: Insert existing data
INSERT INTO audit_log_partitioned (
id, tenant_id, user_id, action, entity_type,
entity_id, changes, ip_address, user_agent, created_at
)
SELECT id, tenant_id, user_id, action, entity_type,
entity_id, changes, ip_address, user_agent, created_at
FROM audit_log;
-- Step 3: Verify data integrity
SELECT COUNT(*) FROM audit_log_partitioned;
SELECT COUNT(*) FROM audit_log;
-- Step 4: Rename tables
ALTER TABLE audit_log RENAME TO audit_log_old;
ALTER TABLE audit_log_partitioned RENAME TO audit_log;
-- Step 5: Update sequences and indexes
-- (handled by the partitioned table definition)
-- Step 6: Drop old table after verification
-- DROP TABLE audit_log_old;
```
### Automating Partition Creation
Use the `setup_audit_partitioning.sql` script to automate partition management:
```bash
# Run the setup script
psql -h localhost -U leocrm -d crm_db -f scripts/setup_audit_partitioning.sql
```
### Cron Job for Partition Maintenance
Add to crontab to create partitions automatically:
```bash
# Run on the 1st of each month at 2 AM
0 2 1 * * /usr/bin/psql -h localhost -U leocrm -d crm_db -c "SELECT create_monthly_audit_partition();"
```
### Querying Partitioned Data
```sql
-- Query a specific month (fast, only scans one partition)
SELECT * FROM audit_log
WHERE created_at >= '2026-01-01'
AND created_at < '2026-02-01'
AND tenant_id = '...';
-- Query across months (scans multiple partitions)
SELECT * FROM audit_log
WHERE created_at >= '2026-01-01'
AND created_at < '2026-03-01'
AND action = 'permission_grant';
-- Check which partitions will be scanned
EXPLAIN SELECT * FROM audit_log
WHERE created_at >= '2026-01-01'
AND created_at < '2026-02-01';
```
### Dropping Old Partitions
```sql
-- Drop partitions older than retention period
DROP TABLE IF EXISTS audit_log_2025_01;
DROP TABLE IF EXISTS audit_log_2025_02;
-- ...
-- Or use a function
SELECT drop_old_audit_partitions(12); -- Keep last 12 months
```
### Performance Considerations
- **Index each partition** — Don't rely on parent table indexes
- **Use `created_at` in WHERE** — Always filter by date for partition pruning
- **Monitor partition count** — Too many partitions can slow planning
- **Archive old partitions** — Consider moving to cheaper storage
- **Vacuum partitions** — Each partition needs independent vacuum
### Monitoring Partition Health
```sql
-- Check partition sizes
SELECT
relname AS partition_name,
pg_size_pretty(pg_total_relation_size(relid)) AS total_size
FROM pg_catalog.pg_statio_user_tables
WHERE relname LIKE 'audit_log_%'
ORDER BY relname;
-- Check row counts per partition
SELECT
relname AS partition_name,
n_live_tup AS row_count
FROM pg_catalog.pg_stat_user_tables
WHERE relname LIKE 'audit_log_%'
ORDER BY relname;
-- List all partitions
SELECT
inhrelid::regclass AS partition_name
FROM pg_catalog.pg_inherits
WHERE inhparent = 'audit_log'::regclass
ORDER BY partition_name;
```
---
## 3. Backup and Recovery
### Database Backup
```bash
# Full backup
pg_dump -h localhost -U leocrm -d crm_db -F c -f /backups/crm_db_$(date +%Y%m%d).dump
# Backup with compression
pg_dump -h localhost -U leocrm -d crm_db -F c -Z 9 -f /backups/crm_db_$(date +%Y%m%d).dump.gz
# Backup specific schema only
pg_dump -h localhost -U leocrm -d crm_db -n public -F c -f /backups/crm_db_schema_$(date +%Y%m%d).dump
```
### Database Restore
```bash
# Restore full backup
pg_restore -h localhost -U leocrm -d crm_db -c /backups/crm_db_20260701.dump
# Restore with parallel workers (faster)
pg_restore -h localhost -U leocrm -d crm_db -j 4 -c /backups/crm_db_20260701.dump
```
### Automated Backup Script
See `scripts/backup.py` for the automated backup solution.
---
## 4. Monitoring and Alerts
### Key Metrics
| Metric | Target | Alert Threshold |
|--------|--------|----------------|
| Database connections | < 50 | > 80% of max |
| Query response time | < 100ms | > 500ms |
| Cache hit ratio | > 95% | < 90% |
| Partition size | < 10GB | > 50GB |
| PgBouncer pool usage | < 80% | > 90% |
### Health Checks
```bash
# Check PgBouncer status
echo "SHOW STATS;" | psql -h localhost -p 6432 -U leocrm -d pgbouncer | grep -E "total_|avg_"
# Check partition health
psql -h localhost -U leocrm -d crm_db -c "SELECT count(*) FROM audit_log WHERE created_at < NOW() - INTERVAL '3 months';"
# Check database size
psql -h localhost -U leocrm -d crm_db -c "SELECT pg_size_pretty(pg_database_size('crm_db'));"
```
+331
View File
@@ -0,0 +1,331 @@
# LeoCRM Permission System
> **Version:** 1.0
> **Date:** 2026-07-29
> **Applies to:** All developers and system administrators
---
## 1. Architecture Overview
The LeoCRM permission system is a multi-layered access control framework that combines:
- **Row-Level Ownership** — Each entity can have an `owner_id` (user who owns it)
- **Entity Permissions (ACL)** — Explicit permission entries for users, groups, or roles
- **ABAC Policies** — Attribute-based policies for fine-grained access control
- **Role-Based Access Control (RBAC)** — Module-level permissions via user roles
- **System Admin Override** — System administrators see everything
### Permission Resolution Order
When checking access to an entity, the system resolves in this order (highest wins):
1. **System Admin**`delete` (full access to everything)
2. **Owner**`owner` (from `owner_id` on the entity)
3. **Direct User Permission** → explicit ACL entry for the user
4. **Group Permission** → ACL entry for a group the user belongs to
5. **Role Permission** → ACL entry for the user's role
6. **Tenant-Owned**`read` (if `owner_id IS NULL`, visible to all with module permission)
7. **No Access**`none`
### Permission Levels
| Level | Value | Description |
|-------|-------|-------------|
| `none` | 0 | Explicit deny (overrides allow) |
| `read` | 1 | View the entity |
| `write` | 2 | Read + edit entity fields |
| `admin` | 3 | Write + delete + manage permissions |
| `delete` | 4 | Admin + transfer ownership |
| `owner` | 5 | Full control (automatic for owner) |
---
## 2. Data Model
### EntityPermission
Stored in the `entity_permissions` table:
| Column | Type | Description |
|--------|------|-------------|
| `id` | UUID | Primary key |
| `tenant_id` | UUID | Tenant scope |
| `entity_type` | String(50) | Entity type (e.g., 'contact', 'dms_file') |
| `entity_id` | UUID | The specific entity |
| `principal_type` | String(10) | 'user', 'group', 'role', 'guest' |
| `principal_id` | UUID | The user/group/role ID |
| `permission_level` | String(20) | 'none', 'read', 'write', 'admin', 'delete' |
| `expires_at` | DateTime | Optional expiration |
| `created_by` | UUID | Who created this permission |
| `created_at` | DateTime | Creation timestamp |
| `updated_at` | DateTime | Last update timestamp |
### EntityPolicy (ABAC)
Stored in the `entity_policies` table:
| Column | Type | Description |
|--------|------|-------------|
| `id` | UUID | Primary key |
| `tenant_id` | UUID | Tenant scope |
| `name` | String(200) | Policy name |
| `entity_type` | String(50) | Target entity type |
| `principal_type` | String(10) | 'user', 'group', 'role' |
| `principal_id` | UUID | Target principal |
| `effect` | String(10) | 'allow' or 'deny' |
| `conditions` | JSONB | Attribute-based conditions |
| `priority` | Integer | Evaluation priority (higher = first) |
| `enabled` | Boolean | Whether the policy is active |
| `created_at` | DateTime | Creation timestamp |
| `updated_at` | DateTime | Last update timestamp |
### OwnedMixin
Adds `owner_id` to any model:
- `NULL` → Tenant-owned (visible to all with module permission)
- `UUID` → Owned by that user
- Set automatically by service layer on creation
- Transfer requires owner, admin, or system_admin role
---
## 3. Entity Permissions API
### List Permissions
```
GET /api/v1/{entity_type}/{entity_id}/permissions
```
Returns all permission entries for an entity.
### Create/Update Permission
```
POST /api/v1/{entity_type}/{entity_id}/permissions
```
```json
{
"user_id": "uuid",
"access_level": "read",
"expires_at": "2026-12-31T23:59:59Z"
}
```
### Revoke Permission
```
DELETE /api/v1/{entity_type}/{entity_id}/permissions/{user_id}
```
### Check Access
```
GET /api/v1/{entity_type}/{entity_id}/access?user_id={uuid}
```
Returns the effective access level for a user.
---
## 4. ABAC Policies
### Policy Conditions Format
```json
{
"operator": "AND",
"rules": [
{"field": "status", "op": "eq", "value": "active"},
{"field": "amount", "op": "gte", "value": 1000},
{"field": "tags", "op": "contains", "value": "vip"}
]
}
```
### Supported Operators
| Operator | Description | Example |
|----------|-------------|--------|
| `eq` | Equals | `{"field": "type", "op": "eq", "value": "company"}` |
| `neq` | Not equals | `{"field": "status", "op": "neq", "value": "archived"}` |
| `gt` | Greater than | `{"field": "amount", "op": "gt", "value": 100}` |
| `gte` | Greater or equal | `{"field": "amount", "op": "gte", "value": 50}` |
| `lt` | Less than | `{"field": "amount", "op": "lt", "value": 10000}` |
| `lte` | Less or equal | `{"field": "amount", "op": "lte", "value": 500}` |
| `in` | In list | `{"field": "status", "op": "in", "value": ["active", "pending"]}` |
| `not_in` | Not in list | `{"field": "status", "op": "not_in", "value": ["deleted"]}` |
| `contains` | String contains | `{"field": "name", "op": "contains", "value": "VIP"}` |
| `starts_with` | String starts with | `{"field": "name", "op": "starts_with", "value": "Confidential"}` |
| `is_null` | Is NULL | `{"field": "email", "op": "is_null"}` |
| `is_not_null` | Is not NULL | `{"field": "email", "op": "is_not_null"}` |
### Policy Evaluation
1. **Allow policies**: OR-joined (at least one must match for access)
2. **Deny policies**: NOT (none may match — deny takes precedence)
3. **Priority**: Higher priority policies evaluated first
4. **Enabled flag**: Disabled policies are skipped
---
## 5. Service Layer
### Entity Permission Service (`app/services/entity_permission_service.py`)
| Function | Description |
|----------|-------------|
| `get_effective_access()` | Get access level for a user on a specific entity |
| `get_visible_ids()` | Get all visible entity IDs for a user |
| `batch_get_effective_access()` | Batch resolve access for multiple entities |
| `check_entity_access()` | Check if user has at least required level |
| `get_cached_visible_ids()` | Get visible IDs with Redis caching |
| `create_permission()` | Create or update a permission entry |
| `update_permission()` | Update an existing permission |
| `delete_permission()` | Delete a permission entry |
| `list_permissions()` | List all permissions for an entity |
| `list_all_permissions()` | List all permissions for a tenant |
| `cleanup_expired_permissions()` | Remove expired permission entries |
| `invalidate_all_user_entity_cache()` | Clear all cached permissions for a user |
| `get_permission_analytics()` | Get permission statistics |
### Policy Service (`app/services/policy_service.py`)
| Function | Description |
|----------|-------------|
| `create_policy()` | Create a new ABAC policy |
| `update_policy()` | Update an existing policy |
| `delete_policy()` | Delete a policy |
| `list_policies()` | List policies for a tenant |
| `build_sql_condition()` | Translate JSONB conditions to SQLAlchemy filters |
| `apply_policy_filter()` | Apply ABAC policies to a query |
---
## 6. Caching
The permission system uses Redis for caching visibility results:
- **Cache key**: `ep_vis:{user_id}:{tenant_id}:{entity_type}`
- **Cache value**: JSON with `visible_ids` and `access_map`
- **TTL**: 5 minutes (300 seconds)
- **Invalidation**: Automatic on permission create/update/delete
### Cache Flow
1. Check Redis cache for user + entity type
2. Cache hit → return cached visible IDs
3. Cache miss → resolve from database, store in cache
4. Permission changes → invalidate affected user caches
---
## 7. Performance Considerations
- **Batch resolution** (`batch_get_effective_access`) is preferred over individual `get_effective_access` calls
- **Redis caching** reduces database load for repeated visibility checks
- **Bitmap optimization** for large entity sets (planned)
- **Indexes** on `entity_type + entity_id`, `principal_type + principal_id`, `tenant_id`, `expires_at`
- **Partitioning** recommended for `entity_permissions` table at scale
---
## 8. Security Considerations
- **Deny takes precedence** over allow in both ACL and ABAC
- **Expired permissions** are automatically excluded from resolution
- **System admin** bypasses all permission checks
- **Audit logging** for all permission changes
- **Notifications** sent to users when permissions are granted/revoked
- **Tenant isolation** enforced via `tenant_id` on all permission entries
---
## 9. Examples
### Grant Read Access to a User
```python
from app.services import entity_permission_service as eps
await eps.create_permission(
db_session,
tenant_id=tenant_id,
entity_type="contact",
entity_id=str(contact_id),
principal_type="user",
principal_id=str(user_id),
permission_level="read",
created_by=current_user.id,
)
```
### Check Access
```python
access = await eps.get_effective_access(
db_session, tenant_id, user_id, "contact", contact_id
)
if access in ("read", "write", "admin", "delete", "owner"):
# User has access
pass
```
### Create ABAC Policy
```python
from app.services import policy_service as ps
policy = await ps.create_policy(
db_session,
tenant_id=tenant_id,
name="VIP Only",
entity_type="contact",
principal_type="user",
principal_id=str(user_id),
effect="allow",
conditions={
"operator": "AND",
"rules": [
{"field": "type", "op": "eq", "value": "company"},
{"field": "name", "op": "contains", "value": "VIP"},
]
},
)
```
### Batch Resolve Access
```python
result = await eps.batch_get_effective_access(
db_session, tenant_id, user_id, "contact", entity_ids
)
for entity_id, level in result.items():
print(f"Entity {entity_id}: {level}")
```
---
## 10. Troubleshooting
### Common Issues
| Issue | Cause | Solution |
|-------|-------|----------|
| User sees nothing | No permissions, not owner, not system admin | Grant explicit permission or set owner_id |
| Expired permission still works | Cache not invalidated | Wait for TTL or invalidate cache manually |
| ABAC policy not applied | Policy disabled or no conditions match | Check `enabled` flag and conditions |
| System admin can't see entity | Entity deleted or wrong tenant | Check `deleted_at` and `tenant_id` |
| Permission creation fails | Duplicate unique constraint | Use upsert (create_permission handles this) |
### Debugging
Enable debug logging:
```python
import logging
logging.getLogger("app.services.entity_permission_service").setLevel(logging.DEBUG)
logging.getLogger("app.services.policy_service").setLevel(logging.DEBUG)
```
+509
View File
@@ -0,0 +1,509 @@
# Permission System Plugin Development Guide
> **Version:** 1.0
> **Date:** 2026-07-29
> **Applies to:** Plugin developers integrating with the LeoCRM permission system
---
## 1. Overview
Plugins can leverage the LeoCRM permission system to add row-level access control to their entities. This guide covers:
- Adding `OwnedMixin` to plugin models
- Using `apply_visibility_filter()` for list queries
- Registering entity types for permission management
- Defining field-level permissions
- Integrating with ABAC policies
---
## 2. Adding OwnedMixin to Plugin Models
To enable ownership tracking for your plugin's entities, add `OwnedMixin` to your SQLAlchemy model:
```python
from app.models.owned_mixin import OwnedMixin
from app.core.db import Base, TenantMixin
class MyEntity(Base, TenantMixin, OwnedMixin):
__tablename__ = "my_entities"
id: Mapped[uuid.UUID] = mapped_column(
PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4
)
name: Mapped[str] = mapped_column(String(200), nullable=False)
# ... other fields
```
### Owner ID Semantics
- `NULL` → Tenant-owned (visible to all with module permission)
- `UUID` → Owned by that user (visible to owner + shared via entity_permissions)
- Set automatically by service layer on creation
- Transfer requires owner, admin, or system_admin role
### Setting Owner on Creation
```python
from app.models.owned_mixin import OwnedMixin
async def create_entity(db: AsyncSession, data: dict, current_user: User):
entity = MyEntity(
tenant_id=current_user.tenant_id,
owner_id=current_user.id, # Set owner automatically
**data,
)
db.add(entity)
await db.commit()
await db.refresh(entity)
return entity
```
---
## 3. Using apply_visibility_filter()
The `apply_visibility_filter()` function filters a query to only return entities the user can see. This is the recommended way to implement list endpoints.
### Basic Usage
```python
from app.services import entity_permission_service as eps
async def list_entities(
db: AsyncSession,
tenant_id: uuid.UUID,
user_id: uuid.UUID,
) -> list[MyEntity]:
# Get visible entity IDs
visible_ids, access_map = await eps.get_visible_ids(
db, tenant_id, user_id, "my_entity"
)
if not visible_ids:
return []
# Query only visible entities
result = await db.execute(
select(MyEntity)
.where(MyEntity.id.in_(visible_ids))
.where(MyEntity.tenant_id == tenant_id)
)
return result.scalars().all()
```
### With Caching
```python
async def list_entities_cached(
db: AsyncSession,
redis: aioredis.Redis,
tenant_id: uuid.UUID,
user_id: uuid.UUID,
) -> list[MyEntity]:
# Use cached version for better performance
visible_ids, access_map = await eps.get_cached_visible_ids(
db, redis, tenant_id, user_id, "my_entity"
)
if not visible_ids:
return []
result = await db.execute(
select(MyEntity)
.where(MyEntity.id.in_(visible_ids))
.where(MyEntity.tenant_id == tenant_id)
)
return result.scalars().all()
```
### With Access Level in Response
```python
async def list_entities_with_access(
db: AsyncSession,
tenant_id: uuid.UUID,
user_id: uuid.UUID,
) -> list[dict]:
visible_ids, access_map = await eps.get_visible_ids(
db, tenant_id, user_id, "my_entity"
)
if not visible_ids:
return []
result = await db.execute(
select(MyEntity)
.where(MyEntity.id.in_(visible_ids))
.where(MyEntity.tenant_id == tenant_id)
)
entities = result.scalars().all()
return [
{
**entity.to_dict(),
"access_level": access_map.get(entity.id, "none"),
}
for entity in entities
]
```
---
## 4. Entity Registration
Register your entity type so it appears in the permission management UI and API.
### In Your Plugin Manifest
```python
from app.plugins.manifest import PluginManifest
class MyPluginManifest(PluginManifest):
# ... other fields
entity_types: list[str] = ["my_entity"]
```
### In Your Plugin Class
```python
from app.plugins.base import BasePlugin
class MyPlugin(BasePlugin):
@property
def entity_types(self) -> list[str]:
return ["my_entity"]
```
### Registering Routes for Permission Management
```python
from fastapi import APIRouter, Depends
router = APIRouter(prefix="/api/v1/my-entities")
@router.get("/{entity_id}/permissions")
async def list_my_entity_permissions(
entity_id: uuid.UUID,
db: AsyncSession = Depends(get_db),
current_user: User = Depends(get_current_user),
):
"""List all permissions for a specific entity."""
return await eps.list_permissions(
db, current_user.tenant_id, "my_entity", str(entity_id)
)
@router.post("/{entity_id}/permissions")
async def create_my_entity_permission(
entity_id: uuid.UUID,
body: PermissionCreate,
db: AsyncSession = Depends(get_db),
current_user: User = Depends(get_current_user),
):
"""Grant permission to a user for this entity."""
return await eps.create_permission(
db,
tenant_id=current_user.tenant_id,
entity_type="my_entity",
entity_id=str(entity_id),
principal_type="user",
principal_id=str(body.user_id),
permission_level=body.access_level,
expires_at=body.expires_at,
created_by=current_user.id,
)
```
---
## 5. Field Definitions
Define which fields of your entity are visible at each permission level.
### Field Permission Schema
```python
from pydantic import BaseModel
from typing import Any
class EntityFieldDefinition(BaseModel):
"""Define field visibility per permission level."""
name: str
type: str # "string", "number", "boolean", "date", "reference"
required: bool = False
readable: dict[str, bool] = {
"read": True,
"write": True,
"admin": True,
"delete": True,
"owner": True,
}
writable: dict[str, bool] = {
"read": False,
"write": True,
"admin": True,
"delete": True,
"owner": True,
}
```
### Registering Field Definitions
```python
from app.core.permission_registry import register_entity_fields
FIELD_DEFINITIONS = [
EntityFieldDefinition(
name="name",
type="string",
required=True,
readable={"read": True, "write": True, "admin": True, "delete": True, "owner": True},
writable={"read": False, "write": True, "admin": True, "delete": True, "owner": True},
),
EntityFieldDefinition(
name="sensitive_data",
type="string",
required=False,
readable={"read": False, "write": False, "admin": True, "delete": True, "owner": True},
writable={"read": False, "write": False, "admin": True, "delete": True, "owner": True},
),
]
# Register during plugin activation
register_entity_fields("my_entity", FIELD_DEFINITIONS)
```
---
## 6. Integrating with ABAC Policies
Your plugin can create and manage ABAC policies for its entities.
### Creating Policies
```python
from app.services import policy_service as ps
# Create an allow policy
policy = await ps.create_policy(
db_session,
tenant_id=tenant_id,
name="Allow VIP my_entities",
entity_type="my_entity",
principal_type="user",
principal_id=str(user_id),
effect="allow",
conditions={
"operator": "AND",
"rules": [
{"field": "priority", "op": "gte", "value": 10},
]
},
)
```
### Applying Policies to Queries
```python
query = select(MyEntity).where(MyEntity.tenant_id == tenant_id)
query = await ps.apply_policy_filter(
db_session, query, "my_entity", user_id, tenant_id, MyEntity
)
result = await db_session.execute(query)
entities = result.scalars().all()
```
---
## 7. Best Practices
### Do's
- ✅ Always set `owner_id` on entity creation
- ✅ Use `get_visible_ids()` for list endpoints
- ✅ Use `batch_get_effective_access()` for bulk operations
- ✅ Register entity types for permission management
- ✅ Define field-level permissions for sensitive data
- ✅ Use Redis caching for frequently accessed permissions
- ✅ Handle permission expiration gracefully
### Don'ts
- ❌ Don't bypass permission checks for list endpoints
- ❌ Don't expose `owner_id` changes without authorization
- ❌ Don't create permissions without audit logging
- ❌ Don't forget to invalidate cache after permission changes
- ❌ Don't use `get_effective_access()` in loops — use batch instead
---
## 8. Testing
### Test Fixtures
```python
import pytest
from app.services import entity_permission_service as eps
from tests.conftest import seed_tenant_and_users
@pytest.mark.asyncio
async def test_my_entity_permissions(db_session):
seed = await seed_tenant_and_users(db_session)
tenant_id = seed["tenant_a"].id
user_id = seed["admin_a"].id
# Create entity with owner
entity = MyEntity(
tenant_id=tenant_id,
owner_id=user_id,
name="Test Entity",
)
db_session.add(entity)
await db_session.commit()
# Check access
access = await eps.get_effective_access(
db_session, tenant_id, user_id, "my_entity", entity.id
)
assert access == "owner"
```
### Mocking Permissions
```python
from unittest.mock import AsyncMock, patch
async def test_list_with_mock_permissions():
with patch(
"app.services.entity_permission_service.get_visible_ids",
new=AsyncMock(return_value=({uuid.UUID(int=1)}, {uuid.UUID(int=1): "read"})),
):
# Your test code
pass
```
---
## 9. API Reference
### Entity Permission Service
| Function | Import | Description |
|----------|--------|-------------|
| `get_effective_access()` | `from app.services import entity_permission_service as eps` | Get access level for a user on a specific entity |
| `get_visible_ids()` | Same | Get all visible entity IDs for a user |
| `batch_get_effective_access()` | Same | Batch resolve access for multiple entities |
| `check_entity_access()` | Same | Check if user has at least required level |
| `get_cached_visible_ids()` | Same | Get visible IDs with Redis caching |
| `create_permission()` | Same | Create or update a permission entry |
| `update_permission()` | Same | Update an existing permission |
| `delete_permission()` | Same | Delete a permission entry |
| `list_permissions()` | Same | List all permissions for an entity |
### Policy Service
| Function | Import | Description |
|----------|--------|-------------|
| `create_policy()` | `from app.services import policy_service as ps` | Create a new ABAC policy |
| `update_policy()` | Same | Update an existing policy |
| `delete_policy()` | Same | Delete a policy |
| `list_policies()` | Same | List policies for a tenant |
| `build_sql_condition()` | Same | Translate JSONB conditions to SQLAlchemy filters |
| `apply_policy_filter()` | Same | Apply ABAC policies to a query |
---
## 10. Example: Complete Plugin Integration
```python
"""Example plugin with full permission system integration."""
from __future__ import annotations
import uuid
from typing import Any
from sqlalchemy import String
from sqlalchemy.dialects.postgresql import UUID as PGUUID
from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.orm import Mapped, mapped_column
from app.core.db import Base, TenantMixin
from app.models.owned_mixin import OwnedMixin
from app.services import entity_permission_service as eps
class MyEntity(Base, TenantMixin, OwnedMixin):
"""Example entity with permission support."""
__tablename__ = "my_entities"
id: Mapped[uuid.UUID] = mapped_column(
PGUUID(as_uuid=True), primary_key=True, default=uuid.uuid4
)
name: Mapped[str] = mapped_column(String(200), nullable=False)
priority: Mapped[int] = mapped_column(nullable=False, default=0)
async def create_my_entity(
db: AsyncSession,
tenant_id: uuid.UUID,
user_id: uuid.UUID,
name: str,
priority: int = 0,
) -> MyEntity:
"""Create a new entity with owner set."""
entity = MyEntity(
tenant_id=tenant_id,
owner_id=user_id,
name=name,
priority=priority,
)
db.add(entity)
await db.commit()
await db.refresh(entity)
return entity
async def list_visible_entities(
db: AsyncSession,
tenant_id: uuid.UUID,
user_id: uuid.UUID,
) -> list[MyEntity]:
"""List entities visible to the user."""
visible_ids, _ = await eps.get_visible_ids(
db, tenant_id, user_id, "my_entity"
)
if not visible_ids:
return []
from sqlalchemy import select
result = await db.execute(
select(MyEntity)
.where(MyEntity.id.in_(visible_ids))
.where(MyEntity.tenant_id == tenant_id)
)
return result.scalars().all()
async def share_entity(
db: AsyncSession,
tenant_id: uuid.UUID,
entity_id: uuid.UUID,
target_user_id: uuid.UUID,
level: str,
created_by: uuid.UUID,
) -> dict:
"""Share an entity with another user."""
return await eps.create_permission(
db,
tenant_id=tenant_id,
entity_type="my_entity",
entity_id=str(entity_id),
principal_type="user",
principal_id=str(target_user_id),
permission_level=level,
created_by=created_by,
)
```