feat(E): Unified Search — 24 Tasks complete

- SPIKE-E: FTS+Vector+Permission benchmark on 10k records (all <30ms) - E-PROV: supports_fts/vector/rag/graph capability flags on all providers - E-FTS/VEC: All 11 providers refactored to BaseSearchProvider with permission filtering - E-PERM: Over-fetch strategy for vector+permission (15x faster than ANY() filter) - E-FUSE: rrf_fusion_multi() for N-way RRF over FTS+Vector+RAG+Graph - E-LLM: Query understanding cleaned up to use central llm_complete() - E-CHUNK: Document chunking module + document_chunks table with HNSW index - E-EMB: Chunk embedding ARQ jobs (index_file_chunks, reindex_chunks) - E-RAG: RAG retrieval via FileSearchProvider.search_rag() - E-GRAPH: GraphRAG BFS traversal via GraphRAGSearchProvider.search_graph() - E-IX-EVT: Auto-indexing via outbox events + delete/cleanup handlers - E-IX-RE: Batch reindex with progress tracking + reindex_all job - E-DATA-LIFE: Lifecycle module (remove/rebuild/restore/correct) + API endpoints - E-K-MEM: AgentMemorySearchProvider - E-P-AI: AIChatSearchProvider - E-P-WF: WorkflowSearchProvider - E-P-COMM: ConversationSearchProvider verified (already on BaseSearchProvider) - E-API: Filter params (date_from/to, tags, sort) + /facets endpoint - E-TOOL: unified_search AI tool registered in ToolRegistry - E-MCP: Search tool in MCP server with normal RBAC/tenant checks - E-UI-CMD: CommandPalette (Cmd+K) with debounced search + recent searches - E-UI-FAC: SearchFacets, SearchResultCard, SavedSearches components - E-TEST: 40 new tests in test_unified_search_phase_e.py (105 total green) - E-DOC: api-documentation.md, plugin-development-guide.md, test-strategy.md updated 105 tests passing, TypeScript clean.
2026-08-14 01:34:58 +02:00
parent 60f30d021b
commit 3d9b76cea4
45 changed files with 5378 additions and 402 deletions
@@ -324,16 +324,112 @@ Agent Builder, Automation Builder, Cron-Scheduler, Agent Runner.

 ### search (Unified Search)

-7 endpoints for cross-entity search.
+Hybrid cross-entity search (PostgreSQL FTS + pgvector) with KI query understanding, RRF rank fusion, visibility filtering, and field-level RBAC.

 | Method | Path | Description |
 |--------|------|-------------|
-| POST | `/api/v1/search` | Unified search across all entities. |
-| GET | `/api/v1/search/providers` | List search providers. |
-| POST | `/api/v1/search/reindex` | Rebuild search index. |
-| POST | `/api/v1/search/similar` | Find similar entities. |
-| GET | `/api/v1/search/suggest` | Search suggestions. |
-| GET | `/api/v1/search/stats` | Search index statistics. |
+| GET | `/api/v1/search` | Hybrid search via query params (same as POST). |
+| POST | `/api/v1/search` | Hybrid search with KI query understanding. |
+| GET | `/api/v1/search/suggest` | Autocomplete suggestions (FTS prefix). |
+| POST | `/api/v1/search/similar` | Find similar entities across all types by embedding. |
+| POST | `/api/v1/search/reindex` | Trigger reindexing of entity types (admin). |
+| GET | `/api/v1/search/providers` | List active search providers + capability flags. |
+| POST | `/api/v1/search/providers/{entity_type}/toggle` | Toggle a provider on/off (admin). |
+| GET | `/api/v1/search/stats` | Search index statistics (indexed/pending per table). |
+| GET | `/api/v1/search/facets` | Available facets (entity types, tags, date ranges). |
+| POST | `/api/v1/search/rebuild/{entity_type}/{entity_id}` | Rebuild index for a single entity (admin). |
+| POST | `/api/v1/search/purge/{entity_type}/{entity_id}` | Purge entity from index (admin, GDPR). |
+
+> **MCP:** Unified search is also exposed as an MCP tool named `search` (category `search`, permission `search:read`) via the MCP server plugin (`app/plugins/builtins/mcp_server/tool_definitions.py`).
+
+#### Search Request (POST `/api/v1/search`)
+
+```json
+{
+  "query": "Max Mustermann",
+  "entity_types": ["contact", "mail"],
+  "limit": 20,
+  "offset": 0,
+  "date_from": "2026-01-01",
+  "date_to": "2026-12-31",
+  "tags": ["vip", "partner"],
+  "sort": "relevance"
+}
+```
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `query` | string | — | Search query (1–500 chars, required). |
+| `entity_types` | list[string] | all | Restrict search to these entity types. |
+| `limit` | int | 20 | Max results (1–100). |
+| `offset` | int | 0 | Pagination offset. |
+| `date_from` | string | null | ISO date `YYYY-MM-DD` — filter by `created_at`/`updated_at >= date_from`. |
+| `date_to` | string | null | ISO date `YYYY-MM-DD` — filter by `created_at`/`updated_at <= date_to`. |
+| `tags` | list[string] | null | Filter results by tags (comma-separated on entities). |
+| `sort` | string | `relevance` | Sort order: `relevance`, `date`, `name`. |
+
+#### Search Response
+
+```json
+{
+  "query": "Max Mustermann",
+  "normalized_query": "max mustermann",
+  "results": [
+    {
+      "entity_type": "contact",
+      "entity_id": "uuid",
+      "title": "Max Mustermann",
+      "snippet": "max@example.com",
+      "score": 0.95,
+      "data": {"type": "person"}
+    }
+  ],
+  "facets": {"types": {"contact": 1}},
+  "summary": "1 Ergebnis",
+  "suggestions": []
+}
+```
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `query` | string | Original query. |
+| `normalized_query` | string | KI-normalized query. |
+| `results` | list[SearchResult] | Ranked results (entity_type, entity_id, title, snippet, score, data). |
+| `facets` | object | KI-generated facet counts. |
+| `summary` | string | Human-readable summary. |
+| `suggestions` | list[string] | Suggested follow-up filters. |
+
+#### GET `/api/v1/search` Query Params
+
+Same fields as the POST body, passed as query parameters: `q` (required), `entity_types` (comma-separated), `limit`, `offset`, `date_from`, `date_to`, `tags` (comma-separated), `sort`.
+
+#### GET `/api/v1/search/suggest`
+
+Query params: `q` (required, 1–200), `limit` (default 10, 1–50). Returns `{"suggestions": ["..."]}`.
+
+#### POST `/api/v1/search/similar`
+
+Body: `{"entity_type": "contact", "entity_id": "uuid", "limit": 5}`. Returns `{"similar": {"mail": [SearchResult...], ...}}`.
+
+#### POST `/api/v1/search/reindex`
+
+Body: `{"entity_types": ["contact"], "include_chunks": true}`. Returns `{"status": "ok", "entity_types": [...], "include_chunks": true, "job_ids": [...]}`. Requires `search:admin`.
+
+#### GET `/api/v1/search/providers`
+
+Returns a list of providers: `{"entity_type", "plugin_name", "is_active", "supports_fts", "supports_vector", "supports_rag", "supports_graph"}`.
+
+#### GET `/api/v1/search/facets`
+
+Returns `{"entity_types": [...], "tags": [...], "date_ranges": {"contacts": {"min": "...", "max": "..."}, ...}}`.
+
+#### GET `/api/v1/search/stats`
+
+Returns per-table `{"total", "indexed", "pending"}` plus `recent_logs` (last 10 index log entries).
+
+#### POST `/api/v1/search/rebuild/{entity_type}/{entity_id}` / `/purge/{entity_type}/{entity_id}`
+
+Admin-only. Rebuild regenerates the embedding + TSV; purge sets embedding/TSV to NULL (and removes chunks for files). Returns `{"status": "ok"|"failed", "entity_type", "entity_id", "message"}`.

 ### reports (Report Generator)

@@ -685,7 +685,194 @@ async def on_deactivate(self, db, service_container, event_bus):

 ---

-## 27. Testing Guide
+## 27. Search Integration
+
+The Unified Search plugin provides hybrid cross-entity search (PostgreSQL FTS + pgvector) with KI query understanding, RRF rank fusion, visibility filtering, and field-level RBAC. Plugins can expose their entities to unified search by implementing a `SearchProvider`.
+
+### 27.1 Creating a SearchProvider
+
+Inherit from `BaseSearchProvider` and implement the required methods. The base class handles visibility filtering automatically (see 27.4).
+
+```python
+# app/plugins/builtins/my_plugin/search_provider.py
+from __future__ import annotations
+
+import uuid
+from typing import Any
+
+from sqlalchemy import text
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.plugins.builtins.unified_search.base_provider import BaseSearchProvider
+
+
+class MyEntitySearchProvider(BaseSearchProvider):
+    """Search provider for MyEntity."""
+
+    entity_type = "my_entity"
+    supports_fts = True
+    supports_vector = True
+    supports_rag = False
+    supports_graph = False
+
+    async def _search_fts_filtered(
+        self,
+        db: AsyncSession,
+        tsquery: str,
+        tenant_id: uuid.UUID,
+        limit: int,
+        visible_ids: set[uuid.UUID] | None,
+    ) -> list[dict[str, Any]]:
+        """Full-text search filtered by visible_ids (None = no filter)."""
+        if visible_ids is not None:
+            sql = text(
+                """
+                SELECT e.*, ts_rank(e.search_tsv, to_tsquery('pg_catalog.german', :q)) AS rank
+                FROM my_entities e
+                WHERE e.tenant_id = :tid
+                  AND e.deleted_at IS NULL
+                  AND e.search_tsv @@ to_tsquery('pg_catalog.german', :q)
+                  AND e.id = ANY(:visible_ids)
+                ORDER BY rank DESC
+                LIMIT :lim
+                """
+            )
+            result = await db.execute(
+                sql, {"q": tsquery, "tid": tenant_id, "lim": limit, "visible_ids": list(visible_ids)}
+            )
+        else:
+            sql = text(
+                """
+                SELECT e.*, ts_rank(e.search_tsv, to_tsquery('pg_catalog.german', :q)) AS rank
+                FROM my_entities e
+                WHERE e.tenant_id = :tid
+                  AND e.deleted_at IS NULL
+                  AND e.search_tsv @@ to_tsquery('pg_catalog.german', :q)
+                ORDER BY rank DESC
+                LIMIT :lim
+                """
+            )
+            result = await db.execute(sql, {"q": tsquery, "tid": tenant_id, "lim": limit})
+        return [dict(r) for r in result.mappings().all()]
+
+    async def _search_vector_filtered(
+        self,
+        db: AsyncSession,
+        embedding: list[float],
+        tenant_id: uuid.UUID,
+        limit: int,
+        visible_ids: set[uuid.UUID] | None,
+    ) -> list[dict[str, Any]]:
+        """Semantic vector search on the entity's embedding column."""
+        # Same pattern as _search_fts_filtered but using `embedding <=> cast(:emb AS vector)`
+        return []
+
+    async def get_embedding_text(
+        self, db: AsyncSession, entity_id: uuid.UUID, tenant_id: uuid.UUID
+    ) -> str:
+        """Return the text used for embedding generation (non-sensitive fields only)."""
+        sql = text(
+            "SELECT name, description FROM my_entities WHERE id = :eid AND tenant_id = :tid"
+        )
+        result = await db.execute(sql, {"eid": entity_id, "tid": tenant_id})
+        row = result.mappings().first()
+        if not row:
+            return ""
+        return " ".join(str(v) for v in row.values() if v)
+
+    def to_search_result(self, entity: object) -> dict[str, Any]:
+        """Convert an ORM entity or dict to a search result dict."""
+        if isinstance(entity, dict):
+            entity_id = str(entity.get("id", ""))
+            name = entity.get("name", "")
+            description = entity.get("description", "")
+        else:
+            entity_id = str(getattr(entity, "id", ""))
+            name = getattr(entity, "name", "")
+            description = getattr(entity, "description", "")
+        return {
+            "entity_type": self.entity_type,
+            "entity_id": entity_id,
+            "title": name,
+            "snippet": description or "",
+            "score": 0.0,
+            "data": {},
+        }
+```
+
+### 27.2 Capability Flags
+
+Each provider declares which search modes it supports via class attributes. The registry and API use these flags to decide which search paths to run and to report capabilities to clients.
+
+| Flag | Default | Meaning |
+|------|---------|---------|
+| `supports_fts` | `True` | Full-text search via PostgreSQL `tsvector`/`tsquery`. |
+| `supports_vector` | `True` | Semantic vector search via pgvector embeddings. |
+| `supports_rag` | `False` | Retrieval-augmented generation over document chunks. |
+| `supports_graph` | `False` | Graph-based search (GraphRAG). |
+
+Set `supports_vector = False` (and implement `_search_vector_filtered` returning `[]`) when an entity has no embedding column, e.g. chat messages or workflows.
+
+### 27.3 Registering a Provider
+
+Register providers during plugin activation. The Unified Search plugin calls `auto_register_providers(db)` on activation, which registers all built-in providers. For a custom plugin, register your provider directly in `on_activate`:
+
+```python
+async def on_activate(self, db, service_container, event_bus) -> None:
+    await super().on_activate(db, service_container, event_bus)
+    from app.plugins.builtins.unified_search.provider_registry import get_search_registry
+    from app.plugins.builtins.my_plugin.search_provider import MyEntitySearchProvider
+
+    get_search_registry().register(MyEntitySearchProvider())
+```
+
+To add a provider to the built-in `auto_register_providers` list, import and append it to the `provider_cls` list in `app/plugins/builtins/unified_search/provider_registry.py`.
+
+### 27.4 Permission Filtering (Automatic)
+
+`BaseSearchProvider.search_fts` / `search_vector` automatically load the calling user's visible entity IDs via `get_visible_ids` and pass them to `_search_fts_filtered` / `_search_vector_filtered`. When `is_system_admin` is true or no `user_id` is provided, `visible_ids` is `None` and no filter is applied. Your `_search_*_filtered` implementation must honor `visible_ids` (add `AND id = ANY(:visible_ids)` when it is not `None`).
+
+### 27.5 Auto-Indexing via Events / Outbox
+
+Entities are indexed automatically when created or updated. The Unified Search plugin subscribes to domain events (e.g. `contact.created`, `contact.updated`, `mail.synced`, `file.uploaded`) and enqueues indexing jobs. For custom entities, publish the corresponding events or enqueue jobs directly:
+
+```python
+from app.core.jobs import enqueue_job
+
+# After creating/updating an entity
+await enqueue_job("index_entity", "my_entity", str(entity_id))
+```
+
+Indexing jobs call `index_entity(entity_type, entity_id, tenant_id, db)`, which uses the provider's `get_embedding_text` to generate an embedding and stores it in the entity's `embedding` column. The `indexed_at` column tracks dedup so unchanged entities are not re-embedded.
+
+### 27.6 Lifecycle Hooks
+
+The Unified Search plugin subscribes to lifecycle events to keep the index consistent:
+
+| Event | Handler | Effect |
+|-------|---------|--------|
+| `entity.deleted` | `handle_entity_delete` | Removes embedding + TSV (and chunks for files). |
+| `entity.restored` | `handle_entity_restore` | Rebuilds the search index. |
+| `entity.corrected` | `handle_entity_correction` | Rebuilds the search index. |
+
+Publish these events (or call the lifecycle functions directly) when your plugin deletes, restores, or corrects entities so the search index stays in sync.
+
+### 27.7 RAG Document Chunking
+
+For RAG over long documents, use `chunk_text` from `app.plugins.builtins.unified_search.chunking` to split extracted text into overlapping chunks before embedding:
+
+```python
+from app.plugins.builtins.unified_search.chunking import chunk_text
+
+chunks = chunk_text(document_text, chunk_size=1000, overlap=200)
+# Each chunk: {"chunk_index": 0, "chunk_text": "...", "chunk_hash": "sha256..."}
+```
+
+Chunks are stored in the `document_chunks` table and embedded at the chunk level for fine-grained vector retrieval. `chunk_hash` is a deterministic SHA-256 of the chunk text, used for dedup.
+
+---
+
+## 28. Testing Guide

 ### 11.1 Backend Tests

@@ -295,3 +295,54 @@ Diese Pipeline ist verbindlich für Phase-Gate-Reviews und muss vor jedem Phasen
 - ✅ Retention: GDPR-Hard-Delete nach konfigurierbarer Aufbewahrungsfrist
 - ✅ Hook-based History: `do_action('entity.after_create/update/delete')` → `record_history()`
 - ✅ Dynamic Permission Checks: Restore-Permission aus RestoreConfig, nicht hardcoded
+
+---
+
+## Phase E — Unified Search Test-Konventionen
+
+### Neue Test-Datei: `tests/test_unified_search_phase_e.py` (40 Tests)
+
+| Test-Gruppe | Tests | Status |
+|-------------|-------|--------|
+| Provider-Capability-Flags (supports_fts/vector/rag/graph, get_providers_by_capability, get_capabilities) | 3 | ✅ |
+| RRF Multi-Fusion (2/3/4 Listen, Multi-Listen-Scoring, Backward-Compat, Empty Inputs) | 5 | ✅ |
+| Chunking (empty/short/long/exact multiple, Overlap, Hash deterministisch, Whitespace-Normalisierung) | 6 | ✅ |
+| Lifecycle (remove_from_index setzt embedding+TSV NULL, rebuild_index, entity.deleted/restored) | 6 | ✅ |
+| API-Filter (date_from/date_to, tags, sort, facets-Struktur) | 4 | ✅ |
+| AI-Tool (Name/Description, Parameter, OpenAI-Schema, Handler kompakt, Fehlerfälle) | 5 | ✅ |
+| Neue Provider (AgentMemory, AIChat, Workflow — Import + Flags) | 4 | ✅ |
+| Sensitive-Fields-Exclusion (nicht in search_tsv, nicht in Embedding-Text, Redaction) | 4 | ✅ |
+
+### Konventionen für Search-Tests
+
+1. **Isolation & Determinismus:** Jeder Test nutzt eine eigene Tenant-ID und eigene Entity-IDs. Keine zufälligen UUIDs — echte IDs aus der DB verwenden.
+2. **Schema-Anpassung idempotent:** Die Test-DB (`create_all`) definiert `search_tsv` als generierte Spalte, Produktion (Migration 0001) als Plain-Column mit Trigger. Der Lifecycle-Helper konvertiert die Spalte idempotent per `DO $$ ... DROP EXPRESSION` und droppt den `contacts_tsv_update`-Trigger, damit `remove_from_index` `search_tsv = NULL` setzen kann. Die Schema-Änderung persistiert über Testläufe (nur Tabellen werden getruncated).
+3. **Patch-Targets:** Funktionen, die innerhalb einer Funktion importiert werden (z.B. `index_entity` in `lifecycle.py`, `get_session_factory` in `ai_tool.py`), müssen am Ursprungsmodul gepatcht werden (`app.plugins.builtins.unified_search.embedding.index_entity`, `app.core.db.get_session_factory`), nicht am importierenden Modul.
+4. **Keine echten LLM/Embedding-Calls:** Alle KI-Aufrufe werden mit `AsyncMock` gemockt. Kein Test darf ein echtes Modell kontaktieren.
+
+### Mock-Patterns für LLM/Embedding-Calls
+
+```python
+from unittest.mock import AsyncMock, patch
+
+# LLM-Query-Verständnis (Normalisierung, Facets, Summary)
+with patch("app.plugins.builtins.unified_search.llm.llm_complete", new_callable=AsyncMock) as mock_llm:
+    mock_llm.return_value = {"normalized_query": "max mustermann", "facets": {}, "summary": "1 Ergebnis"}
+    # ... Test
+
+# Embedding-Generierung
+with patch("app.plugins.builtins.unified_search.embedding.generate_embedding", new_callable=AsyncMock) as mock_emb:
+    mock_emb.return_value = [0.1, 0.2, 0.3]
+    # ... Test
+
+# LLM-Embedding-Client
+with patch("app.plugins.builtins.unified_search.embedding.llm_embed", new_callable=AsyncMock) as mock_llm_emb:
+    mock_llm_emb.return_value = [0.1, 0.2, 0.3]
+    # ... Test
+```
+
+**Regeln:**
+- `llm_complete` liefert ein Dict mit `normalized_query`, `facets`, `summary` (und optional `suggestions`).
+- `generate_embedding` / `llm_embed` liefern eine Liste von Floats (Embedding-Vektor).
+- Bei Fehlerpfaden: `mock_llm.side_effect = Exception("...")` oder `return_value = None` für Fallback-Verhalten testen.
+- DB-Session-Factory in AI-Tool-Handler-Tests: `patch("app.core.db.get_session_factory", return_value=sf)` mit `async_sessionmaker(bind=db_session.bind, ...)`.