feat(E): Unified Search — 24 Tasks complete
Check Cross-Plugin Imports / check (push) Has been cancelled
Check Cross-Plugin Imports / check (push) Has been cancelled
- SPIKE-E: FTS+Vector+Permission benchmark on 10k records (all <30ms) - E-PROV: supports_fts/vector/rag/graph capability flags on all providers - E-FTS/VEC: All 11 providers refactored to BaseSearchProvider with permission filtering - E-PERM: Over-fetch strategy for vector+permission (15x faster than ANY() filter) - E-FUSE: rrf_fusion_multi() for N-way RRF over FTS+Vector+RAG+Graph - E-LLM: Query understanding cleaned up to use central llm_complete() - E-CHUNK: Document chunking module + document_chunks table with HNSW index - E-EMB: Chunk embedding ARQ jobs (index_file_chunks, reindex_chunks) - E-RAG: RAG retrieval via FileSearchProvider.search_rag() - E-GRAPH: GraphRAG BFS traversal via GraphRAGSearchProvider.search_graph() - E-IX-EVT: Auto-indexing via outbox events + delete/cleanup handlers - E-IX-RE: Batch reindex with progress tracking + reindex_all job - E-DATA-LIFE: Lifecycle module (remove/rebuild/restore/correct) + API endpoints - E-K-MEM: AgentMemorySearchProvider - E-P-AI: AIChatSearchProvider - E-P-WF: WorkflowSearchProvider - E-P-COMM: ConversationSearchProvider verified (already on BaseSearchProvider) - E-API: Filter params (date_from/to, tags, sort) + /facets endpoint - E-TOOL: unified_search AI tool registered in ToolRegistry - E-MCP: Search tool in MCP server with normal RBAC/tenant checks - E-UI-CMD: CommandPalette (Cmd+K) with debounced search + recent searches - E-UI-FAC: SearchFacets, SearchResultCard, SavedSearches components - E-TEST: 40 new tests in test_unified_search_phase_e.py (105 total green) - E-DOC: api-documentation.md, plugin-development-guide.md, test-strategy.md updated 105 tests passing, TypeScript clean.
This commit is contained in:
Agent Zero
@@ -685,7 +685,194 @@ async def on_deactivate(self, db, service_container, event_bus):
|
||||
|
||||
---
|
||||
|
||||
## 27. Testing Guide
|
||||
## 27. Search Integration
|
||||
|
||||
The Unified Search plugin provides hybrid cross-entity search (PostgreSQL FTS + pgvector) with KI query understanding, RRF rank fusion, visibility filtering, and field-level RBAC. Plugins can expose their entities to unified search by implementing a `SearchProvider`.
|
||||
|
||||
### 27.1 Creating a SearchProvider
|
||||
|
||||
Inherit from `BaseSearchProvider` and implement the required methods. The base class handles visibility filtering automatically (see 27.4).
|
||||
|
||||
```python
|
||||
# app/plugins/builtins/my_plugin/search_provider.py
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.plugins.builtins.unified_search.base_provider import BaseSearchProvider
|
||||
|
||||
|
||||
class MyEntitySearchProvider(BaseSearchProvider):
|
||||
"""Search provider for MyEntity."""
|
||||
|
||||
entity_type = "my_entity"
|
||||
supports_fts = True
|
||||
supports_vector = True
|
||||
supports_rag = False
|
||||
supports_graph = False
|
||||
|
||||
async def _search_fts_filtered(
|
||||
self,
|
||||
db: AsyncSession,
|
||||
tsquery: str,
|
||||
tenant_id: uuid.UUID,
|
||||
limit: int,
|
||||
visible_ids: set[uuid.UUID] | None,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Full-text search filtered by visible_ids (None = no filter)."""
|
||||
if visible_ids is not None:
|
||||
sql = text(
|
||||
"""
|
||||
SELECT e.*, ts_rank(e.search_tsv, to_tsquery('pg_catalog.german', :q)) AS rank
|
||||
FROM my_entities e
|
||||
WHERE e.tenant_id = :tid
|
||||
AND e.deleted_at IS NULL
|
||||
AND e.search_tsv @@ to_tsquery('pg_catalog.german', :q)
|
||||
AND e.id = ANY(:visible_ids)
|
||||
ORDER BY rank DESC
|
||||
LIMIT :lim
|
||||
"""
|
||||
)
|
||||
result = await db.execute(
|
||||
sql, {"q": tsquery, "tid": tenant_id, "lim": limit, "visible_ids": list(visible_ids)}
|
||||
)
|
||||
else:
|
||||
sql = text(
|
||||
"""
|
||||
SELECT e.*, ts_rank(e.search_tsv, to_tsquery('pg_catalog.german', :q)) AS rank
|
||||
FROM my_entities e
|
||||
WHERE e.tenant_id = :tid
|
||||
AND e.deleted_at IS NULL
|
||||
AND e.search_tsv @@ to_tsquery('pg_catalog.german', :q)
|
||||
ORDER BY rank DESC
|
||||
LIMIT :lim
|
||||
"""
|
||||
)
|
||||
result = await db.execute(sql, {"q": tsquery, "tid": tenant_id, "lim": limit})
|
||||
return [dict(r) for r in result.mappings().all()]
|
||||
|
||||
async def _search_vector_filtered(
|
||||
self,
|
||||
db: AsyncSession,
|
||||
embedding: list[float],
|
||||
tenant_id: uuid.UUID,
|
||||
limit: int,
|
||||
visible_ids: set[uuid.UUID] | None,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Semantic vector search on the entity's embedding column."""
|
||||
# Same pattern as _search_fts_filtered but using `embedding <=> cast(:emb AS vector)`
|
||||
return []
|
||||
|
||||
async def get_embedding_text(
|
||||
self, db: AsyncSession, entity_id: uuid.UUID, tenant_id: uuid.UUID
|
||||
) -> str:
|
||||
"""Return the text used for embedding generation (non-sensitive fields only)."""
|
||||
sql = text(
|
||||
"SELECT name, description FROM my_entities WHERE id = :eid AND tenant_id = :tid"
|
||||
)
|
||||
result = await db.execute(sql, {"eid": entity_id, "tid": tenant_id})
|
||||
row = result.mappings().first()
|
||||
if not row:
|
||||
return ""
|
||||
return " ".join(str(v) for v in row.values() if v)
|
||||
|
||||
def to_search_result(self, entity: object) -> dict[str, Any]:
|
||||
"""Convert an ORM entity or dict to a search result dict."""
|
||||
if isinstance(entity, dict):
|
||||
entity_id = str(entity.get("id", ""))
|
||||
name = entity.get("name", "")
|
||||
description = entity.get("description", "")
|
||||
else:
|
||||
entity_id = str(getattr(entity, "id", ""))
|
||||
name = getattr(entity, "name", "")
|
||||
description = getattr(entity, "description", "")
|
||||
return {
|
||||
"entity_type": self.entity_type,
|
||||
"entity_id": entity_id,
|
||||
"title": name,
|
||||
"snippet": description or "",
|
||||
"score": 0.0,
|
||||
"data": {},
|
||||
}
|
||||
```
|
||||
|
||||
### 27.2 Capability Flags
|
||||
|
||||
Each provider declares which search modes it supports via class attributes. The registry and API use these flags to decide which search paths to run and to report capabilities to clients.
|
||||
|
||||
| Flag | Default | Meaning |
|
||||
|------|---------|---------|
|
||||
| `supports_fts` | `True` | Full-text search via PostgreSQL `tsvector`/`tsquery`. |
|
||||
| `supports_vector` | `True` | Semantic vector search via pgvector embeddings. |
|
||||
| `supports_rag` | `False` | Retrieval-augmented generation over document chunks. |
|
||||
| `supports_graph` | `False` | Graph-based search (GraphRAG). |
|
||||
|
||||
Set `supports_vector = False` (and implement `_search_vector_filtered` returning `[]`) when an entity has no embedding column, e.g. chat messages or workflows.
|
||||
|
||||
### 27.3 Registering a Provider
|
||||
|
||||
Register providers during plugin activation. The Unified Search plugin calls `auto_register_providers(db)` on activation, which registers all built-in providers. For a custom plugin, register your provider directly in `on_activate`:
|
||||
|
||||
```python
|
||||
async def on_activate(self, db, service_container, event_bus) -> None:
|
||||
await super().on_activate(db, service_container, event_bus)
|
||||
from app.plugins.builtins.unified_search.provider_registry import get_search_registry
|
||||
from app.plugins.builtins.my_plugin.search_provider import MyEntitySearchProvider
|
||||
|
||||
get_search_registry().register(MyEntitySearchProvider())
|
||||
```
|
||||
|
||||
To add a provider to the built-in `auto_register_providers` list, import and append it to the `provider_cls` list in `app/plugins/builtins/unified_search/provider_registry.py`.
|
||||
|
||||
### 27.4 Permission Filtering (Automatic)
|
||||
|
||||
`BaseSearchProvider.search_fts` / `search_vector` automatically load the calling user's visible entity IDs via `get_visible_ids` and pass them to `_search_fts_filtered` / `_search_vector_filtered`. When `is_system_admin` is true or no `user_id` is provided, `visible_ids` is `None` and no filter is applied. Your `_search_*_filtered` implementation must honor `visible_ids` (add `AND id = ANY(:visible_ids)` when it is not `None`).
|
||||
|
||||
### 27.5 Auto-Indexing via Events / Outbox
|
||||
|
||||
Entities are indexed automatically when created or updated. The Unified Search plugin subscribes to domain events (e.g. `contact.created`, `contact.updated`, `mail.synced`, `file.uploaded`) and enqueues indexing jobs. For custom entities, publish the corresponding events or enqueue jobs directly:
|
||||
|
||||
```python
|
||||
from app.core.jobs import enqueue_job
|
||||
|
||||
# After creating/updating an entity
|
||||
await enqueue_job("index_entity", "my_entity", str(entity_id))
|
||||
```
|
||||
|
||||
Indexing jobs call `index_entity(entity_type, entity_id, tenant_id, db)`, which uses the provider's `get_embedding_text` to generate an embedding and stores it in the entity's `embedding` column. The `indexed_at` column tracks dedup so unchanged entities are not re-embedded.
|
||||
|
||||
### 27.6 Lifecycle Hooks
|
||||
|
||||
The Unified Search plugin subscribes to lifecycle events to keep the index consistent:
|
||||
|
||||
| Event | Handler | Effect |
|
||||
|-------|---------|--------|
|
||||
| `entity.deleted` | `handle_entity_delete` | Removes embedding + TSV (and chunks for files). |
|
||||
| `entity.restored` | `handle_entity_restore` | Rebuilds the search index. |
|
||||
| `entity.corrected` | `handle_entity_correction` | Rebuilds the search index. |
|
||||
|
||||
Publish these events (or call the lifecycle functions directly) when your plugin deletes, restores, or corrects entities so the search index stays in sync.
|
||||
|
||||
### 27.7 RAG Document Chunking
|
||||
|
||||
For RAG over long documents, use `chunk_text` from `app.plugins.builtins.unified_search.chunking` to split extracted text into overlapping chunks before embedding:
|
||||
|
||||
```python
|
||||
from app.plugins.builtins.unified_search.chunking import chunk_text
|
||||
|
||||
chunks = chunk_text(document_text, chunk_size=1000, overlap=200)
|
||||
# Each chunk: {"chunk_index": 0, "chunk_text": "...", "chunk_hash": "sha256..."}
|
||||
```
|
||||
|
||||
Chunks are stored in the `document_chunks` table and embedded at the chunk level for fine-grained vector retrieval. `chunk_hash` is a deterministic SHA-256 of the chunk text, used for dedup.
|
||||
|
||||
---
|
||||
|
||||
## 28. Testing Guide
|
||||
|
||||
### 11.1 Backend Tests
|
||||
|
||||
|
||||
Reference in New Issue
Block a user