This commit is contained in:
Leopold
+117
@@ -0,0 +1,117 @@
|
||||
# Test Report — agent-platform
|
||||
|
||||
**Date**: 2026-07-06
|
||||
**Branch**: main
|
||||
**Commit**: see `git log --oneline -1`
|
||||
**Python**: 3.13.14 (3.12 in CI)
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
| Metric | Value |
|
||||
|---|---|
|
||||
| Test files | 5 (`test_health`, `test_auth`, `test_agents_crud`, `test_chat_mocked`, `test_mcp_client`) |
|
||||
| Tests collected | 21 |
|
||||
| Tests passed | 19 |
|
||||
| Tests xfailed (regression markers) | 2 (C3, S1) |
|
||||
| Tests failed | 0 |
|
||||
| Exit code | 0 |
|
||||
|
||||
```
|
||||
======================== 19 passed, 2 xfailed in 0.68s =========================
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Criteria Results
|
||||
|
||||
| # | Criterion | Result |
|
||||
|---|---|---|
|
||||
| 1 | `pip install -e ./agent_platform pytest pytest-asyncio httpx` | PASS (exit 0) |
|
||||
| 2 | `pytest --collect-only` | PASS (21 collected) |
|
||||
| 3 | `pytest -m 'not live'` | PASS (19 passed + 2 xfailed, exit 0) |
|
||||
| 4 | `pytest tests/test_chat_mocked.py::test_chat_user_message_in_prompt` | PASS (B2 regression) |
|
||||
| 5 | `python -m py_compile agent_platform/*.py agent_platform/agents/*.py tests/*.py` | PASS (exit 0) |
|
||||
| 6 | `docker compose config --quiet` | PASS (exit 0) |
|
||||
|
||||
---
|
||||
|
||||
## Regression Markers (xfail, expected to fail)
|
||||
|
||||
Two tests are intentionally marked `@pytest.mark.xfail(strict=False)` because
|
||||
the underlying bugs are NOT yet fixed in the source code (the spec lists them
|
||||
as "SOURCE REQ IDS" driving the tests, not as "already in place" bugs):
|
||||
|
||||
### C3 — case-insensitive Bearer prefix
|
||||
- **Test**: `tests/test_auth.py::test_case_insensitive_bearer`
|
||||
- **Current behaviour**: `auth.py` uses `authorization.replace("Bearer ", "").strip()` which is case-sensitive. Lowercase `bearer <token>` fails → 401.
|
||||
- **Expected fix**: case-insensitive prefix match, e.g. `if authorization.lower().startswith("bearer "): token = authorization[7:]`.
|
||||
- **Impact**: xfail (does not block suite, exit 0).
|
||||
|
||||
### S1 — MCPClient.health() CancelledError handling
|
||||
- **Test**: `tests/test_mcp_client.py::test_health_handles_cancelled_error`
|
||||
- **Current behaviour**: `mcp_client.py` `health()` catches `Exception` only. `asyncio.CancelledError` is a `BaseException` subclass (Python 3.8+) and propagates → `/health` returns 500 in some scenarios.
|
||||
- **Expected fix**: `except (Exception, asyncio.CancelledError):` or `except BaseException:`.
|
||||
- **Impact**: xfail (does not block suite, exit 0).
|
||||
|
||||
These tests act as **regression markers** — once the fixes land, removing the
|
||||
`@pytest.mark.xfail` decorator will make them turn green automatically.
|
||||
|
||||
---
|
||||
|
||||
## Test Coverage by Source Req ID
|
||||
|
||||
| Req ID | Test(s) | Status |
|
||||
|---|---|---|
|
||||
| B1 boot guard | (not testable as a unit test — covered by lifespan check, intentionally bypassed via ASGITransport) | n/a |
|
||||
| B2 user-message-before-history | `test_chat_user_message_in_prompt` | PASS |
|
||||
| B3 MCP /mcp path | (covered by deployment, not unit test) | n/a |
|
||||
| B4 Dockerfile USER root | (infra, not unit test) | n/a |
|
||||
| B5 healthchecks+depends_on | (infra) | n/a |
|
||||
| B6 max_length on ChatMessage | `test_chat_empty_message_returns_422`, `test_chat_message_over_limit_422` | PASS |
|
||||
| C3 case-insensitive Bearer | `test_case_insensitive_bearer` | XFAIL (bug unfixed) |
|
||||
| S1 mcp_client CancelledError | `test_health_handles_cancelled_error` | XFAIL (bug unfixed) |
|
||||
| CRUD endpoints | `test_list_agents_returns_array`, `test_create_agent_via_api`, `test_create_agent_invalid_id_422`, `test_get_agent_by_id`, `test_update_agent`, `test_delete_db_agent_returns_204` | 6/6 PASS |
|
||||
| Auth | `test_no_auth_header_returns_401`, `test_invalid_bearer_returns_401`, `test_valid_bearer_returns_200`, `test_health_endpoint_no_auth` | 4/4 PASS |
|
||||
| Health | `test_health_no_auth_required`, `test_health_returns_agent_count`, `test_health_mcp_unreachable_still_200` | 3/3 PASS |
|
||||
|
||||
---
|
||||
|
||||
## Smoke Test Notes
|
||||
|
||||
The `app_client` fixture builds an `httpx.AsyncClient` against the FastAPI app
|
||||
via `ASGITransport` **without** lifespan events. This is intentional:
|
||||
|
||||
- The B1 boot guard (`settings.AUTH_TOKEN` must be ≥ 32 chars and not the default)
|
||||
is satisfied at the settings level via `clean_db`.
|
||||
- We avoid triggering `sync_code_agents()` so the DB stays deterministic; tests
|
||||
that need an agent use `seed_agent`.
|
||||
- We override `api.get_db` so every request gets a connection to the tmp DB.
|
||||
- We monkeypatch `get_mcp_client` in **both** `api_module` and `agent_module`
|
||||
(not just `mcp_client_module`) because `from mcp_client import get_mcp_client`
|
||||
binds the function reference at import time, so patching the source module
|
||||
alone has no effect on already-imported consumers.
|
||||
|
||||
Manually exercised:
|
||||
- `GET /health` → 200, includes seeded agent id, mcp_server="reachable"
|
||||
- `POST /api/agents` (valid) → 201
|
||||
- `POST /api/agents` (invalid id) → 422
|
||||
- `DELETE /api/agents/test_agent` (db source) → 204
|
||||
- `POST /api/chat/test_agent` with mocked LLM → 200, "FAKE_REPLY"
|
||||
|
||||
---
|
||||
|
||||
## Deviations / Notes
|
||||
|
||||
- **Two xfail tests** (C3, S1) — see "Regression Markers" above. Both are
|
||||
written against the expected post-fix behaviour. The spec lists these as
|
||||
"SOURCE REQ IDS" (not as "already in place" bugs), so the xfail is the
|
||||
correct way to land the regression tests now without modifying frozen
|
||||
source code.
|
||||
- **MCP stub**: the `app_client` fixture monkeypatches `get_mcp_client` in
|
||||
three modules (mcp_client, api, agent). Without this, api/agent would use
|
||||
their already-imported reference to the real factory and bypass the stub.
|
||||
- **Test location**: tests live at the repo root (`/tests/`) rather than
|
||||
inside `agent_platform/`, so `pip install -e .` from root works alongside
|
||||
the existing `pip install -e ./agent_platform` for the inner package.
|
||||
Reference in New Issue
Block a user