gqt
766 lines
32 KiB
Markdown
766 lines
32 KiB
Markdown
# KBQA MVP Implementation Plan
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||
|
||
**Goal:** Build the approved local single-user dual-agent knowledge-base application from document upload through real-model RAG, SSE chat, persistent history, and browser-verified cleanup/recovery.
|
||
|
||
**Architecture:** A React three-column workbench talks to one FastAPI process over REST and POST SSE. SQLite is the business source of truth, a durable SQLite-backed single-consumer queue drives indexing, Milvus Lite stores vectors, and LangChain 1.x `create_agent` composes a main agent with a research subagent exposed as a tool.
|
||
|
||
**Tech Stack:** Python 3.12, uv, FastAPI, SQLAlchemy 2.x async, aiosqlite, Alembic, LangChain 1.x, langchain-openai, PyMilvus/Milvus Lite, LangSmith, React, TypeScript, Vite, React Router, zustand, Tailwind CSS, shadcn/ui, npm, pytest, agent-browser.
|
||
|
||
## Global Constraints
|
||
|
||
- SQLite is the only relational database; do not add PostgreSQL, psycopg, JSONB, Redis, Celery, or another queue service.
|
||
- Milvus Lite uses one local file and stores no duplicate chunk正文数据.
|
||
- Use `langchain.agents.create_agent`; do not use `langgraph.prebuilt.create_react_agent` or LangChain 0.3.x APIs.
|
||
- Both agents use `qwen3.5-flash`; embeddings use `text-embedding-v4` with exactly 1024 dimensions.
|
||
- Strict mode: every knowledge answer researches first; empty evidence returns a deterministic insufficiency response.
|
||
- Store user-visible sessions/messages/sources in SQLite; internal agent/tool messages stay in LangSmith.
|
||
- Index asynchronously with a durable SQLite job; restore queued/running/deleting work at startup.
|
||
- Run exactly one Uvicorn worker.
|
||
- Support only UTF-8 Markdown/TXT and text-extractable PDF; do not add OCR.
|
||
- Do not use TDD. Implement first, then smoke-check; add automated tests only in Task 12.
|
||
- Do not mock Chat or Embedding models. Model-path tests belong to the explicitly enabled live suite.
|
||
- Use `uv` for Python and `npm` for Node.js.
|
||
- Keep secrets only in ignored `.env`; rotate credentials currently present in `1.md` before live testing.
|
||
- Test the frontend with `agent-browser` against real Vite and FastAPI processes.
|
||
|
||
---
|
||
|
||
## Planned File Structure
|
||
|
||
```text
|
||
.
|
||
├── .python-version
|
||
├── .env.example
|
||
├── README.md
|
||
├── backend/
|
||
│ ├── pyproject.toml
|
||
│ ├── uv.lock
|
||
│ ├── alembic.ini
|
||
│ ├── alembic/{env.py,versions/0001_initial.py}
|
||
│ ├── src/kbqa/
|
||
│ │ ├── {main,config,database,lifecycle,models}.py
|
||
│ │ ├── api/{errors,middleware,router}.py
|
||
│ │ ├── health/routes.py
|
||
│ │ ├── documents/{models,schemas,repository,service,routes}.py
|
||
│ │ ├── indexing/{loaders,splitter,worker}.py
|
||
│ │ ├── rag/{types,embeddings,vector_store,retriever}.py
|
||
│ │ ├── chat/{models,schemas,repository,sse,service,routes}.py
|
||
│ │ └── agents/{context,llm,prompts,tools,middleware,factory}.py
|
||
│ └── tests/{fixtures,unit,integration,live}/
|
||
└── web/
|
||
├── package.json
|
||
├── vite.config.ts
|
||
├── src/{main.tsx,index.css}
|
||
├── src/app/{router,AppLayout}.tsx
|
||
├── src/lib/{api,types,sse}.ts
|
||
├── src/components/ui/
|
||
└── src/features/{documents,chat}/
|
||
```
|
||
|
||
---
|
||
|
||
### Task 1: Secure bootstrap and dependency lock
|
||
|
||
**Files:**
|
||
- Create: `.python-version`, `.env.example`, `README.md`
|
||
- Create: `backend/pyproject.toml`, `backend/src/kbqa/__init__.py`
|
||
- Modify: `.gitignore`
|
||
- Remove after credential rotation: `1.md`
|
||
|
||
**Interfaces:**
|
||
- Produces: Python 3.12 environment and environment-variable contract used by every later task.
|
||
|
||
- [ ] **Step 1: Rotate exposed credentials**
|
||
|
||
Rotate DashScope and LangSmith keys in their provider consoles. Put only new values in local `.env`. Confirm `git check-ignore -v .env 1.md`, then remove `1.md`; never commit or copy its old values.
|
||
|
||
- [ ] **Step 2: Pin Python and define dependencies**
|
||
|
||
Create `.python-version` containing `3.12`. Create `backend/pyproject.toml` with Python `>=3.12,<3.13` and these bounded dependencies: `fastapi`, `uvicorn[standard]`, `sqlalchemy[asyncio]>=2,<3`, `aiosqlite`, `alembic`, `pydantic-settings`, `python-multipart`, `pypdf`, `langchain>=1,<2`, `langchain-openai>=1,<2`, `langchain-text-splitters>=1,<2`, and `pymilvus[milvus-lite]>=2.6,<3`. Add dev dependencies `httpx`, `pytest`, `pytest-asyncio`, `pytest-timeout`, and `ruff`. Configure Hatchling for `src/kbqa`, pytest marker `live`, Ruff Python 3.12 and line length 100.
|
||
|
||
- [ ] **Step 3: Define safe configuration examples**
|
||
|
||
Create `.env.example` with every variable from Spec section 16. Leave keys empty; use `KBQA_` for application config and standard `LANGSMITH_` names for LangSmith.
|
||
|
||
- [ ] **Step 4: Lock and synchronize**
|
||
|
||
```bash
|
||
uv lock --project backend
|
||
uv sync --project backend --all-groups
|
||
uv run --project backend python -c "import langchain; print(langchain.__version__)"
|
||
```
|
||
|
||
Expected: dependency resolution succeeds and LangChain reports `1.x`.
|
||
|
||
- [ ] **Step 5: Document local commands**
|
||
|
||
`README.md` must cover setup, migration, backend/frontend startup, ordinary/live checks, credential rotation, single-worker constraint, and state that the approved Spec is authoritative while `architecture.md` is reference-only.
|
||
|
||
- [ ] **Step 6: Commit**
|
||
|
||
```bash
|
||
git add .python-version .env.example README.md .gitignore backend
|
||
git commit -m "chore: bootstrap kbqa workspace"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 2: FastAPI foundation, settings, database, and health
|
||
|
||
**Files:**
|
||
- Create: `backend/src/kbqa/config.py`, `database.py`, `lifecycle.py`, `main.py`
|
||
- Create: `backend/src/kbqa/api/errors.py`, `middleware.py`, `router.py`
|
||
- Create: `backend/src/kbqa/health/routes.py`
|
||
|
||
**Interfaces:**
|
||
- Produces: `Settings`, `get_settings()`, `Database`, `get_session()`, `AppError`, `api_router`, `create_app()`.
|
||
|
||
- [ ] **Step 1: Implement typed settings**
|
||
|
||
Define all Spec defaults with `SettingsConfigDict(env_file=".env", env_prefix="KBQA_", extra="ignore")` and:
|
||
|
||
```python
|
||
@lru_cache
|
||
def get_settings() -> Settings: ...
|
||
|
||
def validate_live_settings(settings: Settings) -> None: ...
|
||
```
|
||
|
||
Only model paths call `validate_live_settings`; app import and `/health` must not require credentials.
|
||
|
||
- [ ] **Step 2: Implement SQLite async infrastructure**
|
||
|
||
Create `Base(DeclarativeBase)` and a `Database` holding `AsyncEngine` plus `async_sessionmaker[AsyncSession]`. A SQLAlchemy connect listener executes `PRAGMA foreign_keys=ON`, `PRAGMA journal_mode=WAL`, and `PRAGMA busy_timeout=5000`.
|
||
|
||
```python
|
||
class Database:
|
||
def __init__(self, url: str) -> None: ...
|
||
async def dispose(self) -> None: ...
|
||
|
||
async def get_session(request: Request) -> AsyncIterator[AsyncSession]: ...
|
||
```
|
||
|
||
- [ ] **Step 3: Implement safe errors and request IDs**
|
||
|
||
Define `AppError(code, message, status_code, retryable=False)`. Handle it, `RequestValidationError`, and unexpected exceptions using the approved JSON envelope. Middleware accepts/generates `X-Request-ID`, stores it on `request.state`, and returns it in headers.
|
||
|
||
- [ ] **Step 4: Implement app factory and health**
|
||
|
||
`create_app()` builds resources in lifespan, configures only explicit localhost CORS origins, includes `/api/v1`, and exposes `/health`. Before Task 4 exists, health may use a minimal `MilvusClient(uri).list_collections()` probe; Task 4 replaces that probe with `MilvusVectorStore.ping()`. It never calls Chat/Embedding.
|
||
|
||
- [ ] **Step 5: Smoke verify**
|
||
|
||
```bash
|
||
uv run --project backend ruff check backend/src
|
||
uv run --project backend uvicorn kbqa.main:app --app-dir backend/src --port 8000
|
||
curl http://127.0.0.1:8000/health
|
||
```
|
||
|
||
Expected: lint passes and health reports `app`, `sqlite`, and `milvus` states.
|
||
|
||
- [ ] **Step 6: Commit**
|
||
|
||
```bash
|
||
git add backend/src/kbqa
|
||
git commit -m "feat: add backend application foundation"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 3: SQLite models, repositories, and initial migration
|
||
|
||
**Files:**
|
||
- Create: `backend/src/kbqa/documents/models.py`, `repository.py`
|
||
- Create: `backend/src/kbqa/chat/models.py`, `repository.py`
|
||
- Create: `backend/src/kbqa/models.py`
|
||
- Create: `backend/alembic.ini`, `backend/alembic/env.py`, `backend/alembic/versions/0001_initial.py`
|
||
|
||
**Interfaces:**
|
||
- Produces ORM classes `Document`, `IndexJob`, `Chunk`, `ChatSession`, `Message`, `MessageSource` and their repositories.
|
||
|
||
- [ ] **Step 1: Implement exact ORM schema**
|
||
|
||
Use SQLAlchemy 2.x `Mapped[...]`. Reproduce every field, check/unique constraint, cascade, and index from Spec sections 6.1–6.6. UUID strings are generated in Python; timestamps use one UTC helper.
|
||
|
||
- [ ] **Step 2: Implement document repository**
|
||
|
||
Define the input contract in `documents/repository.py`:
|
||
|
||
```python
|
||
@dataclass(frozen=True, slots=True)
|
||
class StoredUpload:
|
||
document_id: str
|
||
filename: str
|
||
stored_path: str
|
||
file_type: str
|
||
mime_type: str
|
||
file_size: int
|
||
sha256: str
|
||
```
|
||
|
||
```python
|
||
class DocumentRepository:
|
||
async def create_pending(self, upload: StoredUpload) -> Document: ...
|
||
async def page(self, status: str | None, offset: int, limit: int) -> tuple[list[Document], int]: ...
|
||
async def get(self, document_id: str) -> Document | None: ...
|
||
async def page_chunks(self, document_id: str, offset: int, limit: int) -> tuple[list[Chunk], int]: ...
|
||
async def claim_job(self, job_id: str) -> IndexJob | None: ...
|
||
async def mark_indexed(self, document_id: str, chunk_count: int) -> None: ...
|
||
async def mark_failed(self, document_id: str, code: str, message: str) -> None: ...
|
||
async def requeue_unfinished(self) -> list[str]: ...
|
||
```
|
||
|
||
`claim_job` atomically changes queued→running and pending→indexing.
|
||
|
||
- [ ] **Step 3: Implement chat repository**
|
||
|
||
Define the persistence-only source input in `chat/repository.py`:
|
||
|
||
```python
|
||
@dataclass(frozen=True, slots=True)
|
||
class MessageSourceInput:
|
||
document_id: str
|
||
chunk_id: str
|
||
citation_label: str
|
||
score: float
|
||
```
|
||
|
||
```python
|
||
class ChatRepository:
|
||
async def create_session(self, title: str | None) -> ChatSession: ...
|
||
async def page_sessions(self, offset: int, limit: int) -> tuple[list[ChatSession], int]: ...
|
||
async def load_recent_messages(self, session_id: str, limit: int) -> list[Message]: ...
|
||
async def page_messages(self, session_id: str, offset: int, limit: int) -> tuple[list[Message], int]: ...
|
||
async def begin_turn(self, session_id: str, query: str) -> tuple[Message, Message]: ...
|
||
async def complete_answer(self, message_id: str, content: str, sources: list[MessageSourceInput]) -> None: ...
|
||
async def fail_answer(self, message_id: str, error_code: str) -> None: ...
|
||
async def delete_session(self, session_id: str) -> bool: ...
|
||
```
|
||
|
||
`begin_turn` allocates consecutive order indexes and sets first-question title in one transaction.
|
||
|
||
- [ ] **Step 4: Configure Alembic**
|
||
|
||
Use async migration setup, import `kbqa.models`, and write `0001_initial.py` explicitly so names/cascades are reviewable.
|
||
|
||
- [ ] **Step 5: Verify migration**
|
||
|
||
```bash
|
||
uv run --project backend alembic -c backend/alembic.ini upgrade head
|
||
uv run --project backend python -c "import sqlite3; db=sqlite3.connect('data/kbqa.sqlite3'); print(sorted(r[0] for r in db.execute(\"select name from sqlite_master where type='table'\")))"
|
||
```
|
||
|
||
Expected: six application tables plus `alembic_version`.
|
||
|
||
- [ ] **Step 6: Commit**
|
||
|
||
```bash
|
||
git add backend/alembic.ini backend/alembic backend/src/kbqa
|
||
git commit -m "feat: add sqlite persistence model"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 4: RAG storage and durable document indexing
|
||
|
||
**Files:**
|
||
- Create: `backend/src/kbqa/rag/{types,embeddings,vector_store,retriever}.py`
|
||
- Create: `backend/src/kbqa/indexing/{loaders,splitter,worker}.py`
|
||
- Create: `backend/src/kbqa/documents/service.py`
|
||
- Modify: `backend/src/kbqa/lifecycle.py`
|
||
|
||
**Interfaces:**
|
||
- Produces: `SourceCandidate`, `PreparedChunk`, `EmbeddingProvider`, `MilvusVectorStore`, `KnowledgeRetriever`, `DocumentService`, `IndexWorker`.
|
||
|
||
- [ ] **Step 1: Define RAG contracts**
|
||
|
||
```python
|
||
@dataclass(frozen=True, slots=True)
|
||
class SourceCandidate:
|
||
chunk_id: str
|
||
document_id: str
|
||
filename: str
|
||
order_index: int
|
||
page_number: int | None
|
||
content: str
|
||
score: float
|
||
|
||
@dataclass(frozen=True, slots=True)
|
||
class PreparedChunk:
|
||
id: str
|
||
document_id: str
|
||
order_index: int
|
||
page_number: int | None
|
||
content: str
|
||
|
||
@dataclass(frozen=True, slots=True)
|
||
class VectorHit:
|
||
chunk_id: str
|
||
document_id: str
|
||
order_index: int
|
||
score: float
|
||
```
|
||
|
||
- [ ] **Step 2: Implement loaders and splitter**
|
||
|
||
Validate `%PDF-`, UTF-8 text, and non-empty extracted PDF text. Preserve page numbers. Split with `RecursiveCharacterTextSplitter(chunk_size, chunk_overlap, length_function=len)` and generate deterministic UUIDv5 IDs from `document_id:order_index`.
|
||
|
||
- [ ] **Step 3: Implement real embedding provider**
|
||
|
||
Wrap `OpenAIEmbeddings` with DashScope base URL/key, `text-embedding-v4`, and `dimensions=1024`:
|
||
|
||
```python
|
||
class EmbeddingProvider:
|
||
async def embed_documents(self, texts: list[str]) -> list[list[float]]: ...
|
||
async def embed_query(self, text: str) -> list[float]: ...
|
||
```
|
||
|
||
Validate every returned vector has length 1024 and map upstream failures to stable `AppError` codes.
|
||
|
||
- [ ] **Step 4: Implement Milvus Lite wrapper**
|
||
|
||
Create `kbqa_chunks` with VARCHAR `chunk_id` primary key, VARCHAR `document_id`, INT64 `order_index`, FLOAT_VECTOR `vector`, AUTOINDEX, and COSINE.
|
||
|
||
```python
|
||
class MilvusVectorStore:
|
||
def ensure_collection(self) -> None: ...
|
||
def upsert(self, chunks: list[PreparedChunk], vectors: list[list[float]]) -> None: ...
|
||
def search(self, vector: list[float], limit: int) -> list[VectorHit]: ...
|
||
def delete_document(self, document_id: str) -> None: ...
|
||
def ping(self) -> None: ...
|
||
```
|
||
|
||
Run blocking PyMilvus calls with `asyncio.to_thread` at service boundaries.
|
||
|
||
- [ ] **Step 5: Implement retriever**
|
||
|
||
`KnowledgeRetriever.retrieve(query, top_k)` embeds the query, requests extra Milvus candidates, batch-loads SQLite chunks joined to indexed documents, preserves ranking, and returns at most `top_k` sources.
|
||
|
||
- [ ] **Step 6: Implement durable worker**
|
||
|
||
`IndexWorker.start()` requeues unfinished jobs and starts one task. `_process()` claims, clears old chunks/vectors, loads/splits, calls real embeddings, writes chunks, upserts vectors, and marks success. Catch ordinary exceptions to mark failed; re-raise `CancelledError`. `stop()` cancels and awaits the worker.
|
||
|
||
- [ ] **Step 7: Implement upload/deletion service**
|
||
|
||
`DocumentService.upload()` validates 20 MiB, extension/MIME/content, atomically writes `data/raw/{uuid}.{ext}`, creates document/job, enqueues, and returns. `delete()` marks deleting, idempotently removes vectors/file, then deletes SQLite rows. Startup recovery resumes deleting records and treats absent external data as success.
|
||
|
||
- [ ] **Step 8: Verify without model calls and commit**
|
||
|
||
Run Ruff and a script that creates the SQLite schema/Milvus collection but never calls embeddings.
|
||
|
||
```bash
|
||
git add backend/src/kbqa
|
||
git commit -m "feat: add durable document indexing pipeline"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 5: Document HTTP API
|
||
|
||
**Files:**
|
||
- Create: `backend/src/kbqa/documents/schemas.py`, `routes.py`
|
||
- Modify: `backend/src/kbqa/api/router.py`
|
||
|
||
**Interfaces:**
|
||
- Produces: all Spec section 12.1 endpoints and stable `DocumentResponse`, `ChunkResponse`, and `Page[T]` shapes.
|
||
|
||
- [ ] **Step 1: Define response schemas**
|
||
|
||
Use Pydantic v2 `from_attributes=True`. `DocumentResponse` includes status/error/chunk count/timestamps; `ChunkResponse` includes order/page/content; `Page` includes `items`, `total`, `offset`, `limit`.
|
||
|
||
- [ ] **Step 2: Implement all document routes**
|
||
|
||
Implement upload/list/detail/chunks/retry/delete. Enforce `offset >= 0` and `1 <= limit <= 100`. Return 202 for upload/retry, 204 delete, 404 missing, 409 invalid retry state, 413 size, and 415 format.
|
||
|
||
- [ ] **Step 3: Smoke-check non-model branches over real HTTP**
|
||
|
||
Start Uvicorn and use curl to list, reject an unsupported extension, return 404 for a random UUID, and verify request IDs. Do not upload a valid file until Task 12 live verification.
|
||
|
||
- [ ] **Step 4: Commit**
|
||
|
||
```bash
|
||
git add backend/src/kbqa/documents backend/src/kbqa/api/router.py
|
||
git commit -m "feat: expose document management api"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 6: Session persistence and history API
|
||
|
||
**Files:**
|
||
- Create: `backend/src/kbqa/chat/schemas.py`, `routes.py`
|
||
- Modify: `backend/src/kbqa/api/router.py`
|
||
|
||
**Interfaces:**
|
||
- Produces: create/list/delete sessions and paginated message history; reserves `/chat/stream` for Task 8.
|
||
|
||
- [ ] **Step 1: Define session/message/source schemas**
|
||
|
||
`MessageResponse` includes `status`, safe `error_code`, and sources joined from live documents/chunks. Parse citation labels missing source rows into `deleted_source_labels` so the frontend can render “来源已删除” without deleted content.
|
||
|
||
- [ ] **Step 2: Implement session routes**
|
||
|
||
Implement `POST /sessions`, `GET /sessions`, `GET /sessions/{id}/messages`, and `DELETE /sessions/{id}` with exact 201/200/204/404 behavior.
|
||
|
||
- [ ] **Step 3: Smoke-check restart persistence**
|
||
|
||
Create a session over HTTP, list it, restart Uvicorn, list/fetch it again, delete it, then verify it no longer exists.
|
||
|
||
- [ ] **Step 4: Commit**
|
||
|
||
```bash
|
||
git add backend/src/kbqa/chat backend/src/kbqa/api/router.py
|
||
git commit -m "feat: add persistent chat sessions"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 7: LangChain 1.x dual-agent core
|
||
|
||
**Files:**
|
||
- Create: `backend/src/kbqa/agents/{context,llm,prompts,tools,middleware,factory}.py`
|
||
|
||
**Interfaces:**
|
||
- Produces: `AgentBundle.main_agent`, research agent, `research` tool, `search_knowledge_base` tool, and typed custom stream payloads.
|
||
- Consumes: `KnowledgeRetriever`, Settings, request/session metadata.
|
||
|
||
- [ ] **Step 1: Define runtime context**
|
||
|
||
```python
|
||
@dataclass(frozen=True, slots=True)
|
||
class ResearchContext:
|
||
retriever: KnowledgeRetriever
|
||
request_id: str
|
||
|
||
@dataclass(frozen=True, slots=True)
|
||
class ChatRunContext:
|
||
retriever: KnowledgeRetriever
|
||
request_id: str
|
||
session_id: str
|
||
assistant_message_id: str
|
||
```
|
||
|
||
Tool artifacts are serialized `SourceCandidate` lists; never use a global mutable collector.
|
||
|
||
- [ ] **Step 2: Implement model factories**
|
||
|
||
Build `ChatOpenAI` with DashScope-compatible URL/key, `qwen3.5-flash`, async streaming, timeout, and bounded retries. Instantiate lazily, not at import. Add LangSmith `main-agent`/`research-agent` tags and request/session/message metadata per invocation.
|
||
|
||
- [ ] **Step 3: Implement search tool**
|
||
|
||
Use `@tool(response_format="content_and_artifact")`. Accept `query: str` plus injected `ToolRuntime[ResearchContext]`, call `retrieve(query, top_k=5)`, and return readable text plus complete source artifact.
|
||
|
||
- [ ] **Step 4: Build research agent**
|
||
|
||
Use `langchain.agents.create_agent(..., tools=[search_knowledge_base], context_schema=ResearchContext)`. Limit each run to three searches. Prompt it to use only tool evidence, reformulate weak searches, and return a research summary rather than a direct user answer.
|
||
|
||
- [ ] **Step 5: Implement main-agent research tool**
|
||
|
||
The `research(question, runtime)` tool invokes the research agent with `ainvoke`, extracts every search ToolMessage artifact, deduplicates by chunk ID keeping the highest score, caps at eight, emits custom `sources` or `no_evidence` through the stream writer, and returns the summary plus `[S1]…[S8]` evidence.
|
||
|
||
- [ ] **Step 6: Enforce research-before-answer**
|
||
|
||
Implement LangChain 1.x `AgentMiddleware.awrap_model_call`. Before a successful research ToolMessage exists in the current turn, override the model with required tool choice; afterward allow the final answer. Bound model calls and reject a second research call.
|
||
|
||
- [ ] **Step 7: Build main agent**
|
||
|
||
Use `create_agent` with only `research`, strict prompt, middleware, and `ChatRunContext`. The prompt allows only supplied labels and requires inline citations. Return both compiled graphs in:
|
||
|
||
```python
|
||
@dataclass(frozen=True, slots=True)
|
||
class AgentBundle:
|
||
main_agent: CompiledStateGraph
|
||
research_agent: CompiledStateGraph
|
||
```
|
||
|
||
- [ ] **Step 8: Static-check without calling the model**
|
||
|
||
Inspect the compiled agents: main exposes only `research`, researcher exposes only `search_knowledge_base`, and repository search finds no `create_react_agent` import.
|
||
|
||
- [ ] **Step 9: Commit**
|
||
|
||
```bash
|
||
git add backend/src/kbqa/agents
|
||
git commit -m "feat: add langchain dual-agent workflow"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 8: SSE chat orchestration and endpoint
|
||
|
||
**Files:**
|
||
- Create: `backend/src/kbqa/chat/sse.py`, `service.py`
|
||
- Modify: `backend/src/kbqa/chat/routes.py`, `backend/src/kbqa/lifecycle.py`
|
||
|
||
**Interfaces:**
|
||
- Produces: `ChatService.stream(session_id, query, request_id)` yielding encoded SSE bytes and `POST /api/v1/chat/stream`.
|
||
|
||
- [ ] **Step 1: Implement SSE encoding**
|
||
|
||
```python
|
||
def encode_event(event: str, data: Mapping[str, object]) -> bytes: ...
|
||
def encode_ping() -> bytes: ...
|
||
```
|
||
|
||
Serialize compact single-line UTF-8 JSON, terminate each event with two newlines, and encode heartbeat as `: ping\n\n`.
|
||
|
||
- [ ] **Step 2: Implement per-session lock registry**
|
||
|
||
Key process-local locks by session UUID. Reject concurrent generation with 409 and always release in `finally`. This is valid only because the app runs one process.
|
||
|
||
- [ ] **Step 3: Implement stream lifecycle**
|
||
|
||
Begin the SQLite turn, load 20 recent messages, invoke `main_agent.astream` with `stream_mode=["messages", "custom", "updates"]` and `version="v2"`, then map:
|
||
|
||
- immediate `status: researching`;
|
||
- source artifact → `sources`, then `status: answering`;
|
||
- empty evidence → close agent stream and emit deterministic insufficiency token;
|
||
- final main-agent text chunks after answering begins → `token`;
|
||
- 15-second inactivity → ping;
|
||
- validated/persisted answer → `done`.
|
||
|
||
Only labels in the source set are valid. Unknown labels produce `INVALID_CITATION`, mark failed, and emit `error` instead of `done`.
|
||
|
||
- [ ] **Step 4: Handle disconnects and failures**
|
||
|
||
Propagate `CancelledError`, close the LangChain async generator, mark assistant failed, and never persist partial content. Convert DashScope timeout/upstream errors to safe SSE errors. Persist success before `done`.
|
||
|
||
- [ ] **Step 5: Expose POST route**
|
||
|
||
Validate non-empty bounded query. Return `StreamingResponse` with `text/event-stream`, `Cache-Control: no-cache`, `X-Accel-Buffering: no`, and request ID.
|
||
|
||
- [ ] **Step 6: Verify protocol without model calls**
|
||
|
||
Run Ruff and feed Chinese token/source/error/ping payloads through the encoder to verify exact framing.
|
||
|
||
- [ ] **Step 7: Commit**
|
||
|
||
```bash
|
||
git add backend/src/kbqa/chat backend/src/kbqa/lifecycle.py
|
||
git commit -m "feat: stream dual-agent answers over sse"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 9: React workbench foundation
|
||
|
||
**Files:**
|
||
- Create: Vite React TypeScript project under `web/`
|
||
- Create: `web/src/app/router.tsx`, `AppLayout.tsx`
|
||
- Create: `web/src/lib/api.ts`, `types.ts`, `sse.ts`
|
||
- Create/Generate: `web/src/components/ui/*`
|
||
- Modify: `web/vite.config.ts`, `web/src/index.css`
|
||
|
||
**Interfaces:**
|
||
- Produces: typed REST client, incremental POST-SSE parser, routes, responsive shell, and shared shadcn primitives.
|
||
|
||
- [ ] **Step 1: Scaffold and install**
|
||
|
||
Use `npm create vite@latest web -- --template react-ts`. Add React Router, zustand, lucide-react, Tailwind CSS, shadcn/ui dependencies, ESLint, Vitest, and button/dialog-drawer/badge/textarea/scroll-area/tooltip/skeleton primitives. Commit `package-lock.json`.
|
||
|
||
- [ ] **Step 2: Configure proxy and visual tokens**
|
||
|
||
Proxy `/api` and `/health` to `http://127.0.0.1:8000`. Implement the approved deep-green, warm-orange, off-white palette as CSS variables and use a distinct reading style for evidence text.
|
||
|
||
- [ ] **Step 3: Define shared types**
|
||
|
||
Mirror backend DTOs: `Page<T>`, `Document`, `Chunk`, `Session`, `Message`, `Source`, `ApiError`, and a discriminated SSE union for `status/sources/token/done/error`.
|
||
|
||
- [ ] **Step 4: Implement clients**
|
||
|
||
`apiRequest<T>()` parses unified errors and request ID. `streamSSE()` uses fetch POST + `ReadableStream`, retains incomplete UTF-8/lines across chunks, supports LF/CRLF, dispatches complete events, and accepts `AbortSignal`.
|
||
|
||
- [ ] **Step 5: Implement routes and shell**
|
||
|
||
Create `/chat`, `/chat/:sessionId`, `/documents`, `/documents/:documentId`. At ≥1100px show all columns; below 1100px evidence is a drawer; below 768px navigation is also a drawer.
|
||
|
||
- [ ] **Step 6: Static-check and commit**
|
||
|
||
```bash
|
||
npm --prefix web run lint
|
||
npm --prefix web run build
|
||
git add web
|
||
git commit -m "feat: add responsive react workbench"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 10: Document management frontend
|
||
|
||
**Files:**
|
||
- Create: `web/src/features/documents/api.ts`, `useDocuments.ts`
|
||
- Create: `web/src/features/documents/DocumentsPage.tsx`, `DocumentList.tsx`, `DocumentDetail.tsx`, `UploadDocument.tsx`
|
||
- Modify: `web/src/app/router.tsx`, `web/src/app/AppLayout.tsx`
|
||
|
||
**Interfaces:**
|
||
- Produces: upload, list/filter, two-second active-status polling, chunk pagination, retry, deletion confirmation, and selected-document evidence panel.
|
||
|
||
- [ ] **Step 1: Implement typed document API**
|
||
|
||
Expose `uploadDocument`, `listDocuments`, `getDocument`, `listChunks`, `retryDocument`, and `deleteDocument` with exact backend paths.
|
||
|
||
- [ ] **Step 2: Implement polling state**
|
||
|
||
Poll every two seconds only while a visible document is pending/indexing/deleting; stop on unmount and terminal states. Preserve selected document route across refresh.
|
||
|
||
- [ ] **Step 3: Build approved document UI**
|
||
|
||
Left shows navigation and status counts. Center shows upload, filters, filename/type/chunk count/status and retry/delete actions. Right shows metadata and paginated chunks. Never display stack traces.
|
||
|
||
- [ ] **Step 4: Smoke-check real non-model branches**
|
||
|
||
Run both services; reject an unsupported file, show empty state, navigate to a missing document, and check responsive drawers without valid upload/model calls.
|
||
|
||
- [ ] **Step 5: Check and commit**
|
||
|
||
```bash
|
||
npm --prefix web run lint
|
||
npm --prefix web run build
|
||
git add web/src
|
||
git commit -m "feat: add document management workspace"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 11: Chat, streaming, history, and evidence frontend
|
||
|
||
**Files:**
|
||
- Create: `web/src/features/chat/api.ts`, `store.ts`, `useChatStream.ts`
|
||
- Create: `web/src/features/chat/ChatPage.tsx`, `SessionList.tsx`, `MessageList.tsx`, `ChatComposer.tsx`, `EvidencePanel.tsx`
|
||
- Modify: `web/src/app/router.tsx`, `web/src/app/AppLayout.tsx`
|
||
|
||
**Interfaces:**
|
||
- Produces: persistent session navigation, one active stream, incremental answer, abort/retry, source selection, deleted-source labels, and history reload.
|
||
|
||
- [ ] **Step 1: Implement session API and store**
|
||
|
||
Expose create/list/delete sessions and message history. Store only current session ID, transient phase/text/sources/error, and selected evidence; server responses remain authoritative.
|
||
|
||
- [ ] **Step 2: Implement stream hook**
|
||
|
||
Create one `AbortController`; append tokens; replace sources on `sources`; update researching/answering phase; reload history on `done`; expose retry with original query on `error`; abort on route change/unmount.
|
||
|
||
- [ ] **Step 3: Build session and message surfaces**
|
||
|
||
Left lists sessions with new/delete. Center renders user/assistant messages, research phase, failed placeholders, citation chips, and fixed composer disabled during streaming.
|
||
|
||
- [ ] **Step 4: Build evidence panel**
|
||
|
||
Right lists current answer sources and expands one inline with filename, order/page, score, and full chunk. Deleted labels render muted “来源已删除” with no filename/content.
|
||
|
||
- [ ] **Step 5: Implement empty/responsive states**
|
||
|
||
With no indexed documents, guide to `/documents`. Evidence drawer must not destroy center conversation state on medium/mobile widths.
|
||
|
||
- [ ] **Step 6: Check and commit**
|
||
|
||
```bash
|
||
npm --prefix web run lint
|
||
npm --prefix web run build
|
||
git add web/src
|
||
git commit -m "feat: add streaming chat and evidence workspace"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 12: Post-implementation tests, real-model API suite, and agent-browser E2E
|
||
|
||
**Files:**
|
||
- Create: `backend/tests/conftest.py`
|
||
- Create: `backend/tests/fixtures/{knowledge.md,knowledge.txt,knowledge.pdf}`
|
||
- Create: `backend/tests/unit/{test_file_validation,test_splitter,test_sse}.py`
|
||
- Create: `backend/tests/integration/{test_sqlite_cascades,test_session_api}.py`
|
||
- Create: `backend/tests/live/test_live_pipeline.py`
|
||
- Create: `web/src/lib/sse.test.ts`
|
||
- Create: `docs/testing/live-test-report.md`, `docs/testing/screenshots/`
|
||
|
||
**Interfaces:**
|
||
- Produces: ordinary checks, explicitly gated real API/model verification, browser evidence, and final acceptance report.
|
||
|
||
- [ ] **Step 1: Add automated tests after implementation**
|
||
|
||
Test validation, deterministic chunk IDs, SQLite constraints/cascades, session persistence, SSE framing, and frontend incremental parsing. Do not patch/fake Chat or Embedding; ordinary tests avoid those paths.
|
||
|
||
- [ ] **Step 2: Create real knowledge fixtures**
|
||
|
||
Commit UTF-8 Markdown/TXT and a real text-layer PDF with a small unique fact set suitable for a first question, pronoun-based follow-up, and absent-fact question. Generate the PDF once; do not synthesize it during the test.
|
||
|
||
- [ ] **Step 3: Implement black-box live test**
|
||
|
||
Skip unless `KBQA_RUN_LIVE_TESTS=1`. Start real Uvicorn on an available localhost port with isolated SQLite/Milvus/raw paths; use `httpx` over HTTP to:
|
||
|
||
1. upload all three real files;
|
||
2. poll until indexed and assert positive chunk counts;
|
||
3. create a session and parse real SSE;
|
||
4. assert researching→sources→answering→token→done;
|
||
5. assert sources refer to uploaded files/retrievable chunks;
|
||
6. ask a pronoun follow-up and verify persisted history;
|
||
7. ask an absent fact and verify insufficiency;
|
||
8. restart Uvicorn and verify persistence;
|
||
9. delete documents/sessions and verify files, SQLite relationships, and vectors are no longer retrievable.
|
||
|
||
Assert structure/semantic boundaries, never exact model wording.
|
||
|
||
- [ ] **Step 4: Run ordinary verification**
|
||
|
||
```bash
|
||
uv run --project backend ruff format --check backend/src backend/tests
|
||
uv run --project backend ruff check backend/src backend/tests
|
||
uv run --project backend pytest backend/tests/unit backend/tests/integration -v
|
||
npm --prefix web run lint
|
||
npm --prefix web run test -- --run
|
||
npm --prefix web run build
|
||
```
|
||
|
||
Expected: all exit 0 without contacting DashScope.
|
||
|
||
- [ ] **Step 5: Run real live verification**
|
||
|
||
```bash
|
||
KBQA_RUN_LIVE_TESTS=1 uv run --project backend pytest backend/tests/live/test_live_pipeline.py -v -s
|
||
```
|
||
|
||
Expected: real Chat/Embedding calls appear in LangSmith and the black-box flow passes.
|
||
|
||
- [ ] **Step 6: Run agent-browser E2E**
|
||
|
||
Start one Uvicorn worker and Vite. Use the `agent-browser` skill to upload, observe status, create a session, stream a real answer, expand evidence, follow up, reload history, test desktop/narrow widths, delete data, and capture screenshots.
|
||
|
||
- [ ] **Step 7: Write final report**
|
||
|
||
Record commands, versions, timestamps, LangSmith run IDs/links without secrets, screenshots, results, and non-blocking model variability in `docs/testing/live-test-report.md`.
|
||
|
||
- [ ] **Step 8: Run forbidden-marker audit**
|
||
|
||
```bash
|
||
rg -n "postgres|psycopg|JSONB|create_react_agent|langchain.*0\.3|sk-|lsv2_" --glob '!architecture.md' --glob '!docs/superpowers/**' --glob '!.env'
|
||
```
|
||
|
||
Expected: no forbidden implementation or committed secret.
|
||
|
||
- [ ] **Step 9: Commit**
|
||
|
||
```bash
|
||
git add backend/tests web/src/lib/sse.test.ts docs/testing
|
||
git commit -m "test: verify real kbqa workflow"
|
||
```
|
||
|
||
---
|
||
|
||
## Execution Checkpoints
|
||
|
||
1. After Task 3: review schema/migration before indexing depends on it.
|
||
2. After Task 5: review document state machine, recovery, and deletion.
|
||
3. After Task 8: review dual-agent/SSE strictness before frontend coupling.
|
||
4. After Task 11: review complete UI with ordinary checks passing.
|
||
5. After Task 12: review LangSmith traces, browser screenshots, cleanup, and acceptance report.
|
||
|
||
## Execution Notes
|
||
|
||
- Execute with `superpowers:executing-plans`; `subagent-driven-development` is not available in this repository/session.
|
||
- Do not start live tests until exposed credentials are rotated.
|
||
- Never run Uvicorn with more than one worker.
|
||
- If an installed LangChain 1.x signature differs from a snippet, consult official docs/Context7, preserve this plan's interface/behavior, and record the verified signature in the implementing commit.
|