Research Agent Harness

Build agents that research, reason, trace, and cite.

Research Agent Harness is a plugin-driven agent harness for multi-source research and
evidence-grounded answers. It provides the execution loop, tool orchestration,
durable state, streaming events, citation validation, and a DeepSeek-inspired
web interface that an agentic RAG application needs in production.

It is designed as a foundation rather than a single-purpose chatbot: add a
retrieval connector, describe its routing metadata, and let the harness expose
it through the same traceable tool protocol.

Why Research Agent Harness

Most RAG demos stop at “retrieve some chunks and ask a model to summarize
them.” Research Agent Harness treats the surrounding runtime as a first-class system:

• the Agent Harness controls turns, tool budgets, timeouts, concurrency,
  retries, graceful finalization, and model protocol adapters;
• the Evidence Ledger normalizes source records into immutable evidence
  objects instead of trusting prose returned by a tool;
• the Citation Compiler parses OpenAI's official custom-retrieval markers,
  validates source references, and emits output_text URL annotations;
• the Trajectory Ledger persists reasoning, assistant output, tool calls,
  tool results, and terminal state for reconnectable UI rendering;
• the Cordis plugin layer keeps source connectors and product features
  independently addable and removable.

The result is an agent whose answers are inspectable, resumable, and tied to
the records that support them.

Capabilities

• bounded agent loop with Anthropic Messages and OpenAI-compatible adapters;
• JSON Schema tool contracts with validation, timeouts, and concurrency limits;
• progressively disclosed SKILL.md workflows;
• OpenAlex, official arXiv, and NCBI PubMed retrieval tools;
• append-only SQLite event log with user-bound sessions and background runs;
• reconnectable SSE with cursor replay (Last-Event-ID / after);
• server-owned citation numbering, validation, ACL hooks, and hover metadata;
• standard and deep-research orchestration modes;
• optional deep-thinking policy for more deliberate evidence analysis;
• expandable reasoning and tool traces, session history, and trajectory views;
• graceful partial-answer finalization instead of user-visible budget failures;
• React/Cordis frontend with plugin-based UI composition.

Architecture

User / API / UI ── POST run ──► RunManager (browser-independent task)
      ▲                              │
      │ reconnectable SSE           ▼
      └──── seq cursor ───── append-only events / SQLite
                                     │
                                     ▼
                                AgentHarness
      │
      ├── ContextCompiler ──► skill catalog + citation protocol
      │
      ├── LLM adapter ──────► Anthropic or OpenAI-compatible endpoint
      │          │
      │          ▼ tool calls
      ├── ToolRegistry ─────► validation / timeout / concurrency
      │          │
      │          ├── load_skill
      │          ├── openalex_search
      │          ├── arxiv_search
      │          └── ncbi_pubmed_search
      │
      ├── Evidence Ledger ──► normalized, immutable source records
      │
      └── CitationCompiler ─► output_text.text + url_citation annotations

Retrieval results are exposed to the model as block-level sources with stable IDs such as turn0block0 and OpenAI's recommended private Citation Marker grammar (\uE200cite\uE202turn0block0\uE201). The server parses and validates those markers, removes them before display, and returns clean output_text.text plus OpenAI-compatible url_citation annotations. Known provider deviations such as [cite]turn0block0[/cite] are normalized only at this parser boundary; invalid or invented IDs remain rejected, and neither syntax is exposed through the public response or stream. The browser renders annotations directly from the typed message; no citation placeholder or global registry is part of the public protocol. See the official citation-formatting guide and web-search citation output.

Agent modes

The composer exposes two complementary controls:

• Standard keeps retrieval adaptive and favors a concise answer when the
  question does not require external research;
• Deep Research forces source retrieval, gathers multiple relevant records,
  and asks the harness to reconcile evidence before answering;
• Deep Thinking is an independent harness policy that asks the model to
  spend more effort planning, checking evidence, and resolving contradictions.

Deep Research and Deep Thinking are explicit per-run policies, not hidden
prompt conventions. They can be combined, and both add observable metadata to
the durable run_started event.

Frontend stack

The frontend follows the DeepSeek Harness web stack: React 18, TypeScript 6,
Vite 6, Zustand 4.4.7, Immer 10.1.1, TanStack Virtual, pnpm, and CSS Modules.
The checked-in production bundle under app/static includes the resizable
three-column AppFrame, project/workspace grouping, a separate Recent section
for unassociated conversations, semantic conversation nodes, expandable tool
rows, a shared details inspector, and the filtered/virtualized trajectory
timeline.
It directly uses
the MIT-licensed @deepseek-ai/dsh-client-ui-primitives package for Markdown,
code highlighting, disclosure rows, JSON inspection, hover cards, icons, and
state indicators. The FastAPI/SSE adapter and project persistence remain
application-specific; the display layer reuses DeepSeek Harness' Cordis
runtime, RPC contracts, Session object layer, and ui-workspace. The retained
local-workspace adapter is not exposed in the current SaaS profile.

DeepSeek names and brand assets are not used as product identity. Upstream
attribution and the MIT notice are in frontend/THIRD_PARTY_NOTICES.md.

Run

cd research-agent-harness
uv sync --extra dev
cp .env.example .env

Edit .env and set LLM_API_KEY. Do not commit .env.

The Cordis production bundle is checked in, so running the Python service does
not require a frontend build. frontend/ is the preserved pre-Cordis UI source
and is not the bundle served by the current application.

For the Alibaba Cloud Model Studio compatible endpoint used by this project:

LLM_BASE_URL=https://llm-7i80wcrjtcx6gbdo.cn-beijing.maas.aliyuncs.com/compatible-mode/v1
LLM_API_STYLE=openai
LLM_AUTH_MODE=auto
LLM_MODEL=deepseek-v4-flash-0731
LLM_API_KEY=your-key

Start the API and UI:

uv run uvicorn app.main:app --host 127.0.0.1 --port 8000 --timeout-graceful-shutdown 5

Open http://localhost:8000. API documentation is at /docs.

Run tests:

uv run pytest -q
cd frontend && pnpm test

Run deterministic unit/integration coverage first, then optional real-model evaluation cases against a running server:

uv run python scripts/run_evals.py --case rag_definition --case hyaluronic_earliest

The real suite reports completion, source-routing precision, citation validity, absence of fallback/budget failures, turns, tool calls, and per-case latency budgets. Case definitions live in evals/cases.yaml; the latest checked result is recorded in evals/REPORT.md.

API

Non-streaming chat

curl http://localhost:8000/api/chat \
  -H 'content-type: application/json' \
  -d '{
    "message": "Find recent papers about agentic RAG for scientific research",
    "require_sources": true,
    "run_mode": "research",
    "thinking_mode": true
  }'

run_mode accepts standard or research; thinking_mode is an optional
boolean. The browser composer stores these preferences locally and sends them
with the next run.

The response carries finalized text in content[0]:

{
  "content": [{
    "type": "output_text",
    "text": "A source-backed answer.",
    "annotations": [{
      "type": "url_citation",
      "start_index": 0,
      "end_index": 23,
      "url": "https://example.org/source",
      "title": "Source title"
    }]
  }]
}

Streaming chat

The durable protocol separates command submission from event delivery:

POST /api/runs
GET  /api/sessions/{session_id}/stream?user_id=...&after={last_seq}
GET  /api/sessions/{session_id}/events?user_id=...&after={last_seq}
GET  /api/runs/{run_id}?user_id=...
GET  /api/sessions?user_id=...
GET  /api/workspaces?user_id=...
POST /api/workspaces?user_id=...
PATCH|DELETE /api/workspaces/{workspace_id}?user_id=...

POST /api/runs returns 202 immediately. The Agent continues if the browser disconnects. Every SSE frame has an id equal to the durable SQLite event sequence. EventSource reconnects with Last-Event-ID; callers can also send after explicitly. History catch-up is subscribed-before-read and sequence-deduplicated, so an event cannot be lost in the handoff from replay to live delivery.

Persisted streaming events include reasoning_delta, assistant_delta, tool_call, tool_result, citation_validation, and terminal run_completed / run_failed. Delta writes are coalesced to reduce SQLite pressure without losing replay fidelity.

The compatibility endpoint POST /api/chat/stream still emits:

• event: persisted harness lifecycle events;
• result: the final ChatResponse;
• error: a terminal error;
• done: stream termination.

Unlike the older implementation, disconnecting this response does not cancel the run.

User and session binding

Run requests accept user_id and session_id. A session is permanently bound to the first user ID that creates it; cross-user reads, continuation, run inspection, and citation resolution return 403. These IDs establish application-level ownership, not authentication—production deployments should derive user_id from an authenticated principal rather than trusting request input.

workspace_id is optional. Sessions created inside a project persist that
association; sessions without it are shown under Recent. Deleting a project
removes only the workspace registration and moves its conversations to Recent;
conversation history is not deleted. Projects are SaaS-side conversation
groups and do not select or store a user directory. The underlying workspace
path contract remains available for a future local-workspace mode.

Citation detail

GET /api/answers/{answer_id}/citations/{citation_id}

This endpoint resolves citation metadata to the underlying evidence. In a real multi-user deployment, put authorization both in source adapters and in this endpoint before returning excerpts.

The default PublicEvidenceAuthorizer only permits evidence whose acl_ref is public. Internal connectors should set an application-specific ACL reference and inject an EvidenceAuthorizer into create_app that evaluates the authenticated request principal.

Adding a tool

Define a Pydantic input contract and an async handler returning ToolExecutionResult:

class SearchInput(BaseModel):
    query: str

async def search(input: BaseModel) -> ToolExecutionResult:
    request = SearchInput.model_validate(input)
    return ToolExecutionResult(content=f"Searched for {request.query}")

registry.register(ToolSpec(
    name="search",
    description="Search an internal source.",
    input_model=SearchInput,
    handler=search,
))

For a retrieval tool, normalize each source record into an Evidence object. The harness turns those records into official citable blocks centrally; connector-authored prose is not a citation protocol, and the Evidence object remains the authoritative audit record.

Adding a skill

Create skills/<name>/SKILL.md:

---
name: my-workflow
description: When and why the agent should use this workflow.
allowed_tools:
  - search
---
# Workflow

Detailed instructions loaded only when the agent calls `load_skill`.

Only skill metadata is placed in the initial system context. Full instructions are progressively disclosed through the load_skill tool.

Citation guarantees and limits

The built-in validator guarantees that:

• every emitted annotation references a citable source returned in the current run;
• model-invented source IDs are rejected;
• private model markers and raw source IDs are removed before display;
• annotation ranges are persisted with the clean output text;
• source excerpts, locators, hashes, and retrieval times remain available for audit;
• numeric claims are checked after locale/magnitude normalization (1,600, 1600, 4,400 万, 44 million);
• numeric claims without a nearby citation produce warnings.

The included semantic support check is intentionally conservative and lexical. For high-stakes deployments, add a dedicated entailment verifier or human review before changing partial citations to verified.

External API notes

• NCBI asks E-utilities clients to send an application name and email. Configure NCBI_EMAIL; add NCBI_API_KEY if you need higher request limits.
• OpenAlex is the fast general scholarly index. Temporal queries first form a relevance-ranked candidate set, then sort locally so weak ancient matches do not outrank relevant papers.
• The arXiv API asks clients to avoid rapid repeated calls. The adapter serializes calls and keeps a three-second interval.
• The UI should acknowledge arXiv data use if deployed publicly. This demo keeps all source names factual and does not imply endorsement.

Production hardening checklist

• replace the single-process SQLite connection with PostgreSQL for multiple workers;
• encrypt or redact sensitive event payloads;
• enforce connector-level and citation-detail ACLs;
• put shell/browser tools in an OS or container sandbox;
• add idempotency keys to every side-effecting tool;
• add per-user budgets and rate limiting;
• evaluate retrieval recall, citation coverage, citation entailment, latency, and recovery behavior.

Loop and planning behavior

• The final turn exposes no tools and explicitly asks the model to finish from retrieved evidence.
• Turn and tool-call budgets return the best available cited partial answer; they do not become HTTP 502 errors.
• Independent tool calls in one model response execute concurrently under a configurable semaphore.
• Single-source questions stop retrieval after one successful result set with at least three records; explicit multi-source questions may use up to MAX_RETRIEVAL_ROUNDS.
• Equivalent tool name/argument calls are fingerprinted and skipped within a run.
• Tool registrations carry routing metadata (kind, best_for, parallel_safe) that is compiled into the planning prompt, encouraging source selection rather than speculative fan-out.
• Invalid source IDs, missing citations, and numeric conflicts make citation validation fail. If the model returns no usable final text, the server returns a deterministic, citation-backed bibliographic fallback without adding model-memory claims.