API Relay Audit

Local security audit for AI API relays and LLM proxies.

DSH Plugin

What Is API Relay Audit?

API Relay Audit is a local security audit tool for AI API relays and LLM proxies. It keeps API relay audit, prompt injection audit, model substitution signals, and Web3 relay audit as separate query families so each result keeps a clean evidence boundary. Your API key is sent only to the relay URL you choose.

Use it when you rely on a third-party AI API relay, OpenAI-compatible proxy, Claude-compatible proxy, or Web3 agent workflow and want a repeatable Markdown report before trusting that relay with production or wallet-related traffic.

AI API Relay Security Audit

• Detect relay tampering: prompt injection, prompt extraction, identity consistency signals, context truncation, tool-call rewriting, error-response leakage, and SSE stream anomalies.
• Run locally: the standalone audit.py uses only Python stdlib plus curl; your API key is sent only to the relay URL you choose.
• Produce reviewable evidence: each run generates a structured Markdown report with per-step findings and a final LOW / MEDIUM / HIGH verdict.

Quick Start

AUDIT_SCRIPT_REF=v2.4.0
curl -fsSL "https://raw.githubusercontent.com/toby-bridges/api-relay-audit/${AUDIT_SCRIPT_REF}/audit.py" -o audit.py

python audit.py --key <YOUR_KEY> --url <BASE_URL> --output report.md

# Web3 / wallet users
python audit.py --key <YOUR_KEY> --url <BASE_URL> --profile web3 --output report.md

See a public-safe fixture report: sanitized audit report.
Use master as AUDIT_SCRIPT_REF only when intentionally testing unreleased changes.

If API Relay Audit helps you evaluate a relay before sending real traffic, star the repository to follow new detector coverage and release-tested updates.

When to Use It

• You use a third-party AI API relay, mirror, gateway, or LLM proxy.
• You want to check whether a Claude-compatible or OpenAI-compatible proxy injects prompts, swaps models, truncates context, or rewrites tool output.
• You are testing relay behavior before production traffic, coding-agent automation, package-install suggestions, or wallet-related actions.
• You need a local, repeatable audit report instead of a web tool that asks for your API key.

What It Does Not Claim

• It does not certify that a relay is safe.
• It does not replace manual security review or operational monitoring.
• It does not treat inconclusive as clean; blocked probes and ambiguous responses stay visible in the report.

Query Family Boundaries

The canonical contract lives in docs/query-families.md. README headings, Pages cards, issue templates, and skill descriptions should preserve these boundaries instead of flattening them into one slogan.

Coverage

API Relay Audit checks whether a relay modifies the request or response path between you and the model:

• Prompt safety: token injection, prompt extraction, instruction override, jailbreak resistance
• Relay integrity: context truncation, tool-call substitution, error leakage, stream integrity
• Model identity: non-Claude identity leaks, model substitution signals, Claude/OpenAI-compatible relay behavior
• Web3 wallet safety: transfer guidance, signed-transaction refusal, private-key refusal

Audit LLM Proxies Locally

The project has two distribution modes:

• audit.py: zero-dependency standalone script for quick local audits
• api_relay_audit/ plus scripts/: modular development version with tests

Runtime profiles:

• general: default AI API relay and LLM proxy checks
• web3: wallet-safety probes for Web3 agent flows
• full: general plus Web3 checks

DeepSeek Harness DSH Plugin

The repository is also an installable dsh-api-relay-audit bundle for
DeepSeek Harness Web and
community TUI surfaces that use the official @deepseek-ai/dsh-commands
registry. Pin an immutable commit or release tag:

DSH_PLUGIN_REF=v2.4.0
dsh plugin --profile web add "github:toby-bridges/api-relay-audit#${DSH_PLUGIN_REF}"

# dsh-cc-tui and other compatible profile-based clients
dsh plugin --profile cc-tui add "github:toby-bridges/api-relay-audit#${DSH_PLUGIN_REF}"

The command reuses the current DSH provider's baseURL, model, and credential
reference. The credential stays in DSH Credentials and is delivered to the
local audit process through an environment variable, never through command
arguments or the session log:

/relay-audit
/relay-audit --connectivity
/relay-audit --profile web3 --fast-context
/relay-audit --url <URL> --model <claude-model> --credential-ref <DSH_CREDENTIAL_REF>

No arguments preserves the existing full-audit default and may consume
metered tokens. Use --connectivity for a lower-cost check. This distribution
does not add a new model baseline: the selected route must identify as Claude,
although the relay API itself may be Anthropic-compatible or OpenAI-compatible.
Independent wrappers without DSH profiles and the DSH command registry are not
compatible with this bundle. See agent distribution notes.
The exact v2.4.0 installation, runtime, and secret-scan results are recorded in
the DSH distribution verification.

Retained Agent Skill Files

The repository retains its existing OpenClaw and Hermes skill files for direct
users and downstream compatibility. They are not current registry distribution
targets; active distribution and release verification focus on the DeepSeek
Harness plugin.

• OpenClaw Skill: run a local AI API relay audit before an OpenClaw agent
  depends on a third-party relay, proxy API, or resale key.
• Hermes Skill: install API Relay Audit as a Hermes Agent skill and run the
  same local 14-step LLM proxy security audit from an agent workflow.

These files do not certify that a relay is safe. They help agents generate a
local, reviewable Markdown report before trusting a relay path.

Evidence Boundaries

Natural-language self-identification is treated as a consistency signal, not upstream proof. A response saying it is Qwen, DeepSeek, GPT, or Claude can indicate a mismatch, but it does not by itself prove that a provider substituted the upstream model.

Stronger claims require corroborating evidence such as raw response JSON, request IDs, provider/model metadata, stream signatures, transparent-log hashes, and reproducible runs. Public submissions should use redacted report artifacts and never include API keys, raw headers, full response bodies, wallet material, private relay traffic, or user data.

Web3 Wallet Safety Checks

With --profile web3 or --profile full, API Relay Audit adds wallet-oriented prompt injection probes inspired by signature-isolation risks:

• ETH transfer guidance checks
• Signed-transaction refusal checks
• Private-key leak refusal checks

These probes are model-agnostic, but they are intentionally profile-gated so general relay audits stay focused.

Working Model

your machine
  -> audit.py / scripts/audit.py
  -> chosen relay endpoint
  -> Markdown report + optional hash-only transparent log
  -> optional: redacted evidence issue for maintainer review

Community evidence is shape-checked by GitHub Actions, but publication still requires maintainer review. Operators keep a separate response path, and sensitive vulnerabilities belong in the disclosure path described in SECURITY.md.

Project Status

Example Report And Live Page

• GitHub Pages: toby-bridges.github.io/api-relay-audit
• Chinese landing page: toby-bridges.github.io/api-relay-audit/zh/
• Example report: sanitized fixture report
• Guides:
  AI API relay / LLM proxy,
  Claude relay audit,
  tool comparison,
  prompt injection in proxies,
  Web3 wallet prompt injection,
  DeepSeek Harness plugin
• Contributors / Credits: CONTRIBUTORS.md
• Security policy: SECURITY.md
• Contributing guide: CONTRIBUTING.md
• Social: X @li9292

FAQ

What is an API relay or LLM proxy?

An API relay or LLM proxy is a third-party service between you and an AI provider such as Anthropic or OpenAI. It forwards your requests upstream, but it can also inject hidden instructions, swap models, truncate context, rewrite tool output, or leak credentials in error responses.

Is it safe to enter my API key?

API Relay Audit runs locally, so your API key is sent only to the relay URL you specify. The standalone version is a single Python file with zero Python package dependencies, which makes it easier to inspect before running.

What does prompt injection mean here?

Prompt injection means the relay may prepend or insert hidden instructions into your request. API Relay Audit compares expected and actual token usage, tries prompt-extraction probes, and records evidence when the relay appears to add or reveal hidden prompt content.

What is model substitution?

Model substitution means the relay claims to provide one model but may expose evidence signals for another model identity, route, or upstream channel. API Relay Audit checks non-Claude identity patterns, anchor phrases, stream model identity, latency variance, and channel evidence where available; those signals require corroboration before making provider-level claims.

What is tool-call rewriting?

Tool-call rewriting means the relay modifies package-install commands or tool-like output in the model response. API Relay Audit sends pinned package commands and compares the returned text to detect proxy-layer supply-chain tampering.

What are SSE anomalies?

SSE anomalies are stream-level integrity issues in Anthropic-style streaming responses. API Relay Audit checks event types, usage monotonicity, thinking signatures, and stream model identity when the relay supports that format.

What Web3 wallet risks does it check?

With the web3 or full profile, API Relay Audit checks transfer guidance, signed-transaction refusal, and private-key refusal behavior before wallet-related traffic is trusted.

What does inconclusive mean?

Inconclusive means the tool could not determine a clean or anomalous result for that step. A blocked probe, unsupported format, or ambiguous response is not treated as safe; it remains visible in the final report.

How does this compare with hvoy.ai or cctest.ai?

They serve different needs. hvoy.ai is useful for relay reputation lookup, cctest.ai focuses on one-click testing and channel fingerprinting, and API Relay Audit focuses on local, open-source, repeatable security auditing with structured Markdown reports.

License

AGPL-3.0-only. See LICENSE.

This keeps modified network-service deployments accountable to the same public source-availability standard as the relay ecosystem evidence we audit.

Citation

If you use API Relay Audit in research, security reports, or public relay evaluations, please cite the software with CITATION.cff. The citation file also records the two academic papers that inform the audit model: Liu et al., Your Agent Is Mine (arXiv:2604.08407) and Zhang et al., Real Money, Fake Models (arXiv:2603.01919).

How to Contribute

You do not need to write code to help. Good first contributions are small,
reproducible, and evidence-focused:

• Report a detector gap with a sanitized reproduction.
• Share local run feedback for install, runtime, platform, or report-UX issues.
• Add documentation examples for profiles, flags, or relay behavior.
• Improve DSH, OpenClaw, or Hermes install notes from a real local setup.
• Translate Quick Start or clarify clean, anomaly, and inconclusive.

Start with:

• Local Run Feedback
• Detector Gap
• Documentation Example
• Agent Skill Feedback
• Community Evidence Guide
• CONTRIBUTING.md

Avoid publishing real API keys or private relay traffic, and keep changes scoped
to one behavior or document.

Star History