TeachReplay

Teach once, replay anywhere.

🌐 Project website

TeachReplay is a harness-agnostic Teach-by-Demonstration engine: show a workflow once — clicks, typed values, shell commands — and it becomes a reusable, parameterized skill that can be replayed later with different inputs and an explicit success/failure verdict on real computers.

What is TeachReplay?

Teach → Record → Compile → Replay → Verify

1. Teach — start a recording and demonstrate the task once on a computer.
2. Record — the recorder watches the computer's semantic state (URLs, visible text,
   element roles and values) and diffs it into a versioned trajectory. Password fields stay
   masked; every value passes secret redaction.
3. Compile — a deterministic compiler turns the trajectory into a parameterized skill:
   your demonstrated values become inputs with your values as defaults. GUI steps are grounded
   semantically (role + name, never raw coordinates); shell commands become real CLI steps
   with their recorded exit codes.
4. Replay — the engine re-snapshots per step, matches targets semantically, acts, and
   verifies each step's effect. One bounded retry, an optional model-assisted recovery hook,
   then explicit failure.
5. Verify — success requires the recorded condition (URL or confirmation text) to hold.
   "The engine stopped" is never success.

Package overview

Architecture

The core owns all Teach-by-Demonstration logic and depends on nothing but Node.js — an
enforced invariant (independence test). Adapters
contribute only backends, stores, and event sinks:

TeachReplay for DeepSeek Harness

@teachreplay/adapter-dsh adds teach-by-demonstration tools to
DeepSeek Harness agents:
teach_start / teach_stop / teach_compile / teach_replay /
teach_shell — the same Record → Compile → Replay → Verify engine, no core
logic duplicated.

• Targets the real DSH plugin API: @deepseek-ai/cordis /
  @deepseek-ai/dsh-tools are peerDependencies, pinned as
  devDependencies (4.0.1 / 0.1.1-rc.2, the exact pair DSH resolves at
  dsh-v0.1.1-rc.2) so this repo's own build compiles the plugin against
  the real defineTool/Plugin.Object types — no local shims. A prior
  DSH-native rewrite verified inside a full DSH workspace checkout (see the
  adapter README) is
  what this version is based on
• Installation: pnpm add @teachreplay/adapter-dsh — DSH's
  sub-packages are published to npm under the next dist-tag while DSH is
  in developer preview; see the adapter README
  (⚠️ DSH is in developer preview with compatibility-breaking changes;
  re-verify per release)
• Demo: examples/dsh-demo runs
  teach_start → demonstrate → teach_stop → teach_compile → change parameters → teach_replay → success locally — against the mock
  computer through DshTeachSession directly, not through DSH's real tool
  runtime

(DSH does not accept external pull requests while in developer preview —
see its own CONTRIBUTING.md — so this ships as an independent package
rather than an upstream PR.)

Quick start

git clone https://github.com/Ottohere-Mourn/TeachReplay.git
cd TeachReplay
pnpm install
pnpm build
pnpm demo        # standalone: record → compile → change parameters → replay → verify
pnpm test        # 50 tests, including the zero-harness-dependency invariant
pnpm typecheck

The demo runs entirely on the built-in demo computer — no credentials, no network, no harness.

Real evaluation

scripts/mini-benchmark.mjs runs 8 tasks / 14 replays against a real remote Linux computer
(GUI + CLI, SSH env-configured). Every success is checked against ground truth on the machine,
not just the replay's own verdict. Latest real run (results in
scripts/teach-benchmark-results.json):

An honest small-scale sanity check — not a claim of state of the art.

Limitations

• Recording observes state transitions (polled, ~500 ms), not input events — very fast
  actions can coalesce. Slow, deliberate demonstration is the intended input.
• Clicks are inferred from state changes; ambiguous transitions record no click rather than a
  wrong one.
• Model-assisted recovery is an optional hook (ModelBackend) — adapters wire the agent.
• Replay speed is SSH-round-trip bound (~4 s/step on the evaluation box).
• DSH is in developer preview with compatibility-breaking changes — re-verify per release.

Attribution and license

Apache-2.0 (see LICENSE). TeachReplay was extracted from TeachReplay v0.1, developed
inside OpenMausBot (Apache-2.0) — see
NOTICE for the derived-work attributions, including the CDP helper derived from
OpenMausBot's computer-use tooling. Record-to-skill is not a new idea — TeachReplay's value is
the working, verified combination of demonstration → structured trajectory → parameterized
skill → executable replay → explicit verification across GUI + CLI.