Orchestral

A TypeScript orchestration layer for media generation — text-to-image,
image-to-video, text-to-speech, speech recognition — built for local-first, BYOK
apps: capability routing, opt-in semantic fallback, and an asset-handle
protocol.

You describe what a step needs (text-to-image, image-to-video,
automatic-speech-recognition, …), not which model to call. Orchestral routes
that capability to a model you supplied, retries inside the router, knows the
semantically equivalent paths a capability without a model could degrade
through (reporting them on failure by default; redirecting automatically is
opt-in), and passes generated media between steps as opaque handles the host
resolves. It ships no provider SDK and no API keys — calling a model is a
~15-line adapter you write, over whichever SDK you already use. Everything runs
in your process; there is no hosted control plane.

How it relates to the AI SDK / LangChain

Different layers, and Orchestral expects you to keep using the others:

• A provider SDK (the Vercel AI SDK, an official vendor SDK) owns one model
  call — auth, request shape, streaming, transport retries. Your Orchestral
  call adapter is usually a dozen lines over one of these; the examples here
  use the AI SDK's generateImage.
• An agent framework (LangChain / LangGraph, the AI SDK's own tool loop)
  owns the generic tool loop — planning, memory, a graph of steps. Orchestral's
  agent patterns delegate the loop to whichever one you inject.
• Orchestral owns what neither covers for media: routing a capability
  rather than a model id, declaring semantically equivalent fallback paths for
  when no model serves that capability, and threading generated assets between
  steps as handles instead of raw ids.

Quickstart

Not on npm yet — 0.1.0 publishes shortly. Until then, clone this repo and
run the examples.

npm install @orchestral/core @orchestral/runtime @orchestral/patterns zod

Two optional packages sit on top: @orchestral/discovery (the BM25 search
behind a find_pattern tool) and @orchestral/agent (the two agent patterns).
Neither pulls in a provider SDK.

zod v4 (>=4.3 <5) is a peer dependency: pattern input/output schemas are zod
schemas on the public API, so your app and Orchestral must share one zod
instance.

Two runnable hosts live in this repo:

• examples/atomic-hello-world — one atomic
  text-to-image dispatch, with the whole provider bridge in src/ai-sdk-wiring.ts.
• examples/agent-hello-world — an agent-kind
  dispatch: an LLM tool-loop that picks and runs patterns.

pnpm install
pnpm --filter atomic-hello-world start   # needs OPENAI_API_KEY
pnpm --filter atomic-hello-world test    # no key: same wiring, mock model

Minimal example

Registry, model bridge, router, runtime, one dispatch — the whole surface:

import {
  PatternRegistry,
  InMemoryJobStore,
  createDefaultCapabilityRouter,
  type DispatchContext,
  type DispatchResult,
  type ModelCapability,
} from '@orchestral/core'
import { InlineRuntime } from '@orchestral/runtime'
import { createTextToImagePattern } from '@orchestral/patterns'
import { generateImage } from 'ai'
import { openai } from '@ai-sdk/openai'

const registry = new PatternRegistry()
registry.add(createTextToImagePattern())

// The seam you write: your provider SDK behind a ModelCapability envelope.
const model: ModelCapability = {
  capabilities: ['text-to-image'],
  provider: 'openai',
  modelId: 'gpt-image-1',
  inputs: ['text'],
  outputs: ['image'],
  tags: [],
  source: 'user',
  async call<I, O>(input: I, ctx: DispatchContext): Promise<DispatchResult<O>> {
    const startedAt = Date.now()
    const { images } = await generateImage({
      model: openai.image('gpt-image-1'),
      prompt: (input as { prompt: string }).prompt,
      abortSignal: ctx.signal,
    })
    const assets = images.map((img, i) => ({
      assetId: `img-${i}`,
      modality: 'image' as const,
      url: `data:${img.mediaType ?? 'image/png'};base64,${img.base64}`,
    }))
    const output = {
      modality: 'image' as const,
      assets,
      cost: 0,
      latencyMs: Date.now() - startedAt,
      model: 'openai:gpt-image-1',
      provider: 'openai',
    }
    return { output: output as O }
  },
}

const runtime = new InlineRuntime({
  store: new InMemoryJobStore(),
  registry,
  router: createDefaultCapabilityRouter({
    getModels: (cap) => (cap === 'text-to-image' ? [model] : []),
  }),
})

const job = await runtime.submitJob({
  patternId: 'text-to-image',
  input: { prompt: 'a watercolour fox in a misty forest' },
})
console.log(job.status, job.output) // 'done'  { modality: 'image', assets: [...] }

The annotated version of the same wiring is in
packages/orchestral-core/README.md.

What's in the box

@orchestral/patterns ships 25 patterns: 10 atomic ones (one per capability
— text-to-image, image-to-video, text-to-speech,
automatic-speech-recognition, …) and 15 meta pipelines with their prompts
inlined (storyboarding, script planning, idea-to-video, best-of-N image
selection, …). The 2 agent patterns (an orchestrator and a long-form-video
director) live in the optional @orchestral/agent package.

The full table — kind, input slots, outputs, and the host operations each
pattern expects you to supply — is generated from the built package:
pattern catalog.

The three seams

A host adopts Orchestral by satisfying three injection points. Two are
implementations you swap; the third is the call adapter:

Agent patterns add a fourth seam, AgentRunImpl, which drives the inner LLM
tool-loop. It is @alpha, and it ships nothing for the same reason
ModelCapability.call does not: picking an agent framework is the host's call.
examples/agent-hello-world wires one over the AI SDK's tool loop in ~150
lines — copy it and swap in whatever loop you already run.

Packages

All packages are Apache-2.0. The @orchestral/* packages are released together
on one version line; @orchestral/dsh-plugin versions independently.

Honest limitations

This is 0.x. Each package README states its own edges rather than hiding them:

• Agent resume is lossy. The transcript stores a step projection, not raw
  provider messages: tool_use pairing and reasoning blocks are gone on resume —
  runtime § Resume fidelity.
• No durable queue. InlineRuntime runs a job in the caller's tick, and
  abandonOrphanedJobs() marks jobs a dead process left behind as stale;
  ctx.askUser parks in memory only —
  runtime § Runtime semantics worth knowing.
• Deliverable metas need a multimedia backend you supply. Six
  MetaCommonDeps operations (ffmpeg-shaped: concat, subtitles, background
  audio, …) are specified but not implemented here —
  patterns § Deliverable metas.
• One shipped fallback, and taking it is opt-in. image-to-image → caption
  → re-render is the only Alternative in the first-party catalog, and
  InlineRuntime defaults to failing with the applicable paths listed rather
  than redirecting through them (alternatives: 'auto' turns redirects on) —
  runtime § Alternative fallback is opt-in,
  core § Semantic fallback.

Versioning

0.x: minor versions may contain breaking changes, patch versions never do.
The @orchestral/* packages share one version line and are published together —
pin the exact set you tested against. @orchestral/dsh-plugin versions
independently against its dev-preview host. The 1.0 line will follow semver
strictly.

Repository layout

packages/orchestral-core/        @orchestral/core
packages/orchestral-discovery/   @orchestral/discovery
packages/orchestral-patterns/    @orchestral/patterns
packages/orchestral-runtime/     @orchestral/runtime
packages/orchestral-agent/       @orchestral/agent
packages/orchestral-dsh-plugin/  @orchestral/dsh-plugin (independent version line)
examples/                        runnable hosts, ~50 lines each
scripts/smoke-dist.mjs           executes the built dist bundles end to end

Development

pnpm install
pnpm build        # tsdown bundle + tsc declarations + api-extractor rollup
pnpm test         # vitest, all packages and examples
pnpm typecheck
pnpm api:check    # public .d.ts surface vs the committed etc/*.api.md report
pnpm smoke:dist   # build, then run the published dist bundles end to end
pnpm docs:catalog # regenerate the pattern catalog table from the built dist

pnpm api:check failing means the public API changed. Review the diff, run
pnpm api:update, and commit the updated report alongside the change.

License

Apache-2.0 — see LICENSE and NOTICE.

@orchestral/patterns contains prompt text derived from a third-party MIT
project; the affected constants and the upstream license text are listed in
packages/orchestral-patterns/CREDITS.md.

Contributing and security

See CONTRIBUTING.md. For vulnerabilities, please use the
private channel in SECURITY.md rather than a public issue.