01 · Foundation
Durable Execution
Durable execution makes a function's progress as durable as a database row. The workflow is ordinary code, but every completed step is journaled — so the run survives process crashes, deploys, machine loss, and week-long waits, then resumes exactly where it stopped.
It's the answer to a problem every backend eventually hits: any flow that spans more than one request — checkout, onboarding, data pipelines, AI agent loops — is a distributed state machine, whether you designed one or not. Without durability you end up hand-rolling it with queues, cron sweeps, and status columns.
Why it's everywhere in 2026
Long-running AI agents forced the issue: a 40-step agent run that dies at step 39 is unacceptable, and retry-from-zero doubles your LLM bill. Durable engines (Temporal, Inngest, Restate, Cloudflare and Vercel Workflows) became the default substrate under serious agent products.The Failure That Motivates Everything
// A "workflow" as a plain async function. Looks fine. Isn't.
async function onboardUser(userId: string) {
await sendWelcomeEmail(userId); // step 1 succeeds
await provisionWorkspace(userId); // step 2 succeeds
// ── process crashes / deploy restarts HERE ──
await scheduleTrialCheckIn(userId); // step 3 never runs
await chargeCard(userId); // ...or worse, runs TWICE on retry
}
// If this function takes 14 days (trial period), you also need the
// process to live for 14 days. It won't.
The Same Flow, Durable
// The same flow as a durable workflow: each step's result is
// journaled. Crash at any point; the engine resumes exactly where
// it left off — even on a different machine, weeks later.
export async function onboardUser(ctx: WorkflowContext, userId: string) {
await ctx.step('welcome-email', () => sendWelcomeEmail(userId));
await ctx.step('provision', () => provisionWorkspace(userId));
await ctx.sleep('trial-wait', days(14)); // costs nothing while waiting
const usage = await ctx.step('check-usage', () => getUsage(userId));
if (usage.active) {
await ctx.step('charge', () => chargeCard(userId)); // runs exactly once
}
}
Mental model
Think workflow = orchestration (pure, replayable decision logic) and steps = side effects (retryable units whose results get recorded). Everything in this guide falls out of that split.01 · Foundation
Event Sourcing & Replay
The core trick: the engine never checkpoints your call stack — that would be fragile. Instead it records an append-only journal of step results, and reconstructs position by re-running your code against that journal.
Execute
workflow code runs
Journal
each step result appended
Crash
worker dies mid-run
Replay
code re-runs, steps served from journal
Resume
live from first missing entry
A Journal, Concretely
// The engine's journal for one run (simplified)
[
{ seq: 1, type: 'STEP_COMPLETED', id: 'welcome-email', result: 'ok' },
{ seq: 2, type: 'STEP_COMPLETED', id: 'provision', result: 'ws_8231' },
{ seq: 3, type: 'TIMER_STARTED', id: 'trial-wait', fires: '2026-08-27' },
// ── worker crashed here; new worker picks up the run ──
{ seq: 4, type: 'TIMER_FIRED', id: 'trial-wait' },
{ seq: 5, type: 'STEP_COMPLETED', id: 'check-usage', result: { active: true } }
]
// Replay: the new worker re-executes the workflow function from the
// top, but steps 1-2 return their JOURNALED results instantly instead
// of re-running. Execution "fast-forwards" to the first un-journaled
// step and continues live from there.
The consequence
Replay only works if the workflow code takes the same path given the same journal. That single requirement produces every determinism rule on the next page.01 · Foundation
Determinism Rules
Workflow code is re-executed on every replay, possibly months later on a different machine and a newer deploy. Anything that could return a different answer the second time is banned from workflow scope — it belongs inside a step, where the first result is recorded and reused.
Forbidden in workflow scope
Date.now() / new Date()
Math.random() / uuid()
fetch / DB queries / file IO
Reading env vars or feature flags
Iterating unordered maps/sets
Spawning threads / unmanaged timers
The durable equivalent
ctx.now() or a journaled step
ctx.step wrapping the RNG once
All IO inside steps/activities
Config passed as workflow input
Sort before iterating
ctx.sleep / engine-managed timers
// ❌ FORBIDDEN inside workflow code — replay would diverge
const now = Date.now(); // different on every replay
const id = Math.random(); // different on every replay
const res = await fetch(url); // network state changed
if (process.env.FLAG === 'on') { } // env changed between deploys
// ✓ The durable way — nondeterminism goes THROUGH the engine
const now = await ctx.step('now', () => Date.now()); // journaled once
const id = await ctx.step('uuid', () => crypto.randomUUID());
const res = await ctx.step('fetch', () => fetchJson(url));
// Env/config: read it in a step, or pass it in as workflow input
Silent until it isn't
Nondeterminism bugs don't fail on the happy path — they fail on replay after a crash, which is exactly when you can least afford it. Most engines ship a replay-safety linter or runtime sandbox; turn it on in CI.02 · Building Blocks
Steps & Activities
A step (Temporal calls it an activity) is the unit of side effect + record: run this closure, retry it on failure, journal its return value, never run it again once recorded. Steps are where the real world happens — API calls, DB writes, emails, LLM calls.
export async function syncInvoice(ctx: WorkflowContext, id: string) {
// Each step gets its own retry policy — transient failures are
// retried in place; the journal only records the final success.
const invoice = await ctx.step('fetch-invoice',
() => billing.get(id),
{ retries: 5, backoff: 'exponential', maxDelay: '2m' },
);
// A step that exhausts retries fails the run (or hits a
// compensation path you define) — it never half-completes.
await ctx.step('post-to-ledger', () => ledger.post(invoice), {
retries: 10,
nonRetryable: ['ValidationError'], // fail fast on permanent errors
});
}
Sizing Steps
One effect per step
Two API calls in one step means a retry can repeat the first call. Split them.
Small payloads
Step results live in the journal forever. Return IDs and summaries, not 10MB blobs.
Stable names
Step IDs key the journal. Renaming one mid-flight is a breaking change — see versioning.
02 · Building Blocks
Retries & Idempotency
Durable engines give you at-least-once execution of each step with exactly-once recording of its result. The gap between those two is yours to close: a step may succeed and crash before the journal write, so the side effect can happen twice even though it's recorded once.
// At-least-once execution: a step can run, succeed, and crash
// BEFORE its result is journaled. On resume it runs again.
// Side effects must therefore be idempotent.
// ❌ Not idempotent — resume can double-charge
await ctx.step('charge', () => stripe.charges.create({ amount }));
// ✓ Idempotency key derived from the run — retries are no-ops
await ctx.step('charge', () =>
stripe.charges.create(
{ amount },
{ idempotencyKey: ctx.runId + ':charge' },
),
);
Rule of thumb
Derive idempotency keys from runId + stepId — stable across retries and replays, unique across runs. Most payment, email, and provisioning APIs accept one; for your own DB, use an upsert keyed the same way.Sagas
When a later step fails permanently, earlier side effects may need undoing (refund the charge, release the inventory). The saga pattern is just durable execution running your compensation steps in reverse — same journal, same retries.02 · Building Blocks
Signals & Humans
A signal is an external event delivered into a running workflow: a webhook fired, a manager clicked approve, a user replied. The workflow awaits it like a promise — except the wait consumes no process and can last a quarter.
export async function approveExpense(ctx: WorkflowContext, req: Expense) {
await ctx.step('notify-manager', () => slack.dm(req.managerId, req));
// Park here — for minutes or months — until a human decides.
// The waiting run holds no process, no connection, no memory.
const decision = await ctx.waitForSignal<Decision>('manager-decision', {
timeout: days(7),
onTimeout: 'escalate',
});
if (decision === 'escalate') {
await ctx.step('escalate', () => slack.dm(req.vpId, req));
return ctx.waitForSignal<Decision>('vp-decision');
}
return decision;
}
// Elsewhere: an API route delivers the signal into the run
await client.signal(runId, 'manager-decision', 'approved');
Human-in-the-loop
This is the pattern that makes approval flows, agent guardrails, and "reply YES to confirm" loops trivial. The state machine you'd otherwise persist by hand (status columns + polling) is just a suspended await.02 · Building Blocks
Timers & Sleep
sleep(days(30)) in workflow code doesn't hold a process open for a month. The run is evicted entirely; a durable timer wakes it when due. Waiting is free — which changes how you design flows: drip campaigns, trial expirations, SLA escalations become straight-line code.
// Durable sleep: the run is evicted from memory entirely.
// A timer row in the engine's store wakes it later.
await ctx.sleep('cooldown', hours(6));
await ctx.sleep('trial', days(30)); // 30 days costs $0 of compute
// Cron-style recurrence is just a workflow that loops
export async function weeklyDigest(ctx: WorkflowContext) {
while (true) {
await ctx.sleepUntil('next-monday-9am');
await ctx.step('send', () => sendDigest());
}
}
Cron, but debuggable
A while-loop workflow with sleepUntil replaces cron jobs — with retries, history, and per-run observability that crontab never had. Use continue-as-new to reset the journal on each iteration so history stays bounded.03 · In Practice
Durable Agents
An agent loop is the worst-case long-running flow: dozens of expensive model calls, unreliable tools, occasional human approval, minutes-to-days of wall time. Put each model call and each tool call in its own step and the loop becomes crash-safe — a restart replays the journaled turns instead of re-buying them.
// A crash-safe agent loop: every model call and tool call is a
// journaled step. Kill the process mid-task; it resumes mid-thought.
export async function agentRun(ctx: WorkflowContext, task: string) {
const history: Msg[] = [{ role: 'user', content: task }];
for (let turn = 0; turn < 50; turn++) {
const reply = await ctx.step('llm-' + turn, () =>
llm.chat({ messages: history, tools: TOOLS }),
);
history.push(reply);
if (reply.stop === 'end_turn') return reply.content;
for (const call of reply.toolCalls) {
const result = await ctx.step('tool-' + turn + '-' + call.id, () =>
runTool(call), // retried independently on failure
);
history.push({ role: 'tool', id: call.id, content: result });
}
// Human-in-the-loop: park until approval for risky actions
if (reply.needsApproval) {
await ctx.waitForSignal('approved');
}
}
}
Token bills survive crashes
Replay serves finished LLM calls from the journal — a crash at turn 39 of 40 re-buys zero tokens.
Guardrails as signals
Risky tool call? Park on waitForSignal until a human approves. The agent holds no resources while parked.
Tool retries isolated
A flaky tool retries by itself with backoff — the model isn't re-asked, the loop doesn't restart.
03 · In Practice
Platform Landscape
The 2026 field agrees on the model (journal + replay + steps + signals + timers) and differs on packaging: self-hosted engine vs serverless, code-first vs event-first.
Temporal
The reference implementation. Self-host or cloud; polyglot SDKs; heaviest concepts, most control. The enterprise default.
Inngest
Event-driven functions with steps, serverless-first, great DX. Workflows trigger off typed events; strong queue/flow control.
Restate
Durable execution as a log-backed runtime; virtual objects give you durable state + RPC in one model. Low-latency focus.
Cloudflare Workflows
Steps + sleeps native to Workers; pairs with Durable Objects for stateful coordination at the edge.
Vercel Workflow
'use workflow' / 'use step' directives compile ordinary functions into durable ones on Fluid Compute.
DBOS
Durability implemented as Postgres transactions around your functions — the journal is your database.
Dialect Comparison
// Same idea, different dialects (2026):
// Temporal — workflow + activities, full replay engine
export async function order(itemId: string) {
const paid = await activities.charge(itemId); // journaled
await sleep('3 days');
await activities.ship(itemId, paid.receiptId);
}
// Inngest — step.run inside event-driven functions
inngest.createFunction({ id: 'order' }, { event: 'cart/checkout' },
async ({ event, step }) => {
const paid = await step.run('charge', () => charge(event));
await step.sleep('wait', '3d');
await step.run('ship', () => ship(paid));
});
// Vercel Workflow — 'use workflow' directive, steps via 'use step'
export async function order(itemId: string) {
'use workflow';
const paid = await charge(itemId); // a 'use step' function
await sleep('3d');
await ship(itemId, paid.receiptId);
}
Choosing
Already on Vercel/Cloudflare? Use the platform-native one. Complex multi-service orchestration with ops teams? Temporal. Event-heavy product backend? Inngest. The concepts transfer — pick for operational fit, not lock-in fear.03 · In Practice
Failure Modes
Durable execution removes a class of failures and introduces its own. These three cause most production incidents:
1 · Non-determinism Drift
A deploy changes workflow code while runs are in flight; replay takes a different path than the journal recorded. Every engine ships a versioning/patch primitive for exactly this:
// The poison pill: you deploy v2 of a workflow while 10,000 v1
// runs are mid-flight. Replay of old journals against new code
// diverges → runs fail.
// Every engine has a versioning gate for this:
if (ctx.patched('use-new-tax-step')) {
await ctx.step('tax-v2', () => calcTaxV2(order)); // new runs
} else {
await ctx.step('tax', () => calcTax(order)); // old replays
}
// Rule: never delete or reorder journaled steps for in-flight runs.
// Add behind a patch gate; retire the gate when old runs drain.
2 · Unbounded History
Symptom
A loop journals thousands of steps; replays get slower, payloads bloat, and the run eventually hits engine history limits. Fix: continue-as-new — end the run and start a fresh one carrying forward only compact state.3 · Poison Pills & Hot Retries
Symptom
A permanently-failing step retries forever, burning quota and masking the real bug. Fix: classify errors — nonRetryable for validation/permission failures, capped attempts + dead-letter path for the rest, alerting on runs stuck past their SLA.drifthistory bloathot retriescontinue-as-new
04 · Execution
Durable Checklist
Click items to mark complete. Progress tracked per session.
COMPLETION
0 / 11 complete
Workflow Hygiene
All nondeterminism (time, random, IO, env) goes through stepsCritical
Every external side effect has an idempotency keyCritical
Steps are small: one retryable side effect per stepDesign
Workflow inputs/outputs are small, serializable payloadsDesign
Reliability
Retry policies set per step, with nonRetryable error classesRetries
Timeouts on every step and every signal waitRetries
Compensation path defined for partially-completed flowsSaga
Code changes to live workflows gated behind version patchesCritical
Operations
Dashboard access to inspect / replay / cancel stuck runsOps
Alerting on run failure rate and journal growthOps
Long loops broken up with continue-as-new (bounded history)Design