I’ve recently joined the Durable Task team under Azure Serverless, and I’m very excited. Durable tasks are such an interesting mix of complexity and elegance.
I’ve written about Durable Task and Durable Task Scheduler (DTS) before, including my DTS introduction. It still surprises me how many developers don’t really know them, let alone use them. They tackle failure and recovery problems every distributed system has to deal with. Yes, even yours with one microservice and a database.
Plenty of us can quote the fallacies of distributed computing. Quoting them, though, isn’t the same as handling the failures they describe in our code. Knowing that the network can fail doesn’t tell a workflow halfway through an order what to do when it does. Durable Task helps recover workflow progress. It doesn’t solve every problem in a distributed system.
I find the Durable Task Framework fascinating, but I think the apparent “magic” also puts people off. How can an orchestration make progress after its original execution in memory is gone? In this post, I want to go under the hood and show you the secret sauce.
And we can actually look. The Durable Task Framework/Core and modern .NET SDK are open source. We’ll follow their worker implementation, not the private internals of the managed DTS backend.
I’ll follow the standalone .NET gRPC worker in Durable Task SDK v1.26.0, which depends on DurableTask.Core 3.9.0. That’s the implementation path for this walkthrough, not a claim about every deployed package combination or Azure Functions configuration.
A small orchestration to follow
Here is the complete orchestration declaration we’ll follow, based on the public source. [DurableTask] marks the class for the optional generator described below. Activity implementations and registration are omitted; this is a teaching example, not an executed sample.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
using System.Threading.Tasks;
using Microsoft.DurableTask;
[DurableTask(nameof(OrderFlow))]
public sealed class OrderFlow : TaskOrchestrator<string, string>
{
public override async Task<string> RunAsync(TaskOrchestrationContext context, string input)
{
/* L1 */ string orderId = input;
/* L2 */ decimal subtotal = await context.CallActivityAsync<decimal>("ReadSubtotal", orderId);
/* L3 */ decimal total = subtotal + 10m;
/* L4 */ string receipt = await context.CallActivityAsync<string>("ChargeOrder", new { orderId, total });
/* L5 */ return receipt;
}
}
If you’d rather avoid strings, the optional Microsoft.DurableTask.Generators package generates strongly typed helpers for activities and orchestrations defined as classes, including calling sub orchestrations and starting orchestrations. The SDK’s typed example uses CallSayHelloTypedAsync and ScheduleNewHelloCitiesTypedInstanceAsync. That example uses Functions, but the generator also supports standalone workers.
The generated methods still delegate to CallActivityAsync, CallSubOrchestratorAsync, or ScheduleNewOrchestrationInstanceAsync. They add convenience and type safety at compile time, not a different replay or history model. The generator is versioned separately and marked as preview in the code we’re following; that is not a claim about tested package compatibility or the latest release. I’m keeping the strings here so we can see the scheduling identities.
For the walkthrough, the input is "order-42", ReadSubtotal produces 100m, and ChargeOrder produces "receipt-7" after receiving a total of 110m. Those values are illustrative, not measured results.
The extra 10m gives us a local calculation to track. When a later invocation reaches L3, does it recover a saved total variable, or calculate the value again?
It calculates it again. Let’s see why.
Who calls RunAsync, and what happens at await?
Before L1 can run, some application code has to request an orchestration instance. Here, client is an already configured DurableTaskClient, connected to the same DTS task hub as a running worker. That worker has OrderFlow, ReadSubtotal, and ChargeOrder registered.
1
2
string instanceId = await client.ScheduleNewOrchestrationInstanceAsync(
nameof(OrderFlow), input: "order-42");
The await in this starter waits for successful scheduling and returns the instance ID. It does not wait for the order to finish, and it does not call RunAsync on the starter’s call stack. The [DurableTask] attribute on our class is a generator marker, not something that starts the method.
DTS schedules the work. Your worker runs the C#. The worker receives orchestration work and the available history from DTS. It looks up the registered orchestrator to find the code for OrderFlow. The starter and worker can be hosted in the same process; these are different roles, not a requirement for separate machines.
Inside the worker, a library runner reads the available history and drives the orchestration forward. Its .NET name is TaskOrchestrationExecutor. This is library code inside the worker, not DTS itself or a class you have to write. When it processes the starting ExecutionStarted record in the history, the SDK adapter deserializes the input and calls RunAsync with the context and "order-42". Now L1 runs.
The runner takes the recorded history and runs or replays our code as far as that history allows, collecting instructions for what should happen next. It returns the remaining instructions to the worker to send to DTS.
Core calls these instructions actions (the orchestration’s decisions about what should happen next) and returns them in OrchestratorExecutionResult.Actions.
The runner has just read a record saying this instance started and used it to enter RunAsync. Records like this are history events. They are the incoming history the runner uses to reconstruct progress, not C# events that application code subscribes to.
When RunAsync requests ReadSubtotal for "order-42", the call adds a request to the orchestration context. This local action describes work to ask for next. Creating it locally does not durably record the request. When a new activity is scheduled, a history event records that fact; another records its successful result, 100.
RunAsync itself returns the receipt, not those actions. The runner executes or reconstructs the method against the incoming history and returns a result object containing the actions still outstanding. During replay, matching recorded schedules removes candidate actions, so they are not sent again.
At L2, CallActivityAsync creates a scheduling instruction and a local result task. The instruction, called an action, says: ask DTS to run ReadSubtotal with "order-42" as input. The ordinary .NET task is what L2 awaits to get the subtotal. Both exist in worker memory. Creating them does not mean the activity has been dispatched or that DTS has durably recorded the request.
If that result task is incomplete, ordinary C# await yields without blocking its thread. This pauses the method, not the runner’s history processing. An already completed task need not make the method pause.
The runner keeps reading any available history. A completion event later in this same pass can supply a result and let the method continue. A previously recorded schedule can be matched to the reconstructed instruction instead of sent again.
After processing the pass’s available history, the runner returns the remaining actions in OrchestratorExecutionResult.Actions, and the worker sends them to DTS. For new activity work, those actions request scheduling. Finishing this response is not finishing the order workflow. One pass can return several actions; an await is not itself a durable checkpoint.
Here is the actual return statement from Durable Task Core’s TaskOrchestrationExecutor.ExecuteCore, with the surrounding method omitted. The Actions property contains the remaining actions collected in the orchestration context:
1
2
3
4
5
return new OrchestratorExecutionResult
{
Actions = this.context.OrchestratorActions,
CustomStatus = this.taskOrchestration.GetStatus(),
};
Once a new activity is scheduled, a separate activity handler in a worker runs it and reports its outcome. A recorded outcome can then be supplied as history for another orchestration pass. In the reconstruction path below, RunAsync starts again and history resolves the new local task at L2. The activity is not calling back into the old suspended method.
Here’s the same flow as a conventional sequence diagram, read from top to bottom. The client gets an instance ID, the first worker pass asks for ReadSubtotal, and its recorded result lets a later pass recompute 110m and ask for ChargeOrder.
The columns are roles, not necessarily separate processes. Activation bars on the orchestration worker’s lifeline mark separate passes; a later pass may use the same worker process or a different one. Solid arrows show messages or calls, and dashed arrows show replies. This is a conceptual sequence for the reconstruction path, not a network capture or timing trace. The result reaches the later pass through history, not through a callback to the original waiting task.
Figure 1. A conceptual sequence from scheduling OrderFlow to the next action for ChargeOrder. The two activation bars on the orchestration worker mark separate history passes, not the lifetime of a process. Arrows show causal flow, not measured timing. The DTS lifeline shows its public history and scheduling contract, not an unpublished storage, transaction, or acknowledgement design. Activity results become history; they do not resume the original CLR task.
What survives is history, not the suspended method
DTS keeps each instance’s history as part of the task hub’s managed state, and a worker receives those records to reconstruct execution. We are describing that contract, not the service’s database or storage layout.
For ReadSubtotal, two records in our example say:
- Activity 0, named
ReadSubtotal, was scheduled with input"order-42". - Activity 0 completed successfully with result 100.
These are conceptual descriptions, not an exact data format or the whole history. Starting and housekeeping records are omitted. The 0 identifies the activity, not the record’s position in history.
What about loops? Calling the same activity in a
fororforeachis fine if, for the same inputs and recorded history, the loop reproduces the same ordered durable calls. Each call gets its own sequence ID from the runtime counter, and replay recreates those IDs at the same calls. Core matches the ID, action type, and activity name, not the name alone. These IDs are not loop indices or global counters for an activity: timers, sub orchestrations, and event sends share the sequence. Changing iteration order or which durable calls occur can break matching; the same number of calls is not enough.
The first record tells the runner that this activity was already scheduled. During replay, L2 creates a candidate action again. The runner matches the record against that action and removes the candidate from its outgoing actions, rather than scheduling ReadSubtotal anew. This record does not say the activity finished, and matching it does not complete the await. Core represents this scheduling record with the .NET class TaskScheduledEvent.
The second record supplies the successful outcome. Processing it provides the result to the reconstructed local task, so RunAsync can continue. Core represents this completion record with TaskCompletedEvent. Its TaskScheduledId is 0 here, connecting the result 100 to the activity that was scheduled.
For this call, the action requests ReadSubtotal, the history events record its scheduling and result, and the Task is the local object L2 awaits. Those event classes belong to the runtime’s bookkeeping. Application code calls the orchestration and activity APIs; it normally does not construct history objects or raise C# events. DTS preserves history data, while the worker interprets it using local .NET objects.
On reconstruction, L2 assigns the recorded result to a fresh subtotal variable as 100m. L3 calculates 110m again. The old variable slots and Task objects are not restored. That does not mean 110m can never appear in history: L4 sends it as part of ChargeOrder’s input. It can be recorded there as serialized input data, not as a snapshot of every local variable or the suspended CLR stack.
No orchestration thread is parked waiting for the activity. The worker process doesn’t have to stop at every await either. Ending a pass through history isn’t the same as ending a process.
Does Worker 1 keep all those waiting tasks in memory?
Now, you’ve all heard something I didn’t say. Metaphorically, since you’re reading this. You’re probably thinking:
“Worker 1 ran the first pass. If Worker 2 resumes the orchestration after the activity finishes, what happens to Worker 1 and the
TaskCompletionSourceit created?”“If Worker 1 handles 10,000 orchestration instances, each waiting for an activity that takes a week, does it keep 10,000 pending tasks and state machines in memory for that whole week?”
No. A waiting workflow does not require its original invocation or local tasks to stay alive. An ordinary async await does not park a thread with an OS stack either. Durable execution lets the workflow make progress without its original continuation in memory.
Worker 1 requests ReadSubtotal by returning a scheduling action. Once new actions and history have been committed, the runtime can unload that execution. When the activity reports 100 and the result is durably recorded, DTS keeps it in the instance history. The next pass needs that history, not Worker 1’s memory.
An eligible Worker 2 connected to the same task hub, with compatible code registered, can receive that work. It reruns RunAsync and creates a new local Task and TaskCompletionSource. The TaskScheduled record matches the reconstructed request; the TaskCompleted record, through TaskScheduledId, supplies 100 to Worker 2’s new task. Nothing needs to find Worker 1’s original helper or call its old continuation. The durable activity ID is not a .NET Task.Id or a memory address. Switching workers alone does not rerun an activity whose successful result is already recorded.
The standalone gRPC pass can finish while the workflow waits. It does not require those 10,000 original orchestration task objects to be retained for a week. Once unreferenced, they can be collected. That does not promise immediate garbage collection, zero overhead, free waiting, or scaling to zero. Some hosts cache execution state, but recovery does not depend on the original worker or cache surviving.
There is one important distinction: a waiting orchestration is not the same as an activity that keeps running for a week. An activity executing, awaiting I/O, or sleeping for that long can retain activity worker resources. Its execution is not checkpointed in the middle, and failure can require another attempt with the usual side effect concerns. If the work is only waiting for time to pass or an outside answer, use a durable timer or wait for an external event instead.
Replay, line by line
Let’s walk the five lines through three logical activations. Here, an activation means one pass in which the worker processes orchestration history and produces actions.
This simplified trace follows the public source. It is not three measured deliveries or a promise about physical service batches, and housekeeping events are omitted. With no other durable operations in this example, the inspected Core counter assigns activity IDs 0 and 1. Those IDs correlate actions with scheduled tasks; they are not positions in the event history.
Read each strip from top to bottom: code position on the left, history cursor on the right, and pending actions, open result tasks, and locals below. A TaskCompletionSource is the helper supplying the awaited task’s result. These are settled teaching checkpoints based on the public source, not a live debugger recording. Click any strip to open the full SVG and enlarge it.
The runner reads Earlier history first to reconstruct this execution. New events for this step are the events newly delivered for the current step; “new” does not mean their data is unrecorded. The read counts track events, not unfinished activities, and records already read stay stored.
A: ask for the subtotal
The starting event supplies the input. L1 reads "order-42". L2 calls ReadSubtotal, creating action 0 and an open local task. There is no result yet, so the method yields at L2. L3 through L5 have not executed.
The executor returns the new action to schedule ReadSubtotal with ID 0. Once that action is accepted, scheduling history records the operation. The activity runs separately and reports its outcome.
Notice what is missing: no durable record of a local variable called subtotal with an instruction pointer beside it.
Figure 2A. Activation A. With no earlier history to read, ExecutionStarted reaches L2 and creates candidate 0 plus a pending local result task. The runner returns Schedule 0 after processing available history. The task is not persisted.
B: use the subtotal, then ask for the charge
Now the available history includes the scheduled operation and a completion representing 100m.
The orchestration starts at L1 again. At L2, its code creates a candidate action 0 and another local task. Replay is not simply “look in a dictionary and instantly return a completed task.”
Core’s HandleTaskScheduledEvent matches the recorded scheduling event against the reconstructed action using the sequence ID, action kind, and activity name. It then removes that candidate from the outbound action map.
Matching the schedule removes the recreated action from the outgoing list; it does not, on its own, supply the result.
The completion handler uses the recorded activity ID to find its open local task, supplies the serialized result, and then removes the entry. This excerpt from Durable Task Core’s TaskOrchestrationContext.HandleTaskCompletedEvent keeps the existence check; the method wrapper and duplicate event branch are omitted:
1
2
3
4
5
6
7
8
int taskId = completedEvent.TaskScheduledId;
if (this.openTasks.ContainsKey(taskId))
{
OpenTaskInfo info = this.openTasks[taskId];
info.Result.SetResult(completedEvent.Result);
this.openTasks.Remove(taskId);
}
The order matters: SetResult runs before removal, and the continuation can run inside that call. The snapshots show the settled state afterward.
When the executor processes the completion for ID 0, HandleTaskCompletedEvent sets the reconstructed completion source’s result. The async chain can then continue within this same activation, deserialize the value, and assign subtotal = 100m. Core’s orchestration synchronization context and synchronous task scheduler keep those continuations inside this pass through history.
Why the context matters. When an ordinary
awaitneeds to suspend, a capturedSynchronizationContextcontrols how its continuation, the code after thatawait, is scheduled. The runtime installs its own context and uses its scheduler to keep that code in the runner’s controlled history processing.ConfigureAwait(false)opts out of that capture and can escape that path, so don’t add it inside orchestrator code. It does not necessarily switch threads on every await.Keeping this context does not promise the same OS thread or process across activations. Each reconstruction has its own objects and context. Neither normal capture nor
ConfigureAwait(true)makesTask.Delayor HTTP I/O durable. Activities follow ordinary .NET async guidance separately.
L3 calculates 100m + 10m, producing 110m. L4 calls ChargeOrder, creating the new action 1 with { orderId, total }. Its result is not available yet, so the method yields again.
The outstanding action is now Schedule 1: ChargeOrder. There is no new Schedule 0 merely because L2 executed again.
Figure 2B. Activation B. Matching the earlier Schedule 0 removes the candidate, not the task. After all earlier records are read, IsReplaying is false before completion 0 advances L2, L3 and L4 and computes 110. The runner returns only the new Schedule 1.
C: use both recorded operations, then return
With the second activity’s result available, the method again starts at L1.
L2 reconstructs action 0; history matches its schedule and supplies 100m. L3 calculates 110m again. That is a fresh calculation, not a saved local restored from a checkpoint.
L4 reconstructs action 1. Its recorded scheduling event removes the candidate from the outbound map, and its completion supplies "receipt-7". L5 returns that receipt. The executor can now emit orchestration completion, rather than a new activity schedule.
In this pass, result 0 comes from earlier history, while result 1 can be newly delivered for this step. Neither activity is newly scheduled just because both call sites ran again.
Figure 2C. Activation C. Both schedules are matched, and recorded result 100 drives a fresh calculation of 110. All earlier records have been read while L4 still waits. Completion 1, delivered for this step, supplies receipt-7, and the runner emits orchestration completion without scheduling either activity again.
Zoom in on C. Read the Earlier history cards from oldest to newest. Earlier events left counts the earlier records still to read, moving from 4 to 0 as the cursor advances. It is not a count of unfinished activities, and reading a card does not delete its stored record. Matching a schedule removes a candidate action, while processing its completion resolves the open task. One completion can advance several source lines, and an await can span several events.
Figure 3. “Earlier events left” counts earlier history records still to read, not pending tasks. Positions 1 through 5 are walkthrough order, not EventId or activity IDs. The result delivered for this step waits until all earlier records are read. Housekeeping events are omitted.
Only after the last record in the earlier history (TaskScheduled for activity 1) has been processed does the executor set IsReplaying to false, before processing new events. L4 is still waiting on task 1. The newly delivered completion then resolves it, and L5 returns the receipt.
Reading all the earlier history does not mean every task has finished or every newly delivered event has been processed. It isn’t a general signal of workflow completion, and another activation can replay again.
Now suppose history contains TaskScheduled(0) but no completion or failure for it. Replay still matches and removes the candidate scheduling action, but the reconstructed task stays incomplete. Pending does not mean “schedule it again because we replayed.” Redelivery of an outstanding activity is a separate delivery concern.
No crash is needed for any of this. In the reconstruction path, normal activity results drive the same processing of history. Replay is part of normal progress, not just recovery.
Can the runtime keep the local execution around?
The available caching path depends on the host and its configuration.
The standalone gRPC path used above constructs a new TaskOrchestrationExecutor. A separate runner used primarily by Azure Functions isolated hosting has an extended sessions path that can reuse an executor and process new events. Functions also documents instance caching that varies by provider.
That tells us what those paths can do, not which options a particular deployment enables. The diagrams show reconstruction, not a promise that every host unloads after every await or replays everything for each notification.
The code restrictions stop looking arbitrary
These mechanics explain why orchestrator code has to be deterministic.
For a given recorded history, the orchestration must reconstruct compatible durable operations in the required order. If a read of the wall clock, an unseeded random value, or an external response changes which operation comes next, the old history may no longer fit the new execution. Core performs useful matching checks, but that doesn’t guarantee it will detect every possible nondeterministic change for you.
For time, use context.CurrentUtcDateTime rather than reading the wall clock directly. For a GUID that is safe to replay, use context.NewGuid(). Replay safety is the point: NewGuid() is not a source of fresh, unpredictable randomness on every replay. If you need an external or independently random value, obtain it through an activity so its result can participate in the recorded history.
The same distinction applies to I/O. Awaiting HttpClient directly inside an orchestrator does not turn the HTTP request into a durable activity or record its response for replay. Put that I/O inside an activity. An HttpClient used by the activity implementation is a different case from an orchestrator awaiting it directly.
For a durable wait, the modern API here is context.CreateTimer(...), with TimeSpan or DateTime overloads and a cancellation token. Ordinary Task.Delay is normally nonblocking, but it is not a durable timer. The orchestrator code constraints follow from this boundary: ordinary async work does not become durable simply because it appears inside an orchestration method.
And then there are logs. L3 can execute more than once, so an ordinary log beside it can appear more than once. context.CreateReplaySafeLogger("OrderFlow") returns a logger that suppresses writes while IsReplaying is true. That cuts replay noise. It doesn’t guarantee that each audit event is recorded exactly once.
Do not “fix” repeated execution by placing durable actions inside if (!context.IsReplaying). Replay needs to reconstruct those actions to match the history. Suppressing the log is useful. Suppressing the operation changes the program.
Replay is not retry
There are three mechanisms worth keeping separate:
- Replay of a recorded outcome: orchestration code executes again and history resolves its reconstructed task. That does not itself request another execution of a successfully recorded activity.
- A configured retry after failure: a recorded failure faults the reconstructed task. An explicitly configured retry policy or handler can deliberately arrange another attempt; the SDK’s retry branches are explicit.
- Redelivery of unacknowledged activity work: the activity execution contract is at least once. An activity may execute again when its completion was not durably recorded, even if some external work already happened.
The boundary to watch is inside ChargeOrder.
Suppose the payment system accepts the charge for 110m. Before the activity’s completion is durably recorded, its worker fails. The payment system has done something real. The orchestration history does not yet contain a successful result it can replay.
When the activity runs again, it can reach that payment system again. History can recover orchestration progress; it cannot close the gap between the external effect succeeding and the activity outcome being durably recorded. Using a durable framework doesn’t put the payment system and the scheduler in one atomic transaction.
The charge operation still needs idempotency or deduplication at the business level. For example, use a stable key for this particular operation, honored by an integration that supports it. The key must distinguish a duplicate attempt from another legitimate charge.
Now move the failure to after the successful completion has been durably recorded. If worker memory is lost then, replay can supply "receipt-7" and advance to L5 without scheduling that completed logical activity again. That recovers a recorded outcome. It doesn’t promise every external effect happened exactly once.
Back to the debugger
The await is still ordinary C#. The framework call creates an action to perform a durable operation and a local task. History can later match that operation and complete a reconstructed task, letting compatible code calculate its locals and continue.
So when you see L2 or L3 execute again, ask two separate questions: which source lines are being replayed, and which actions are actually being emitted?
That is the secret sauce: the method can keep making progress without keeping its original stack alive. Its activities still have to handle external effects correctly.
Once you can follow the history and the actions, the apparent magic becomes something you can reason about in your own code. In upcoming posts, I’ll dig into more of the overlooked capabilities and hidden gems in the Durable Task ecosystem.