It is 9:40 on the last business day of the month. Your primary cloud region has started returning intermittent errors: some API calls succeed, others time out, and the managed database is accepting reads but stalling on writes. The provider’s health dashboard calls it a “partial service disruption.” Your own status page, after a tense Slack thread, says “degraded.”
The web tier failed over within minutes. But 9,000 in-flight agent runs are frozen. Among them is your largest customer’s month-end close, a chain of reconciliation, approval, and reporting steps that their finance team expects to finish before the end of the day.
The outage lasts four hours. Your contract promises 99.9% availability. The question nobody can answer in the incident channel is simple: can those runs be safely resumed in the other region, or will doing so post duplicate journal entries?
Why stateful workflows don’t fail over like web servers
Stateless services fail over easily because any instance can handle any request.
Long-running workflows and agents break that assumption. A run that spans minutes or days has accumulated history: which steps completed, what the LLM returned, which tools were called, which human approvals are pending. That history lives in a specific database in a specific region, and the workers executing the run hold leases or locks on it. When the region degrades, the run is pinned there.
Three things make this worse for agentic products:
- Partial failures are the common case. Often one dependency slows down while everything else looks healthy. Health checks pass, workers stay registered, and runs hang.
- Failover creates two failure modes, not one. If you resume runs in a second region while the first is still partly alive, both regions may execute the same run: split brain. If you fail over before replication has caught up, the second region resumes from an older checkpoint and repeats or loses steps.
- The LLM provider has regions too. Your agent may depend on a model endpoint hosted in the same affected region, or on a provider having its own regional incident. Your workflow can be perfectly replicated and still unable to make progress.
And many disaster recovery plans were written for the stateless tier and never tested against a backlog of half-finished workflows.
Why the usual workarounds fall short
The first instinct is often “we’ll just restart failed runs from the beginning in the other region.” For a month-end close that has already posted entries to an ERP system, sent notifications, and collected two approvals, restarting from scratch means duplicate side effects and angry auditors.
The second instinct is active-active everything: run workers in both regions against a multi-master database. Without a clear rule about which region owns a given run, two workers can pick up the same run after a network partition and race each other. Conflict resolution in the database does not know that “post journal entry” must happen exactly once.
The third workaround is manual replay of stuck runs by an engineer. That works for dozens of runs, not for 9,000 during an incident.
Design principles for regional resilience
None of these principles requires a specific vendor.
Decide RPO and RTO per workflow class
Not every run deserves the same treatment. Group workflows into classes and set a recovery point objective (how much progress you can afford to lose) and a recovery time objective (how long a run can be paused) for each. A month-end close might need near-zero progress loss and resumption within an hour. A nightly enrichment job can tolerate restarting tomorrow. Synchronous cross-region replication is expensive; pay for it only where the contract demands it.
Replicate workflow state at the database level
Durable workflow engines typically persist each run as an event-sourced history: every completed step is recorded, and after a crash the engine replays history and skips steps that already finished. That property is what makes cross-region recovery possible, but only if the history exists in the second region. Use database-level replication for the state store backing your workflows, and know your replication mode. Asynchronous replication means some recent steps may be missing after failover, which is where idempotency (below) becomes mandatory.
Enforce single-writer ownership
At any moment, exactly one region should own a given run. Make ownership explicit: a region-level fencing token, a lease in a strongly consistent store, or a manual “promote secondary” switch that also demotes the primary. The rule is that the old region must be prevented from writing before the new region starts resuming.
Make every activity idempotent
Assume any step can run twice. Pass an idempotency key derived from the run ID and step name to every external call: payment APIs, ERP postings, ticket creation, emails. Check for an existing result before acting. With this in place, a resumed run that repeats a step because of replication lag produces the same outcome instead of a duplicate.
Route traffic between regions deliberately
Separate “where new runs start” from “where existing runs resume.” During an incident you may want to send all new runs to the healthy region immediately while existing runs wait for a controlled ownership transfer. Your routing layer should support both decisions independently.
Treat LLM endpoints as a failover dependency
Configure more than one model endpoint, across regions or providers, behind a circuit breaker. When error rates or latency cross a threshold, the breaker opens and calls go to the fallback. Test your prompts on the fallback model before you need it. Because a durable workflow records the LLM response as a completed step, a run that switches providers mid-flight does not need to repeat earlier calls.
Implementing it with a durable workflow runtime
If you are building on open source, Dapr is one practical foundation. Dapr Workflow stores an event-sourced history of each run in a pluggable state store such as PostgreSQL or Cosmos DB, replays after a crash, and does not re-execute completed activities. Because the state store is pluggable, much of your cross-region strategy follows the database’s replication strategy, though you should also confirm how timers and reminders are recovered in the secondary region. Orchestrator code must be deterministic, which pushes all I/O into activities, exactly where you want your idempotency keys.
Dapr also ships resiliency policies, defined in YAML, that add timeouts, retries, and circuit breakers to calls between services and components. Activity calls can carry their own retry policy:
import dapr.ext.workflow as wf
from datetime import timedelta
wfr = wf.WorkflowRuntime()
retry = wf.RetryPolicy(first_retry_interval=timedelta(seconds=2), max_number_of_attempts=5, backoff_coefficient=2, max_retry_interval=timedelta(minutes=1))
@wfr.workflow(name=”month_end_close”)
def month_end_close(ctx: wf.DaprWorkflowContext, data: dict):
for batch in data[“batches”]:
key = f”{ctx.instance_id}:{batch[‘id’]}”
yield ctx.call_activity(post_entries, input={“batch”: batch, “idempotency_key”: key}, retry_policy=retry)
return {“status”: “closed”}
@wfr.activity(name=”post_entries”)
def post_entries(ctx: wf.WorkflowActivityContext, data: dict):
# skip the write if the ERP has already accepted this idempotency key
The post_entries activity is where you would check an idempotency key against the ERP before writing.
Running this yourself across regions means owning the replication, the ownership handoff, and the routing. If you would rather not, managed options exist. As one example, Diagrid offers Catalyst, a platform built on Dapr that lists multi-region failover, described as traffic routing between regions and clouds, among its features. Catalyst can be Diagrid-hosted, run in your own cloud account, or self-hosted. As with any vendor claim, verify how failover behaves for your workloads, what recovery objectives it actually achieves in your setup, and run your own failure tests before writing those numbers into a customer contract.
A roadmap you can start this quarter
A reasonable order of work:
- Inventory and classify. List every workflow type and agent, its typical duration, its external side effects, and its contractual importance. Assign each a class with an RPO and RTO.
- Move long-running work onto durable execution. Any run that outlives a single request should have its progress persisted step by step, whether through a durable workflow platform or an engine you operate yourself.
- Add idempotency keys to every side-effecting activity. Start with the money-moving and customer-visible ones.
- Set up and monitor replication. Track replication lag as a first-class metric and alert on it.
- Define the ownership handoff. Write down, and automate, how the secondary region becomes the writer and how the primary is fenced.
- Add LLM fallbacks. Configure secondary model endpoints behind circuit breakers and evaluate output quality on them.
- Run game days. Simulate a partial regional outage with real (non-production) in-flight runs. Measure how many resumed cleanly, how many duplicated a side effect, and how long it took.
And a short checklist to review before each game day:
- ☐ Every workflow class has a documented RPO and RTO
- ☐ Workflow state is replicated to a second region, and lag is monitored
- ☐ Only one region can write a given run at a time
- ☐ All external side effects carry an idempotency key
- ☐ New-run routing and existing-run resumption can be switched independently
- ☐ At least one fallback LLM endpoint is configured and tested
- ☐ The failover runbook has been executed end to end in the last quarter
Conclusion
Back to that last business day of the month. With per-class objectives, replicated history, single-writer ownership, and idempotent activities, the incident looks different. Your team fences the degraded region, promotes the secondary, and lets durable runs resume from their last recorded step. The month-end close picks up after its second approval instead of starting over, and the activity sees the idempotency key already recorded in the ERP and skips the repeated posting. The conversation with your largest customer is about a short delay, not corrupted books.
Stateless failover is well understood. Stateful, agentic failover is a design discipline. Build it in deliberately, and test it before the region decides to test it for you.
