🧯 Partial Failure Is a Feature: Best-Effort Fan-Out in Azure Durable Functions

What a fan-in really promises, why "all or nothing" is usually the wrong default for batch work, and how to design pipelines that fail gracefully without failing dangerously.

Batch jobs rarely fail because of something dramatic.

They fail because of one item.

One malformed file. One unexpected null. One API call that timed out at the wrong moment.

Here is a perfectly reasonable piece of orchestration code:

tasks = [context.call_activity("process", item) for item in items]
results = yield context.task_all(tasks)

Schedule everything in parallel, wait for all of it, carry on.

It reads like "process every item". What it actually means is closer to "process every item, unless one fails, in which case tell me nothing about any of them".

That gap between what the code says and what it promises is what this post is about.

We will use Azure Durable Functions as the concrete platform, and a hypothetical nightly job that imports a folder of data files as a running example. But the principles apply just as well to Promise.all, asyncio.gather, a Celery chord, or a plain for loop.

1. Fan-out/fan-in in one minute

Fan-out/fan-in is the pattern of splitting work into many independent tasks, running them in parallel (fan-out), then waiting for all of them and combining the results (fan-in).

Durable Functions gives you three building blocks for it:

• Activities do the real work: read a file, call an API, write to a database.
• An orchestrator coordinates them: which activities run, in what order, with what inputs.
• The framework records every step in a history. If the host restarts, the orchestrator is replayed from that history, so a long-running job picks up where it left off.

Two consequences of that design matter for everything that follows:

1. The orchestrator schedules work when it yields. Building a list of tasks and yielding them together runs them in parallel. Yielding them one by one runs them one by one.
2. Activities run at least once. A crash after the work but before the result is recorded means the activity can run again.

Keep both in mind. They come back.

2. What a fan-in actually promises

Every language has a fan-in primitive, and they don't all behave the same way:

The default in almost every case is fail-fast: one failure turns the whole fan-in into an exception, and the successful results are no longer handed back to you.

Fail-fast is a sensible default. If you're fetching a user, their permissions, and their settings to render one page, and one of those fails, there is no page. Stopping early saves work.

But a batch job isn't one page. Imagine a restaurant kitchen where one burnt dish sends every table home, including the tables whose food was already plated. That's fail-fast applied to independent work.

In our import, one bad file means:

• the other files in that step are effectively lost,
• every later step never runs,
• and the operator gets a single error instead of a list of what went wrong.

Fail-fast is the right default for work that is genuinely all-or-nothing, and the wrong default for work that is merely parallel.

3. Two questions before you choose

Before changing any code, decide which kind of work you have. Two questions are enough:

1. Are the items independent? Does item 7 failing make item 8's result wrong, or just incomplete?
2. Is a partial result useful? Would 98 imported files out of 100 be better than zero?

Most batch work (imports, notifications, report generation, sync jobs) lands in the third row.

The rest of this post is about doing that row well.

4. Treat failures as data

The core move is simple to state: a failed item should produce a result, not an exception.

Give every unit of work the same result shape, success or failure:

@dataclass
class ItemResult:
    item: str
    ok: bool
    detail: dict = field(default_factory=dict)  # counts on success
    error: str | None = None                    # reason on failure

Now the fan-in can never fail because of an individual item, because an individual item never raises:

@app.activity_trigger(input_name="item")
def process(item: str) -> dict:
    try:
        count = import_file(item)
        return asdict(ItemResult(item, ok=True, detail={"rows": count}))
    except InvalidInput as exc:
        return asdict(ItemResult(item, ok=False, error=str(exc)))

Once failures are values, you can count them, group them, show them to an operator, and make decisions with them. An exception tells you that something broke. A result tells you what.

Why not just catch it in the orchestrator?

The tempting alternative is to leave the activities alone and wrap each call in the orchestrator:

results = []
for item in items:
    try:
        results.append((yield context.call_activity("process", item)))
    except Exception as exc:
        results.append({"item": item, "ok": False, "error": str(exc)})

It isolates failures. It also runs everything sequentially, because of the first consequence from section 1: the orchestrator only schedules work when it yields, and this loop yields one task at a time.

You can keep parallelism and catch in the orchestrator, by scheduling everything and then looping on task_any to inspect each task as it finishes. It works, but it's more orchestration code, it has to stay replay-safe, and it's easy to get subtly wrong.

Push error handling down into the unit of work, and keep the orchestrator boring.

5. Not all failures are the same

"Return failures as data" doesn't mean "swallow everything". Different failures deserve different treatment, and the first step is telling them apart:

Retrying a malformed file three times just produces the same error three times, slower. Not retrying a 429 turns a one-second hiccup into a missing result.

For transient errors, you have two options:

• Retry inside the activity, then return the final failure as data. The fan-in stays intact, and you control exactly what counts as transient.
• Use the platform's retry, call_activity_with_retry with RetryOptions. It's cleaner and keeps retry timing durable, but once retries are exhausted it raises into the orchestrator, and you're back to fail-fast unless you catch it there.

Either is fine. What matters is that you've chosen deliberately.

And platform failures still happen outside your activity's try. So keep one coarse safety net around the fan-in:

try:
    results = yield context.task_all(tasks)
except Exception as exc:  # timeout, host crash: should be rare
    results = [{"item": i, "ok": False, "error": str(exc)} for i in items]

It's blunt (one platform failure marks the whole step as failed), but it fails in the safe direction, and it should almost never fire.

Classify before you catch: permanent failures are reported, transient ones are retried, and platform ones are caught as a last resort.

6. Choose the unit of isolation

"Isolate failures" raises an immediate question: isolate them at what level?

The unit of isolation is the smallest piece of work that can fail without affecting its neighbours. Choosing it is a trade-off:

• Too coarse, and one bad item takes out a large batch. The blast radius is big.
• Too fine, and you pay overhead per unit. In Durable Functions, every activity adds events to the orchestration history, and very large fan-outs make that history slow to replay.

Isolation doesn't have to mean "one activity per item". You can layer it:

• One activity per group of work (one per data source, or per batch of a few hundred items) keeps history small.
• Inside that activity, one database savepoint per item means a bad item rolls back alone, while the good ones commit.

for f in files:
    try:
        with conn.begin_nested():  # savepoint: this file only
            imported += import_file(conn, f)
    except Exception as exc:
        failures.append({"file": f.name, "error": str(exc)})

The crucial detail is what happens at the end. If the function raises because failures isn't empty, the outer transaction rolls back and takes the good files with it. The savepoints only help if the function returns.

For very large fan-outs, the same idea scales up: split the work into sub-orchestrations, each fanning out over a chunk, so no single history grows without bound.

Isolate at the smallest unit whose failure you can explain to an operator, and no smaller than your platform can afford.

7. Retries mean idempotency

Best-effort pipelines retry more: inside activities, across platform retries, and when an operator re-runs a partial job to pick up the failures.

Combine that with the second consequence from section 1 (activities run at least once) and you get a hard requirement:

Every activity must be safe to run twice.

That usually means:

• Upserts on a natural key, not blind inserts. Importing the same file twice should leave the same rows, not duplicates.
• Check-then-act on side effects. Before sending an email, check whether it was already sent for this run.
• Deterministic inputs. The activity should get everything it needs from its input, not from "whatever is in the folder right now".

The orchestrator has a stricter rule of its own. Because it is replayed from history, it must be deterministic: no I/O, no datetime.now(), no random numbers, no sleep. Use context.current_utc_datetime and timers instead, and put anything non-deterministic in an activity.

A best-effort pipeline that isn't idempotent turns "retry the failures" into "duplicate the successes".

8. Destructive steps need complete inputs

This is the principle that matters most, and the one that's easiest to miss.

Imagine the import has a final cleanup step: delete records that no longer appear in the source files. Upstream removed a product, so we remove it too. Sensible housekeeping.

After a complete import, "not present" means "deleted upstream".

After a partial import, "not present" might mean "failed to import this time".

Picture a librarian who removes every book that isn't on today's delivery list. On a normal day, that keeps the shelves accurate. On the day half the delivery truck broke down, it empties the shelves.

Making the pipeline best-effort is exactly what creates this risk. Before, a failed import never reached the cleanup step. Now it does.

So the rule is:

For every step, ask what it assumes about the steps before it. Destructive steps run only when everything they depend on succeeded.

any_failed = any(not r["ok"] for r in results)
 
if any_failed:
    cleanup = {"ok": False, "skipped": "upstream steps incomplete"}
else:
    cleanup = yield context.call_activity("delete_missing_records", None)

Gating is the simplest guard. Depending on how much damage a wrong delete would do, a few others are worth knowing:

• Generation marking. Stamp every record touched in this run with a run id. Only records from older generations are candidates for deletion, and only after a complete run.
• Deletion thresholds. Refuse to delete more than, say, a few percent of records in one run without a human confirming. A sudden mass delete is far more likely to be a bug than a real upstream change.
• Soft deletes. Mark records as deleted and purge them later, so a mistake is reversible.

"Best effort" is safe for additive steps: import what you can, rebuild what you can. It's dangerous for destructive ones, because a destructive step treats absence as a fact, and a partial run makes absence ambiguous.

9. Make status honest

Once a pipeline can partially succeed, "did it work?" no longer has a yes/no answer. Your status reporting has to reflect that.

Durable Functions already gives you one signal: the orchestration's runtime status. It's worth being clear that this answers a different question from the one an operator is asking:

A best-effort orchestrator should almost always reach Completed. If it ends as Failed, that's a bug in the orchestrator itself, not a bad input.

The output is where the real answer lives. A useful report has:

• an overall outcome (completed or partial),
• per-step results with counts,
• every failure by name, with a reason a human can act on,
• and skipped steps, with why they were skipped.

{
  "status": "partial",
  "steps": { "import": { "ok": 98, "failed": 2 }, "cleanup": { "skipped": "upstream steps incomplete" } },
  "failures": [{ "item": "prices-2026-10.csv", "error": "missing column: currency" }]
}

Compare that to a generic 500. One tells you what to fix. The other tells you to start digging.

A green tick that hides failures and a red cross that hides successes are both lies. partial, with names, is the truth.

10. One run at a time

Best-effort pipelines are often re-run: on a schedule, by an operator retrying failures, or by two people clicking "start" at once. Overlapping runs writing to the same data is a different kind of failure, and it needs a different guard.

The instinct is to build a lock: a row in a table, a flag in a cache. Durable Functions already has something better: the instance id, which identifies exactly one orchestration.

Use a fixed instance id for a job that must be a singleton, and "is it already running?" becomes a status lookup:

status = await client.get_status("nightly-import")
if status and status.runtime_status in LIVE:   # Pending, Running, Suspended, ...
    return json_response({"running": True}, 409)
await client.start_new("import_orchestrator", instance_id="nightly-import")

Three things make this robust rather than just plausible:

Close the race. Two callers can both see "not running" and both call start_new. By default, Durable lets you start an id that's already running, and the running instance can be replaced without any error. Setting overridableExistingInstanceStates to NonRunningStates in host.json makes that second start fail instead:

{
  "extensions": {
    "durableTask": {
      "overridableExistingInstanceStates": "NonRunningStates"
    }
  }
}

Catch that failure and return the same 409. Then test the race for real, with two concurrent triggers. See the instance management docs for the details.

Treat 409 as information, not an error. It means "a run is already in progress, here it is". A UI should show the live run, not a failure.

Know what it doesn't cover. A singleton orchestration stops two orchestrations overlapping. It doesn't stop a script, a manual fix, or another service writing the same tables. If you need that, the guarantee belongs in the database: a unique constraint or an advisory lock.

And a small one: remember that Suspended is still live.

Enforce exclusivity with something the platform guarantees, not a flag your code hopes is accurate.

11. The same principles, outside Durable Functions

None of this is really about Durable Functions. Take a weekly job that emails each organization a digest: for every organization, look up recipients from an identity provider's API, then send.

It can fail in two familiar ways:

• One organization's API error aborts the whole run. Fail-fast over independent work. The fix: isolate per organization, and per recipient within it.
• It holds a database connection while waiting on the API. One slow response and the idle connection times out, killing the run for everyone.

That second one deserves its own rule: never hold a scarce resource across a network call. Read what you need, release the connection, then do the slow external work. Otherwise your connection pool's health depends on someone else's latency.

And the observability rule carries over too: every send writes a sent or failed record, so isolated failures are visible afterwards instead of silently skipped.

Different system, same questions: what's the unit of isolation, what's transient, what's destructive, and how will anyone know what happened?

12. Working with an AI agent

Reliability work is mostly reasoning about failure. AI coding agents are good at enumerating failure modes, if you ask them to. They're less good at noticing unprompted that a step is destructive.

A few habits help:

Ask for failure analysis before changes. Something like:

For each step in this pipeline, describe what happens to the run, sibling tasks, and already-completed work when (a) one input is malformed, (b) a dependency is briefly unavailable, (c) the step times out. Then list every step that deletes or overwrites data, and what it assumes about earlier steps.

The second half is what surfaces the cleanup problem. Ask only "make this best-effort", and an agent will happily make the destructive step best-effort too. It's doing exactly what you asked.

Write the platform rules down. Orchestrator determinism, "activities in a fan-out don't raise", "destructive steps are gated": a few lines in a project instruction file prevent a lot of plausible-looking mistakes.

Ask for tests at the failure level. "One item fails, the rest succeed: assert the run is partial, the failure is reported by name, and the cleanup step is never called." That last assertion is the one you'd regret not having.

Humans decide what is destructive. Agents enumerate failure modes and write the tests that pin them down.

13. Checklist

• Each fan-in is consciously fail-fast or best-effort, not fail-fast by default.
• Units of work return a result for failures instead of raising.
• Failures are classified: permanent reported, transient retried, platform caught.
• The unit of isolation is chosen deliberately; savepoints protect good items.
• Every activity is idempotent; every orchestrator is deterministic.
• Destructive steps run only after complete success of what they depend on.
• Run output distinguishes completed from partial, with failures by name.
• Singleton jobs use a fixed instance id, with the start race closed.
• No scarce resource is held across a network call.
• Every side effect is recorded, success or failure.

14. The bigger lesson

"All or nothing" sounds rigorous. For batch work, it usually means "nothing, and no idea why".

But the opposite isn't "anything goes". Best effort is a deliberate design, with sharp edges you have to guard.

Here are the principles worth carrying into your next pipeline:

Know what your fan-in promises. Most default to fail-fast. Choose it on purpose, or choose something else.

Failures are data. Return them, count them, name them. An exception says something broke; a result says what.

Classify before you catch. Retry what's transient, report what's permanent, and keep a safety net for the platform.

Isolate at the right size. Small enough that one bad item costs one item, large enough that the platform can carry it.

Assume everything runs twice. Idempotent work makes retries and re-runs safe.

Destructive steps need complete inputs. When a run is partial, absence is ambiguous, and nothing should delete based on it.

Make status honest. partial, with names, beats both a green tick and a 500.

A pipeline that keeps the good work, names the bad work, and refuses to delete anything on incomplete data is both more useful and safer than one that gives up at the first error.

Partial failure isn't the bug. Pretending it can't happen is.

Thanks for reading.