Resonate

Further reading Making retries safe with idempotent APIs (Amazon Builders' Library) · Fallacies of distributed computing (Wikipedia) · Futures and promises (Wikipedia) · What is Durable Execution? (Resonate)

1. i.

   Did it happen?

   Say a request times out. Did the other side do the work? There is no telling, and simply retrying could launch two machines where you wanted one. Finding out means a reconciliation step, a lot of heavy lifting for an edge case. The general version, agreeing on anything over a link that can drop messages, was the first computer communication problem proven to be unsolvable.

2. ii.

   Retries need idempotence

   Retrying is still the best medicine, since a surprisingly large number of transient faults go away on a second try. But a retry is only safe if the operation can be repeated without extra side effects, a property called idempotentSafe to repeat: doing it twice has the same effect as doing it once.. Making operations idempotent pushes the problem to the service, which now has to recognise that a request is a repeat of one it has already seen.

3. iii.

   Memory that dies

   A function's call stack lives in the memory of one process. When the process dies, every local variable and pending call dies with it. Deploys roll, machines get recycled and containers move, while the work itself can take hours or months. A plain retry starts the function over from the top, and repeats whatever the first attempt already did.

4. iv.

   Scaffolding everywhere

   So teams write retries, timeouts, idempotency keysA unique ID sent with a request so that a repeat can be recognised and ignored. and state machines that remember where a job was when its process died. Resonate puts that defensive scaffolding at "more than two-thirds of what you ship", and whatever the true share, almost none of it is the product.

Further reading Making retries safe with idempotent APIs (Amazon Builders' Library) · Two Generals' Problem (Wikipedia) · What is Durable Execution? (Resonate) · Resonate - Fully Open Durable Execution (Resonate)

Further reading What is Durable Execution? (Resonate) · Resonate - Fully Open Durable Execution (Resonate)

1. Step 1: Promises that outlive processes

   The basic unit is a durable promise: a promise with a home outside any process, written down the moment it's made and settled when the work is done. When a machine crashes, a new process on any machine reads the last promise that was kept and continues from there. The process can die. The promise is still kept.

2. Step 2: Replay what's done

   On recovery the function starts again from the top. Steps whose promises already resolved hand back their recorded results instantly, and the first unfinished step runs for real. A step that was mid-flight when the process died runs again, so a charge against a card network still needs to be idempotent. Each promise's ID doubles as the idempotency key for that step.

3. Step 3: Three verbs

   Most code rests on three calls: run a step durably, call another function, or sleep, for a second or a month. The server is a single binary whose state lives in a Postgres database the team already runs, readable with plain SQL.

4. Step 4: A spec you can check

   The protocol is written down as an executable, machine-checkable model in Lean 4. The server is tested by pushing the same random operations through several storage backends and a reference model at once, and every one must answer the same way.

Further reading What is Durable Execution? (Resonate) · How Resonate works (Resonate) · Resonate - Fully Open Durable Execution (Resonate) · How Resonate is tested (Resonate)

• What happens when the code changes mid-flight?

  Replay only works if a function does the same thing the second time around, which is why durable functions must be deterministicAlways doing the same thing, in the same order, given the same inputs.. A workflow that sleeps for a month may wake up to a new version of its own code. Temporal, another durable execution system, handles this with patches that work like feature flags, applying a change to new executions without disrupting ones already running.

• Can anything run exactly once?

  Over a link that can lose messages, perfect agreement is provably impossible. What these systems offer instead is a function that runs effectively once and to completion, built from recorded results, retries and idempotent steps.

Further reading Constraints and gotchas (Resonate) · Temporal Workflow Execution overview (Temporal) · Patching (Temporal) · Two Generals' Problem (Wikipedia) · How Resonate works (Resonate)

distributed program

Software whose work is split across several processes or machines that talk over a network.

async/await

A way of writing code that pauses at a slow operation and resumes when its result arrives.

promise

A placeholder for a result that isn't ready yet.

idempotent

Safe to repeat: doing it twice has the same effect as doing it once.

durable execution

Running a function so that its progress is saved and it can resume after a crash.

idempotency key

A unique ID sent with a request so that a repeat can be recognised and ignored.

deterministic

Always doing the same thing, in the same order, given the same inputs.

1. 1Making retries safe with idempotent APIs · Amazon Builders' Library
2. 2Fallacies of distributed computing · Wikipedia
3. 3Futures and promises · Wikipedia
4. 4Two Generals' Problem · Wikipedia
5. 5What is Durable Execution? · Resonate
6. 6Resonate - Fully Open Durable Execution · Resonate
7. 7How Resonate works · Resonate
8. 8Constraints and gotchas · Resonate
9. 9How Resonate is tested · Resonate
10. 10Temporal Workflow Execution overview · Temporal
11. 11Patching · Temporal