All projects
02 / INDEPENDENT ENGINEERING DEMONSTRATORPublic without sign-in
Reliability & incident recovery

Integration LabA failed delivery should leave a useful trail.

Send signed synthetic events through real HTTP, inject receiver failures and recover with visible retries, deduplication and dead-letter history.

Open live demoView repository
WATCH THE APPLICATION

From failed delivery to recovery.

1 minute 40 seconds. Real application screenshots, synthetic narration, English captions and a full transcript.

Read the walkthrough transcript

Investigate a failed delivery

Integration Lab is a working demonstration of webhook failure and recovery. Its orders are synthetic, while signatures, HTTP requests and database writes are real.

Inspect the failure history

Run outage scenario sends four requests to an unavailable receiver. The virtual retry clock advances explicitly. The delivery becomes a dead letter, with every failed attempt still visible.

Restore service and replay

Set the receiver to Healthy, choose Replay and repair sender, then send the next attempt. The receiver records one order effect. The earlier failure history remains available.

Check duplicate safety

Queue a duplicate and send it. The receiver returns its stored response without applying the order again. Compare the delivery history with the order ledger.

Recover an ambiguous outcome

A harder case loses the response after committing. The receipt and order update are real; the failure response is injected. Retrying still leaves one order effect.

Inspect the design

The public service uses a Worker and D1. A local ASP.NET Core implementation shares its HTTP contract. The architecture board explains the receipt, retry and audit boundaries.

Understand the demonstration's limits

This is a bounded synthetic demonstration. The browser sends deliveries and controls virtual retry time. Recorded validation and source code let you inspect the behaviour and its limits.

01 / FUNCTION & ABILITIES

A failed delivery should leave a useful trail.

01

Inject controlled failures

Exercise outages, throttling, invalid signatures, expired timestamps, bad payloads and a lost response after a successful commit.

02

Inspect the delivery history

Follow correlation IDs, raw HTTP outcomes, retry deadlines, attempt cycles and the order-effect ledger.

03

Recover safely

Replay a dead letter, repair the synthetic sender and confirm that retries and duplicate events retain one effect.

04

Compare real runtimes

The public Worker/D1 API and runnable ASP.NET Core/SQLite reference implement the same contract and acceptance scenarios.

02 / PLANNED BOUNDARIES, VISIBLE FLOW

From the whiteboard
to a working application.

Play a scenario, step through the decisions or select a component to inspect its responsibility. The diagrams follow the implemented design.

THE ARCHITECTURE BOARD02 / Integration Lab
Illustrated flow
Integration & webhook reliability lab: architecture and data flowThe browser delivers a reserved envelope to a fixed receiver. Retry scheduling uses an explicitly advanced virtual clock; it is not a background queue. Select a component or use the readable step list below.VISITOR BROWSERPUBLIC WORKER + D1 / EQUIVALENT LOCAL .NET APIqueue / claimreserved signed envelopereal HTTP POSTvalidate + commitretryable failurewhen duebudget exhaustedexplicit replayretained historyThe visitor queues synthetic work, obtains one reserved signed envelope and posts it to the fixed receiver.Operations consoleBrowser HTTP senderThe API claims a delivery attempt and returns its signed envelope without exposing the session signing secret.Synthetic sender APIReserve + sign envelopeThe receiver checks timestamp, signature, raw bytes, event identity and order version before committing an effect.Fixed HTTP receiverHMAC + idempotencyThe optimistic update commits the receipt, business effect, attempt status and audit transition together.Atomic SQL outcomeReceipt + one effectThe operator explicitly advances virtual time. A retry uses the recorded deadline rather than an invisible timer.Virtual retry clockBackoff / Retry-AfterExhausted or non-retryable work remains inspectable. Replay opens a new attempt cycle and keeps prior evidence.Dead-letter reviewReplay + repair senderCorrelation IDs, attempts, outcomes and the ledger can be inspected and exported without access credentials.Audit & evidenceAttempts + safe exportfollow the evidence, then the decision
Step 1 / 8 · click any component
FOLLOW THE FLOW

Reserve a delivery

The console asks the API to claim one synthetic delivery attempt.

Read all 8 steps without animation

Follow a failed delivery into dead letters, then replay it safely.

The browser delivers a reserved envelope to a fixed receiver. Retry scheduling uses an explicitly advanced virtual clock; it is not a background queue. Arrows and moving dots are an explanation of the implementation, not live telemetry.
03 / A FIVE-MINUTE WALKTHROUGH

Try it for yourself.

Keep the live demo open
  1. 01

    Run an outage

    Select Run outage scenario. Four actual HTTP requests fail while the explicitly virtual retry clock advances by 5, 15 and 45 seconds.

  2. 02

    Inspect the dead letter

    Open its correlation ID, attempt responses and order ledger. Confirm that the failed delivery has not created an order effect.

  3. 03

    Restore the receiver

    Select Healthy, then Replay & repair sender and Send next attempt. The receipt and one order effect are committed while failed history remains.

  4. 04

    Send the duplicate

    Select Queue duplicate and send it. Inspect the cached success and confirm that the order ledger still has one effect.

  5. 05

    Test uncertain delivery

    On a new order, select Lose response after committing. Retry after advancing the clock and verify that the already committed effect is not duplicated.

04 / ENGINEERING JUDGEMENT

Why it works this way.

Commit the outcome together

Receipt, order effect, delivery status and audit transition commit in one optimistic SQL update. A concurrent loser reloads and recomputes.

Keep retries inspectable

Backoff, Retry-After handling and replay are explicit state transitions. Operators can see why an event is due, rejected or in dead letters.

Use a fixed receiver

The browser posts a reserved signed envelope to one controlled receiver endpoint. The session signing secret is not returned to the browser.

05 / INSPECT THE EVIDENCE

Tested behaviour.
Explicit limits.

  • Shared HTTP acceptance checks exercise the real .NET and Worker APIs, persistence, session expiry and concurrent updates.

  • Published validation covers rejected authentication, replay, deduplication and lost-response recovery.

  • The source includes the contract, state model, retained failures and an operator runbook.

Read the recorded validation