Produce a defensible outcome for one already-authorized operation: confirmed
applied, confirmed not applied, or unresolved. A timeout, lost response, expired
local lease, or worker restart does not establish which outcome occurred.

Ambiguous API Write Recovery

Produce a defensible outcome for one already-authorized operation: confirmed applied, confirmed not applied, or unresolved. A timeout, lost response, expired local lease, or worker restart does not establish which outcome occurred.

Preserve the attempt

Identify the provider, API version, HTTP method, content type, account or tenant, original request payload, payload digest, client operation ID, original idempotency key if any, remote resource or operation ID, and send/checkpoint times. Keep credentials and private payloads out of shared evidence. A payload hash checks equality; it is not a provider-supported operation identifier.

Keep the uncertain attempt and its original approval intact. Inspect retrying workers and SDK defaults before adding another sender. If code changes are in scope, use the existing durable claim and conditional update mechanism so two recovery workers cannot independently overwrite the result. A local lock alone cannot prevent a remote duplicate after a crash. Before any permitted retry, persist the original operation identity, payload revision and final-send marker durably; do not send if that checkpoint cannot be saved. If the provider succeeds but saving its receipt fails, preserve the uncertain attempt and reconcile it again. A local persistence failure does not reverse the external effect. Never place the external write inside a database transaction callback that may automatically rerun.

Read the applicable provider contract

Verify the actual endpoint and version, key scope, payload comparison, retention window, in-progress behavior, and result lookup method. Do not invent a universal idempotency header or retention duration. HTTP semantics distinguishes a retryable operation from an arbitrary non-idempotent request; the absence of a response is not evidence that a write never happened.

Provider guarantees differ. For example, Stripe's idempotency reference distinguishes API v1's cached result, including an execution-time error, from API v2's completion behavior. Check the applicable version instead of treating every 500 as a fresh attempt. AWS's retry design discussion explains why caller identity, original intent, changed parameters and late requests matter. Neither source grants another provider the same guarantees.

Reconcile before deciding to send

Prefer the provider's documented read-only operation or resource lookup using the original identifiers. Confirm account ownership and operation identity, then compare the relevant immutable request fields. Similar copy, timestamps, or filenames alone do not identify an attempt. A successful HTTP lookup proves only that the lookup completed; inspect the returned operation state.

An empty search, partial page, permission-filtered result, deleted object or eventually consistent 404 is not proof of non-execution. Follow documented pagination and bounded status polling where relevant. If lookup is unavailable or inconclusive, retain that uncertainty rather than converting it to failure.

Evidence	Recovery action
Exact owned operation confirms the intended effect	Persist its receipt against the same attempt; do not resend
Authoritative contract and evidence establish that execution never began	Record that finding; a new attempt may follow the existing authorization and review process
Original attempt is still running	Poll its documented status within a deadline; do not start a competing operation
Outcome unknown, with a documented retry guarantee still valid for this exact attempt	An authorized retry can reuse the original key, original payload, account and endpoint; retain its result under the same attempt
Guarantee absent, expired, or incompatible with the stored request	Keep the attempt unresolved and explain the specific evidence needed; no automatic resend

Never generate a fresh key to escape an uncertain result, clear its receipt to unlock a retry, or edit the payload under the old approval. Reusing a key after its retention window may create a new operation. A parameter mismatch requires investigation, not a replacement key. If a permitted retry is also inconclusive, stop at the documented recovery boundary; do not turn it into an unbounded loop. Changing the destination or intended effect requires the normal new-operation review. Recovery does not authorize deletion, compensation or another account.

Verify the recovery behavior

Use an isolated provider fixture or authorized sandbox with an observable side-effect counter. Exercise a committed write whose response was lost, a valid same-key replay, an expired key, an inconclusive lookup and a mismatched account. Check external effects and request history, not just the client's returned text. The lost-response case should recover the exact receipt without a second write; the supported replay may issue another request but must preserve one effect. Unsafe cases must retain uncertainty without a new mutation. For the negative control, use a separate resettable fixture with explicit authorization to create duplicate effects. Disable deduplication or expire its key, inject a blind retry, and require the duplicate-effect assertion to fail. Do not run that control on the incident being recovered.

Record the tested environment, provider contract/version, original-attempt reference, evidence timestamp, outcome, reason, requests made, and observed effect count. Separate lookup time from original execution time. State which real-provider, concurrency and crash behaviors remain untested. A fixture pass does not prove production delivery or improvement across arbitrary agents.