AppSurface Search
Guide

PostgreSQL Work protocol v1

Source of truth

View source Edit this page

Last updated

This is the normative operation and lock manifest for Durable slice 3. It specifies observable behavior, transaction boundaries, and ordering. Internal C# and SQL may be decomposed differently, but must preserve these invariants. The package starts no worker; tests and later hosting code drive one operation at a time.

Typed Work definitions are an authoring projection over this same protocol. The definition captures contract facts and builds the existing request; it does not add a persistence state, change the fingerprint, or authorize execution. See the typed Work definition migration guide for syntax-only replacement, semantic/exit rollout, duplicate replacement, exact Flow registration lookup, and caller concurrency obligations.

Authoritative records

PostgreSQL is the only durable truth. The Work/shared schema owns store metadata and migration history, scope tombstones and generation, payload-free dispatch, Work and its immutable acceptance snapshot, append-only history/stale observations, effect permits, and runtime epoch history.

  • A terminal fact is the immutable history row committed with a terminal Work state.
  • A permit records the exact attempt, lease generation, scope generation, runtime epoch, and activity identity that authorized one provider operation. A claim alone never authorizes an external effect.
  • A quarantined observation is append-only evidence from a stale completion. It never changes current Work.
  • ActivityId is immutable and is also the Provider contract's ProviderKey; there is no independent key hash.
  • Notifications are optional payload-free latency hints. Discovery remains authoritative.

Global lock order

Every operation locking more than one protocol record uses: scope -> Work -> dispatch -> permit -> history.

Schema application instead takes one package session advisory lock before reading migration history and holds it through history validation, migration transactions, and final status validation. Runtime mutations take the shared, transaction-scoped form of that lock before validating the active epoch, so migration or epoch rotation waits for in-flight mutations and a stale epoch cannot commit afterward. Provider I/O never occurs while a database transaction or connection is held.

Operation manifest

Operation Transaction and locks Required validation Result and durable effects
Get schema status Read-only deployment connection Migration hashes and reader/writer ranges Missing, compatible, inconsistent, too old/new; includes StoreId and nullable active epoch.
Generate script No database connection Caller supplies the exact reviewed installed version Deterministic forward-only SQL for the selected pending migrations; never rerun blindly.
Apply migrations Migration owner; session lock for full operation History before/after each migration Applies pending known migrations only; mismatch fails closed.
Initialize epoch Deployment transaction under schema lock StoreId, null active epoch, non-empty candidate/audit One-time activation plus append-only history; never done by a writer.
Rotate epoch Deployment transaction under schema lock StoreId and expected active epoch CAS Records previous/new epoch and audit; stale expectation changes nothing.
Accept Work Caller/client transaction; scope then Work/dispatch Target, StoreId, epoch, registry, fingerprint, active scope Atomically creates generation-1 scope, Work, dispatch/history, or exact duplicate/conflict.
Discover Short dispatcher transaction; payload-free function access (discover_work_dispatch) Requested contract names + versions, due/state only Registry-scoped bounded candidate identities; lost notification cannot hide Work.
Claim Short scoped transaction; canonical lock order Scope, revision, due state, epoch; classify expired permit One winner advances attempt/lease; unsafe expired permits suspend.
Renew lease Short scoped transaction; canonical lock order Exact revision, attempt, lease/scope generation, epoch, owner, lifetime Extends exact active lease; stale renewal changes nothing.
Prepare No store transaction Registry decodes immutable contract/payload/safety Produces landed prepared invocation; unavailable history suspends ASDUR109 before permit.
Acquire permit Short scoped transaction; canonical lock order Exact current claim, active scope/epoch, no pre-effect cancellation Commits one immutable exact-fence permit before provider I/O.
Prepare application handoff Caller ReadCommitted transaction; shared migration fence then scope -> Work -> dispatch -> exact permit, followed by app locks Active exact invocation, target/schema/StoreId/epoch, scope generation, claim/lease owner/activity, live lease/deadline, granted exact permit and opt-in admission Application-owned callback once; provisional preparation until caller commit. Denial/exception/refusal removes callback writes; native locks remain through caller completion for callback outcomes. No native transition or provider I/O. See handoff contract.
Complete Short scoped transaction; canonical lock order Exact permit/claim; result matches registration Commits terminal fact, retry, or suspension; stale result is quarantined only.
Cancel Short scoped transaction; canonical lock order Fingerprint and current state Before permit terminalizes canceled-before-effect; after permit records intent and follows safety.
Disable scope One scoped transaction; canonical lock order Active generation and actor/reason Permanent tombstone and atomic suspension of nonterminal Work.
Recover expired permits Payload-free discovery then scoped candidate transaction Every expired permit and exact Work fence Safe classes may reclaim; unsafe classes suspend without repeating effects.
Reconcile/resolve/release Short scoped transaction; canonical lock order Host authorization, command fingerprint, state/evidence/audit Applies only the safety transition supported by evidence and appends history; recovery release moves exact ambiguous evidence with the authorized epoch and rolls back if an expected exact move fails. Legacy Work selects its current attempt; opted-in Work selects the most recent unresolved admitted permit, including an earlier attempt. Recovery releases with no uncertain effect retain ordinary timing checks.

Prepare application handoff

The handoff fence validates an existing exact native invocation in the application's caller-owned ReadCommitted transaction. It acquires the shared migration fence before schema/StoreId/epoch checks, then the scope, Work, dispatch and exact permit rows in the global order. Application authority and ledger locks follow those native locks inside the callback. The validator handles legacy Work and opted-in invocation admission explicitly, uses current locked renewal facts, and never creates scope, grants another permit, consumes admission, or writes history.

Successful preparation is provisional until caller commit. Callback denial, exception and post-callback refusal remove callback writes while retaining acquired native locks through caller completion; cancellation observed at the final check removes operation writes using the enclosing savepoint. Pre-callback refusal may release operation locks. The host owns finite lock, statement and idle-transaction limits and total transaction duration. Caller rollback grants no handoff; failed or ambiguous commit requires ledger reconciliation. Provider I/O follows resource release inside the current invocation and the existing provider safety rules.

Acceptance identity matrix

CommandId and IdempotencyKey are both mandatory and independently unique within a scope.

Resolution Fingerprint Outcome
Neither key resolves N/A Accept new Work even if an unrelated row has the same fingerprint.
One key resolves Same known schema/hash Return Duplicate with original WorkId, CommandId, revision, and accepted time.
Both keys resolve to the same row Same known schema/hash Return that same Duplicate.
One/same-row keys resolve Different or unknown fingerprint Fail ASDUR102; do not mutate.
Keys resolve to two rows Any Fail ASDUR102; do not choose a winner or mutate.

Concurrent first use creates one active generation-1 scope. A disabled scope is a permanent tombstone and returns ASDUR107; it is never silently reactivated.

Provider-effect safety

Safety Unknown outcome after permit Applied evidence Not-applied evidence
Idempotent Exact-fence recovery may retry. Encoded result may terminalize success. Retry under a new claim/permit.
ProviderKeyed Recovery may retry with the same ActivityId/ProviderKey. Encoded result may terminalize success. Retry with the same stable key.
ReconcileBeforeRetry Suspend for reconciliation. Reconciliation may terminalize with encoded result. Only explicit proof may authorize retry.
ManualResolution Suspend for manual resolution. Applied requires encoded result and terminalizes. ProvenNotApplied may authorize retry only via operator command.

No unknown post-permit outcome becomes FailedTerminal. Cancellation after permit does not change this matrix.

Typed executor exits

An exit-aware IDurableWorkExitExecutor<TWork,TResult> can return only an executor fact. It cannot select a Work state, set retry delay, bypass a fence, attach arbitrary metadata, or classify exceptions. V1 registration is fixed to ProviderKeyed; legacy IDurableWorkerExecutor<TWork,TResult> registrations preserve their exact behavior. The adopter exit-aware Work guide defines the public factories and application-code boundary.

Returned executor fact Completion fact after a committed permit Required PostgreSQL behavior
Succeeded(result) Succeeded Persist the encoded result under the existing success/cancellation rules.
RetryBeforeEffect(code) ProvenNoEffect The executor proved it did not begin a provider mutation; read-only provider I/O is allowed. Apply the normal retry/deadline/cancellation/fence rules and record proven_no_effect; do not infer this fact from a read success, exception type, cancellation, or absence of a permit.
FailedTerminal(code) FailedTerminal This is a local terminal fact, not proof that a permitted provider effect was absent. Apply the existing post-permit provider-safety matrix.
AmbiguousExternalOutcome(code) AmbiguousExternalOutcome Preserve ambiguity and apply the existing reconciliation/manual-resolution behavior.
Exception, cancellation, lease loss, codec failure, or legacy-boundary compatibility failure AmbiguousExternalOutcome with ASDUR106 Treat the post-permit outcome as unknown.

Application codes are bounded, identifier-safe, application-owned facts (for example, app.gmail.sender_list_transient) and cannot use the reserved ASDUR prefix, case-insensitively. The provider must not put raw response data, payload bytes, provider diagnostics, or exception text in an application exit code.

Exit-aware Work requires a provider that calls InvokeExitAsync. An old success-only provider can preserve a success, but a non-success exit throws DurableWorkExitCompatibilityException and follows the ASDUR106 path. Rolling deployment is therefore ordered: deploy capable provider/runtime binaries, restart every worker that can discover the contract, register and accept a new immutable exit-aware Work version, then observe it. To roll back, stop accepting that version but retain capable workers until it drains or is intentionally suspended; never return to an old-only fleet that could claim an accepted exit-aware version.

Retry v1

Only exponential-v1 is supported. After failed attempt n >= 1:

delay = min(MaximumRetryDelay, InitialRetryDelay * 2^(n - 1))
eligible_at = PostgreSQL clock_timestamp() + delay

Calculation is cap-first and overflow safe, with no jitter or provider Retry-After. Unknown algorithms and invalid bounds fail before mutation.

Execution-policy v1

The optional execution-policy reference defines the additive deadline-only and fixed-plan contracts. Existing requests retain fingerprint schema v1 and completion-relative exponential-v1 behavior. Opt-in requests use semantic fingerprint v2 and persist their immutable policy with acceptance. For a plan, offset i is anchored to accepted-at A, attempt number i + 1 becomes eligible at A + O[i], and the exclusive admission cutoff is the minimum of A + MaximumCircuitDuration, A + MaximumElapsedTime, and the optional deadline. For deadline-only policy it is the minimum of the elapsed horizon and absolute deadline; retry spacing remains legacy backoff.

The PostgreSQL timestamp is sampled under the authoritative lock. Equality with a cutoff is closed. A request's acceptance instant is its insert time, so caller-owned transaction delay consumes the window. Overdue offsets are not rebased or automatically consumed: every next attempt still requires the existing safe sequential retry transition. The execution snapshot is descriptive; claim, effect permit, invocation admission, retry release, and completion must revalidate current fences and timing in the store. Deadline expiry never proves that an external effect did not occur. The store preserves two separate internal timing witnesses. execution_admission_closed_at and execution_admission_closed_reason retain the first observed admission closure and are never overwritten by a later cutoff. execution_deadline_reached_at independently retains the first authoritative observation at or after the absolute deadline, even when circuit or slot exhaustion closed admission earlier. These storage facts add no public API member. See the outcome and diagnostics rules and schema-12 migration procedure.

Caller-owned transaction contract

Local preflight failures before SQL preserve transaction usability: inactive/wrong target, empty epoch/StoreId, unregistered Work, invalid payload/policy, or unsupported retry policy. Expected database domain outcomes also preserve usability: disabled scope, exact duplicate, command conflict, or stale fence.

PostgreSQL errors, timeout, network loss, server cancellation, or any aborting SQLSTATE require rollback. The writer never commits, rolls back, disposes, replaces, or opens a second connection for the caller's transaction. Savepoints are unsupported. Endpoint/database comparison is a configuration guard; durable identity is the StoreId read through the supplied transaction and compared with the deployment-time value.

Security and diagnostics

The migration owner alone owns DDL. The dispatcher reads payload-free discovery. The scoped runtime has minimum DML under forced RLS; PUBLIC has no privileges and runtime roles have neither ownership nor BYPASSRLS. Transaction-local scope context is defense in depth, not application authorization: a credential holder can deliberately select a scope. The dispatcher does not read appsurface_durable.dispatch directly for Work discovery; instead it calls appsurface_durable.discover_work_dispatch(text[], text[], integer) defined by migration 0009 and constrained by policy work_contract_discovery_owner.

Diagnostics preserve ASDURxxx, concrete Npgsql/PostgreSQL exception types, inner exceptions, stack traces, and SQLSTATE, but exclude connection strings, credentials, parameter values, payloads, and provider responses. See the ASDURxxx catalog.

Scale and contention gates

The strict PostgreSQL verification gate uses PostgreSQL 16.5 via postgres:16.5@sha256:53f3e608f9475ce120ced2d0f430b89458d7faa28530e0b0977a6af64d294877 and must prove all of the following:

  • registry-scoped Work discovery uses discover_work_dispatch and ix_work_contract_dispatch_lookup across 100,000 Work/dispatch rows in 100 scopes;
  • 32 simultaneous claimers sharing an eight-connection pool complete within 30 seconds and produce one lease owner;
  • disabling a scope projects 10,000 pre-effect Work items and dispatch rows within 30 seconds; and
  • concurrent migration owners serialize, canceled lock waiters do not leak the session lock, and a terminated migration connection can retry safely on a new physical session.

The checked-in reference-workload JSON manifests in this repository still record PostgreSQL 17.5 evidence and are preserved as historical proof for earlier releases.

These are regression gates, not throughput promises. Operators must benchmark their own payload sizes, retention, indexes, PostgreSQL settings, and contention patterns before setting production capacity.