Incremental Lookup and Resumable Live Results

Abstract and status

This proposal adds bounded snapshots and a durable live change feed to overlay lookup. A client can consume the beginning of a large catalogue, finish a coherent snapshot, then receive changes without losing the changes that happened during the scan. Reconnection resumes a retained sequence or explicitly starts recovery.

It is an opt-in companion to BRC-24, using the common representations and source semantics in BRC-192, and negotiated through BRC-194. Existing /lookup behavior is unchanged. This version uses complete HTTP response bodies and bounded long polling, so existing BRC-103/104 response authentication can verify each body before its contents are trusted. It does not redefine an unverified prefix of a signed body as authenticated streaming.

1. Scope and capability

The profile identifier is https://bsv.brc.dev/overlays/0193#lookup-live-v1. Supporting it requires both incremental snapshots and live replay. A finite-only service MUST NOT advertise this profile. The provider binds every session to its identity, chain, lookup service, query, rules, access scope and epoch. The Scope representation is defined by BRC-192; provider is the authenticated identity key, or the exact HTTPS origin for an explicitly unauthenticated public service.

This version serves public or access-controlled, uncharged queries. It MUST NOT charge per poll or silently invoke a payment agent. Private paid acquisitions use BRC-195 separately. The lifetime authentication mode is selected at open and cannot downgrade. TLS is required in production; HTTP requires an explicitly configured local development transport.

2. Messages and endpoints

All requests are POST with Content-Type: application/json, Cache-Control: no-store and a version-1 body. Each response is a complete JSON object. These suffixes are appended to the advertised base URL path without dropping that path. Redirects are not automatically followed.

/overlay/v1/lookup/open accepts:

type Open = {
  version: 1; requestId: string; service: string; query: unknown;
  requiredRulesDigest?: Hex32;
  limits: { maxBytes: U32; maxObservations: U32; waitMs: U32 };
  extensions?: Record<string, unknown>; critical?: string[]
}
type Read = {
  version: 1; session: string; cursor: string;
  limits: { maxBytes: U32; maxObservations: U32; waitMs: U32 }
}
type Close = { version: 1; session: string }

/overlay/v1/lookup/read accepts Read; /overlay/v1/lookup/close accepts Close and returns {version:1, closed:true}. Close is idempotent for the same principal. Session/cursor identifiers are opaque, unguessable, at most 1024 UTF-8 bytes, and scoped to the principal or anonymous session capability. A session ID is not authority to impersonate an authenticated principal. Open request IDs are stable retry identifiers, 16–128 ASCII letters, digits, _ or -, with at least 128 bits of randomly generated entropy. Same principal and request ID with changed open parameters is conflict.

Open returns a Batch and read returns the next Batch:

type Group = {
  id: string; sequence: U64; observations: Observation[]
}
type Batch = {
  version: 1; session: string; scope: Scope;
  phase: 'snapshot' | 'live'; groups: Group[];
  cursor: string; snapshotComplete: boolean;
  through: U64; highWater: U64;
  expiresAt: U64; replayUntil: U64;
  limits: { maxBytes: U32; maxObservations: U32; waitMs: U32 };
  extensions?: Record<string, unknown>; critical?: string[]
}

queryDigest = digest("lookup-query", {service, query}). The provider's rulesDigest identifies the exact selection, ordering and visibility rules. A requested unequal rules digest fails before opening. scope.access identifies the evaluated authorization policy and principal partition without disclosing a credential. epoch changes when continuity or those semantics cannot be preserved.

Effective limits are the componentwise minimum of request, provider and profile limits. Every limit is positive except waitMs, which can be zero. Profile maxima are 4 MiB complete response body, 1024 observations and 25000 ms wait. Requests exceeding maxima are clamped and the result is returned. A provider MUST support at least a 64 KiB response allowance, 1 observation and immediate reads; this minimum allowance does not promise every possible group fits it. A client whose byte limit cannot hold the envelope or next whole group gets the bounded limited response specified in section 8, not a truncated group. It can explicitly raise its limit or use another service. No hidden payload download occurs outside these budgets.

3. Snapshot and live boundary

On open, the provider atomically establishes an immutable query snapshot S and log watermark W. All visible mutations through W are reflected in S; every relevant mutation after W is recoverable from the retained log. An MVCC snapshot, immutable versioned index or equivalent transactional construction is acceptable. Offset pagination over a changing live table does not satisfy this requirement.

Snapshot groups are emitted in a deterministic provider-defined order fixed for the session. Until the last snapshot batch, snapshotComplete=false, phase=snapshot, through=W. The last snapshot batch sets snapshotComplete=true and still has phase=snapshot. Its cursor starts replay strictly after W; the following response has phase=live. This mandatory empty-or-nonempty boundary prevents a final snapshot page from being confused with a log update. An empty collection still has a completed snapshot batch.

Snapshot group and observation IDs are stable for the retained session and occupy a session-specific namespace disjoint from live IDs. Snapshot group sequence is W. Live groups have stable IDs within the epoch and are delivered in strictly increasing provider log order. through is the highest examined provider sequence, including irrelevant records skipped by this query; highWater is the highest committed sequence known when building the response. Both are U64; through <= highWater. During a snapshot through=W. During live replay through never decreases. Sequence gaps therefore do not by themselves signify missing query results.

Each committed domain change group is indivisible: for example, removing a spent listing and adding its replacement cannot be exposed as two separately committed client revisions. Groups may span multiple topics only when the provider's declared rules promise that scope. One transaction need not produce a single universal event across unrelated providers. If a group exceeds the selected budget, return limited with no cursor advancement; do not hide it.

Membership transitions are query-specific. An entering output produces output; a leaving output produces withdraw; a relevant actual spend can additionally produce spend evidence. A changed context for an existing row produces a new output observation. A provider MUST describe all query membership changes its rules promise, including expiry and local eviction. An expired row MUST NOT disappear silently from a live query. A chain-view reset is either a coherent invalidation/reassessment group or reset-required; it is not silently represented as a confirmed spend.

4. Durability, retry and cursor meaning

The cursor returned by a batch denotes the position immediately after that batch. A client MUST durably record all observations or durable, bounded pending work for them, and their source checkpoint, in one local commit before using that cursor. Successful network receipt alone is insufficient. Verification and projection may follow later; the UI must expose their pending state. A corrupt or unauthenticated batch is not committed locally. There is no provider acknowledgement in this read-only cursor protocol. Multiple cursors MUST NOT be advanced concurrently for one client session branch.

Read is non-destructive. Repeating the same (session,cursor) is permitted and MUST NOT skip a relevant group. A nonempty committed result or snapshot page is replay-stable for the retained session: IDs and contents cannot change, although a retry may contain a different bounded prefix if the client changes limits. Empty live polls can observe newer data. Consumers deduplicate by observation identity, not by arrival count or HTTP request ID. If a process crashes after commit and before its next request, replay is harmless.

expiresAt is session expiry; replayUntil is the provider's promised retention deadline for the returned cursor. Neither may exceed what the provider can actually retain. Reads made before both deadlines, under unchanged authorization, MUST resume or report an operational failure; they cannot deliberately discard promised history. Version 1 uses fixed deadlines established at open; it does not extend them on activity. Storage loss or epoch discontinuity produces reset-required, never an empty successful stream. Signed authentication does not make an operator's durability claim infallible.

A long poll returns promptly after an eligible committed group or at the selected wait bound. At timeout it returns an empty live batch with coherent watermarks. Concurrent writes MUST wake the waiter without a scan/register race: observe watermark and install the wake mechanism atomically, or recheck after installing it. Disconnect and cancellation release the waiter; they do not consume events. A durable restart retains the epoch/log or forces reset. Internal overlay propagation outboxes alone do not meet this retained query log contract.

5. Reset, authorization and source adapters

An expired cursor, lost snapshot, changed query/rules, lost log interval or incompatible epoch returns reset-required. The client opens a new snapshot and reconciles source membership using a new generation. Old records may remain visibly stale while replacement is incomplete; they MUST NOT be reported as freshly verified absent. After the replacement snapshot is complete, remove only that source's unseen memberships in one coherent revision. Do not delete another provider's receipt, evidence history or a paid acquisition.

Authorization is checked on every request and immediately before serializing each batch. Revocation terminates the session with unauthorized and invalidates its cursors. Renewing credentials can retain a session only if the evaluated access scope is identical. Permission widening/narrowing that changes query results requires a new epoch/snapshot. Log retention is not permission to disclose past private rows after revocation.

Overlay Express integrations use optional lookup/source companions that can create a consistent snapshot, read bounded pages, retain coherent mutations and wait for changes. Existing LookupService.lookup() implementations remain valid finite services. A wrapper over that finite method MAY supply a finite adapter; it MUST NOT claim this profile unless it actually implements the snapshot/log boundary. Database drivers must propagate cancellation and bound hydration, rather than assembling the entire formula or BEEF set before emitting its first batch.

6. Errors and compatibility

Error responses are {version:1,error:{code,message,retryable,retryAfterMs?}}, with no successful cursor. Codes use BRC-192 plus not-found. HTTP mappings are 400 invalid, 401 unauthorized, 404 not-found, 409 conflict/equivocation/reset-required/context-changed, 410 expired, 413 limited, 422 unsupported and 503 unavailable; retryAfterMs is a bounded U32. A transport cancellation is local, not a fabricated server response. Unknown errors cannot be interpreted as empty lookup success. An authenticated error is verified before it changes trusted state. A 402 here is an unsupported charging behavior and MUST NOT trigger automatic funding.

Providers SHOULD rate-limit open sessions, retained bytes, pending waiters and per-principal work, and return explicit capacity errors. Cursors and private responses MUST NOT enter public logs, shared caches or GASP messages. GASP remains available to both strict and federated providers; this client feed does not require all providers to have the same data or a shared global sequence.

7. Required conformance

Exercise a mutation before the snapshot boundary, during the last page and between registering a waiter and sleeping; none can be lost. Also exercise empty snapshots, duplicate pages, commit-before-crash, cursor-before-commit rejection, sparse sequences, expiry, large atomic groups, cancellation, restart, provider reset, rule changes, authorization revocation during a poll, source-specific reconciliation and reorganization. A live demonstration MUST show a second client learning newly admitted state without reload and recovering a change missed while disconnected. Packet fixtures define reproducible boundary examples; implementation acceptance additionally needs a real durable adapter and HTTP/authentication tests.

8. Session state, clock and transport contract

Open has durable states absent, open, closed and expired. Its idempotency key is (selected service, authenticated principal, requestId); in anonymous mode the high-entropy requestId is the opening bearer capability and substitutes for principal until a session exists. It must be protected like a cursor. Store the complete normalized Open, selected manifest, generated session, exact first Batch, snapshot evaluation time T, W and deadlines in one transaction before responding. Identical retry returns that saved first Batch, even if reads advanced; changed parameters conflict. Never create a second snapshot for that key. Closing/expiry retains a permanent compact keyed digest fence: a repeated Open then returns expired, never a new snapshot. Implementations unable to retain opening fences indefinitely may reject new opens for an explicitly retired service epoch; they may not recycle identifiers under a still accepted epoch.

For open committed at T, expiresAt=T+sessionSeconds; replayUntil=expiresAt+replaySeconds, with checked arithmetic and session/replay parameters frozen from the selected manifest. New reads require now < expiresAt and now < replayUntil. Retention through replayUntil preserves crash/replay records, not permission to read after session expiry. The provider may compact snapshot/log bytes then, retaining the opening/closed fence. Unknown or expired reads return reset-required. A close with a syntactically valid identifier always returns closed:true, including unknown, expired or inaccessible sessions, without revealing existence. Only an authorized matching session is mutated. Close invalidates future serialization; a poll whose response gate preceded close may complete, one whose gate follows close returns expired. Repeating close is harmless.

Establish T, apply every timer-driven expiry with deadline <= T to the query index/log, and capture S/W atomically. A row is in S only if authorized, present and not expired at T. Ordinary data expiry after T remains in S and produces an ordered withdraw after W, including an expiry during the last page. Before declaring live completeness through a later watermark, the provider processes all timers due through that watermark's evaluation time. Access revocation, root serving prohibition and destructive privacy-policy changes are different: they prohibit disclosure now. They invalidate the session and return unauthorized or reset-required before another prohibited row is serialized; they cannot silently edit immutable S. This rule also applies to replay of a historical live output. A snapshot expired row may thus be observed followed by a withdrawal, whereas a currently prohibited row must never be newly serialized.

Every group's observations equal the batch scope. Within an epoch and query scope, a live group ID maps to exactly one sequence and immutable ordered observation array, independent of session/limits. An observation ID maps to exactly one group in its namespace. Snapshot groups are ordered by the provider's registered query rules; live groups have strictly increasing sequence greater than the incoming through. Clients reject decreasing, duplicated-with-different-content or out-of-range sequences (sequence > through in the returned Batch, or through > highWater). A sparse sequence is permitted. Snapshot/live namespaces avoid collisions when a fact appears in both. Rebatching may change the prefix length, not group boundaries, contents, order or identity. The client checks these invariants and the final snapshot boundary; it trusts the authenticated provider's assertion of selection completeness. Neither hashes nor sparse watermarks prove that an omitted group did not exist.

HTTP Content-Encoding MUST be identity for this profile. Limits count the complete UTF-8 response body, including envelope, before parsing; the client enforces the actual length rather than trusting the echoed limits. Requests are at most 1 MiB, response bodies 4 MiB, and total HTTP header field bytes at most 16 KiB (HTTP/2/3 measured after header decompression). Cursors/base64 do not exempt body limits. Decoded BEEF work is bounded separately by BRC-192 verification context; a received group can remain pending/limited without being mislabeled invalid. limited has optional limit:{kind:"envelope"|"group"|"permanent-group",minimumBytes?:U32,minimumObservations?:U32} inside error. Its complete response is at most 4096 bytes independently of a too-small requested maxBytes, contains no private payload or successful cursor, and never advertises a minimum above hard maxima. Clients must budget this separate error bound. A permanent-group failure requires a different explicitly selected profile/service or an application change; repeated reset/open cannot make it consumable. Providers must bound groups at mutation admission, or stop promising complete consumption under this profile and return this explicit failure.

The long-poll deadline is min(request arrival + effective waitMs, session expiry); expiry at serialization returns reset-required, not a successful batch with an expired promise. All responses, including errors, send Cache-Control: private, no-store and Vary: Authorization, x-bsv-auth-identity-key, x-bsv-overlay-capability, x-bsv-overlay-profile. Browser endpoints expose all required BRC-104 and selection response headers and allow only configured requesting origins/credentials; wildcard credentialed CORS is forbidden. Proxies must disable buffering transformations/compression, preserve signed headers/body, disable shared caching and set upstream timeouts above 25 seconds plus bounded transport overhead. Reauthentication may resend transport requests; it cannot advance a cursor or invoke payments.

Clients verify the entire BRC-104 response and selected peer/headers before parsing observations into trusted receipt commits; no application event is published from a truncated, corrupt or unverified prefix. The client checkpoint store, not the server, rejects advancement before a receipt/pending-group commit. Server crash recovery covers snapshot/log/open response atomicity and waiter registration; client recovery covers receipt commit, dependency hydration, verification acceptance and projection separately. The trace contract names these crash points without confusing a read cursor with a remote acknowledgement.

9. Reconciliation across sources and delayed workers

A provider log orders that provider's membership assertions. It does not assign a global transaction order or make a source authoritative over Bitcoin history. Clients implement BRC-192 section 10 for actual spend edges, durable first-eligible selection, non-final replacements, selected-chain overrides and reorganization. Complete evidence from another authorized source can settle a dependency or reveal that an output in this snapshot has already been spent. A delayed snapshot row cannot make that output current again.

Receipt can advance through a durably queued group while acceptance still waits. Acceptance of membership in one source generation MUST follow the retained snapshot order and then ascending live sequence, regardless of verification completion. A group awaiting verification blocks later membership publication in that same generation; it does not stop receipt within bounded pending budgets. At the final snapshot boundary, all preceding snapshot groups must be accepted before the source is marked complete or its live membership is published. During reset, the new generation may be displayed explicitly as partial while the old one is stale; the authoritative replacement membership swaps only when the entire replacement snapshot is accepted. Late work and replay from the retired generation cannot change it.

Apply each group's committed observation array in order. output sets that source's membership present and retains the newest context; withdraw sets it absent. The same outpoint may therefore enter, leave and re-enter. Stable identity deduplication and the generation/sequence fence make replay of an older output or withdrawal a no-op, never a reversal. Record the last applicable sequence and observation identity per membership, including absent rows until their resumable generation is retired. Snapshot groups sharing W use the retained session page/group/array order rather than comparing W alone. Finite refresh adapters use serial local refresh generations and make no live replay claim.

A source's pending group is distinct from the cross-source dependency/conflict component barrier. A complete but unfinished earlier competitor prevents publishing a later competing selection; an incomplete earlier candidate is unresolved and has no first-eligible priority yet. When a group becomes acceptable, publish its membership and every affected graph/currentness change coherently under BRC-192's accepted revision. Rebuilding a projection from observations sorted for presentation, without the retained order and reconciled graph, is nonconforming. No subscription callback, replacement, replay or reorg may itself initiate a wallet action.

Required client vectors include reversed worker completion and restart, successor evidence preceding its parent from another provider, accepted-chain override followed by reorg, and output/withdraw/re-entry with stale replay and a retired generation. The common reconciliation corpus complements the snapshot/log and crash traces; provider qualification still requires those paths through the actual authenticated service and durable client store.