From b10c2744d6c1f180dd4b417d35f3c28dc9ba86fc Mon Sep 17 00:00:00 2001 From: Martin Vogel Date: Sat, 29 Aug 2026 08:34:17 +0200 Subject: [PATCH] docs: define runtime trace observation model Signed-off-by: Martin Vogel --- docs/RUNTIME_TRACE_MODEL.md | 541 ++++++++++++++++++++++++++++++++++++ 1 file changed, 541 insertions(+) create mode 100644 docs/RUNTIME_TRACE_MODEL.md diff --git a/docs/RUNTIME_TRACE_MODEL.md b/docs/RUNTIME_TRACE_MODEL.md new file mode 100644 index 000000000..5bc3cb6bd --- /dev/null +++ b/docs/RUNTIME_TRACE_MODEL.md @@ -0,0 +1,541 @@ +# Runtime Trace Observation Model + +## Status and recommendation + +This document is an architecture contract, not an implementation. It recommends a +durable, versioned runtime-observation sidecar whose publications can be read alongside +a pinned static index generation. Runtime observations MUST NOT patch static `CALLS` +edges, fabricate static nodes, or appear as persistent edges in the default graph. + +The model intentionally separates facts observed at runtime from facts derived from +source. A caller may explicitly request a `RUNTIME_CALL` overlay; existing static +search, architecture, and default trace behavior remain unchanged. + +This document-only change does not alter the runtime wire version, runtime schema +version, runtime semantic version, runtime artifact version, static semantic index +version, or artifact schema version. In particular, the static semantic index version +remains 3. + +## Scope + +The model defines: + +- stable identities for spans, producer contributions, observations, endpoints, and + composite publications; +- deterministic ingestion, aggregation, publication, and replay semantics; +- generation-scoped resolution from runtime endpoints to static symbols; +- an opt-in runtime call overlay without changing the stored static graph; +- concurrency, retry, conflict, recovery, privacy, and cardinality boundaries; and +- six bounded implementation packages that can be reviewed and shipped separately. + +## Non-goals + +This design does not: + +- infer new static `CALLS` edges or change confidence on existing ones; +- create placeholder nodes in the static graph for runtime-only identities; +- persist runtime edges in the generic graph edge table; +- make runtime data visible to existing graph tools unless an overlay is requested; +- define distributed tracing transport, sampling policy, or collector deployment; +- promise complete traces, causal proof from timing alone, or source-line attribution; +- store arbitrary span payloads, headers, request bodies, query strings, or secrets; +- average percentile summaries or treat a producer's p99 as a mergeable percentile; +- provide a compatibility promise for a future mutation API before its versioned + contract exists; or +- change any runtime or static version merely by adding this document. + +## Current surface + +The current `ingest_traces` surface requires `project` and `traces[]`. Each trace item +permits `caller`, `callee`, and `count`, but those item fields are not currently +individually required. Its handler counts the supplied records and returns +`status="accepted"`, `traces_received`, and an unimplemented note; this is not an +accepted durable mutation, and the handler does not open or mutate a store. + +The existing trace helper surface can extract `service.name`, HTTP method, HTTP path, +HTTP status, span kind, duration, URL path, and p99. Verified span-kind values are: + +- `1`: internal; +- `2`: server; and +- `3`: client. + +Generic graph edges are unique by source, target, type, and the local-name +discriminator. Generic edge upsert merges properties with `json_patch`. Those rules are +appropriate for the static graph but are not the contribution or aggregation rules in +this document. The store already exposes `BEGIN IMMEDIATE`, `COMMIT`, and `ROLLBACK`. +Static publication already uses opaque generations and request-scoped reads. + +## Normative terms + +`MUST`, `MUST NOT`, `SHOULD`, and `MAY` are normative. "Canonical JSON" means UTF-8 +JSON with a fixed schema, lexicographically sorted object keys, arrays in their +specified order, no insignificant whitespace, and integers rather than locale- or +platform-dependent numeric strings. Hashes are computed over those canonical bytes. + +## Identity layers + +The following identities are different and MUST NOT be substituted for one another. + +### Trace and span identity + +A span is identified within one producer namespace by: + +```text +(project, producer_id, producer_epoch, trace_id, span_id) +``` + +`trace_id` and `span_id` are opaque byte strings rendered in one canonical lowercase +hex form at the boundary. `producer_epoch` is part of span identity so a producer can +reuse trace or span identifiers in a later epoch without collision. The tuple identifies +an input span for diagnostics and deduplication; it does not identify a static node or +an aggregate. + +### Producer contribution identity + +Every accepted batch has a producer-stable identifier: + +```text +contribution_id = H(canonical JSON { + producer_id, + producer_epoch, + source_batch_id +}) +``` + +`producer_id` names the authenticated logical producer, not a transient process. +`producer_epoch` changes only when that producer intentionally starts a new identity +domain. `source_batch_id` is stable across delivery retries. The resulting +`contribution_id` is scoped by project. + +An accepted contribution is immutable. Retrying the same `(project, producer_id, +contribution_id)` with the same canonical payload hash is an idempotent success and +returns the original disposition. Retrying it with different canonical bytes is a +conflict and MUST be rejected without mutation. A producer MUST NOT reuse another +producer's namespace, and the service MUST NOT silently mint a replacement ID to turn +a conflict into a second contribution. + +The payload hash covers the canonical normalized allowlisted contribution map together +with its pinned wire and runtime semantic version fields. It does not cover original +envelope ordering, unsupported fields, discarded fields, or raw payload bytes. + +### Endpoint identity + +Every endpoint has an explicit tag and a canonical key. + +1. `symbol`: a source-derived symbol reference `(project, qualified_name)`. This stable + endpoint identity deliberately excludes a static generation. A separate resolution + record maps it within one named static generation and resolver version. A later + generation MUST resolve again and MUST NOT inherit a node identifier from an older + generation. +2. `runtime`: a runtime-only endpoint derived from an allowlisted protocol identity, + such as canonical service name plus normalized HTTP route and method. It is stored + only in the sidecar and never creates a static node. +3. `unknown`: an opaque, scoped endpoint used when the allowlisted evidence cannot + identify either a symbol or a runtime service endpoint. Unknown keys MUST be stable + only inside their documented scope (for example, producer plus trace) and MUST NOT + collapse unrelated missing values into one global endpoint. + +Endpoint canonicalization is versioned by `runtime_semantic_version`. All strings are +Unicode-normalized, methods and protocol tokens use their specified canonical case, +and absent values are distinct from empty values. Raw URL queries, fragments, and +credentials are excluded. + +### Observation identity + +An observation describes one directed runtime perspective, not one delivery. Its +canonical key is: + +```text +observation_key = H(canonical JSON { + runtime_semantic_version, + project, + protocol, + perspective, + source_endpoint_key, + target_endpoint_key, + operation_key +}) +``` + +The key excludes counters, timestamps, duration summaries, status summaries, +contribution IDs, and static database row IDs. `operation_key` contains only the +versioned allowlisted operation identity, such as canonical HTTP method and normalized +route template. Consequently, identical observations from multiple producers converge +while semantically different directions or operations remain distinct. + +### Composite publication identity + +A reader-visible snapshot is identified by the atomic pair: + +```text +{ static_generation, runtime_generation } +``` + +The pair is the head identity, but its halves are independent immutable publications. +A runtime generation identifies runtime contributions and aggregates without embedding +a static generation. Generation-scoped symbol resolutions and overlay projections are +keyed by the pair plus resolver version. Neither half may be silently advanced while a +request is using the pair. + +## Protocol and perspective mapping + +Ingestion parses and normalizes only versioned, allowlisted span fields into the +contribution, then derives a perspective from that evidence. The raw span payload is +not durably preserved. The HTTP mapper uses the existing service-name, HTTP method, +HTTP path or URL path, status, span-kind, and duration helpers. The route field is +normalized as a route template when the source provides one; a raw high-cardinality URL +path MUST NOT be promoted to a template by guesswork. + +For the verified span kinds: + +- `client` (`3`) represents an outbound attempt. The source is the resolved local + symbol when available, otherwise a tagged local runtime or scoped unknown endpoint. + The target is the allowlisted remote service/HTTP endpoint or a scoped unknown + endpoint. The perspective token is `outbound`. +- `server` (`2`) represents an inbound handling attempt. The source is an allowlisted + peer/runtime endpoint or a scoped unknown endpoint. The target is the resolved local + handler when available, otherwise the local service/HTTP runtime endpoint. The + perspective token is `inbound`. +- `internal` (`1`) represents local work. A directed call projection is produced only + when explicit parent/child evidence supplies two endpoints. Otherwise the + observation remains queryable as internal activity but does not become a + `RUNTIME_CALL`. The perspective token is `internal`. + +Unsupported or absent span kinds are retained only as normalized, allowlisted non-call +observation metadata if policy allows; unsupported fields and raw payload bytes are +discarded. They MUST NOT be guessed into one of the three perspectives. Status and +duration contribute to summaries but never to observation identity. + +Symbol resolution is best-effort and generation-scoped. Ambiguous matches remain +tagged runtime or unknown endpoints. Resolution MUST fail closed: it must not choose a +symbol by filename substring, short-name coincidence, timing, or insertion order. + +## Sidecar model + +The durable sidecar is conceptually separate from the static graph store even if an +implementation initially shares the same database file. It contains these logical +relations: + +### `runtime_endpoints` + +Stores the endpoint tag, canonical endpoint key, and canonical allowlisted identity. +For a `symbol` endpoint, the durable identity contains project plus qualified name but +no static generation. Generation and static database row IDs belong only to resolution +records or caches, not durable endpoint identity. + +### `runtime_contributions` + +Stores project, producer ID, contribution ID, canonical payload hash, ingestion state, +and immutable canonical contribution metadata. States include at least `staged` and +`finalized`; an aborted transaction leaves neither state visible. + +### `runtime_observation_contributions` + +Maps `(contribution_id, observation_key)` to that contribution's canonical summary. +The map entry is immutable after finalization. Replacing it requires a new contribution +identity. + +### `runtime_aggregates` + +Stores the deterministic fold of finalized contribution-map entries for each +observation key and runtime generation. It is derived state and can be rebuilt from +finalized contributions. + +### `runtime_resolutions` + +Maps a stable runtime or unknown endpoint key to an optional stable SymbolEndpoint +`(project, qualified_name)` plus its current static node/cache result, unresolved +result, or ambiguity result. Each record is keyed by project, runtime generation, +static generation, stable endpoint key, and resolver version. Resolution records are +derived caches, not endpoint or observation identity, and a record from one static +generation MUST NOT be reused for another. + +### `runtime_publications` + +Stores immutable runtime generations independently of static generations. A separate +atomic per-project composite head pairs one current runtime generation with one current +static generation. Publication metadata includes the relevant runtime versions; the +composite head records the static index/artifact versions used for compatibility +checks. + +No table above is a static graph node or edge table. The implementation MUST NOT reuse +generic `json_patch` edge upsert for contribution merging. + +## Deterministic aggregation + +Before immutable summaries are stored, one contribution is normalized deterministically: + +1. canonicalize only allowlisted fields and compute the exact span identity + `(project, producer_id, producer_epoch, trace_id, span_id)`; +2. deduplicate records with the same span identity and identical normalized bytes; +3. reject the entire contribution if one span identity has different normalized bytes; +4. group the remaining unique normalized records by `observation_key`; +5. after keyed exact-span deduplication/set union supplies idempotence, fold integer + counts and duration sums, fixed status buckets, and fixed-schema histogram bins over + the unique normalized span map using associative and commutative exact addition; and +6. sort keys and canonical-serialize the contribution map before hashing or storage. + +The normalized contribution map and its immutable summaries MUST be byte-identical for +every input ordering and worker partitioning of the same unique normalized span set. +Discarded or unsupported fields cannot change the payload hash. + +For each observation, aggregation is a map union keyed by contribution identity: + +```text +aggregate(observation) = fold(sorted union { + contribution_id -> canonical contribution summary +}) +``` + +Union of identical key/value entries is idempotent. A differing value under the same +key is a conflict, not a winner selection. Map union is associative and commutative; +sorting by contribution ID before serialization makes the resulting bytes independent +of producer, retry, transaction, or worker order. + +Canonical contribution summaries MAY contain integer count, integer duration sum in a +fixed unit, fixed-schema status buckets, and a mergeable duration distribution whose +encoding is pinned by `runtime_semantic_version`. Aggregate totals are exact integer +sums over the contribution map. Derived display values are computed after the fold. + +Percentiles are not additive. A batch p99 MUST NOT be averaged, weighted-averaged, or +selected as the aggregate p99. The existing p99 helper may summarize a single raw batch +for diagnostics, but a published cross-contribution p99 requires a versioned mergeable +distribution (for example fixed histogram boundaries). Without such a distribution, +the aggregate p99 is unavailable. + +Canonical aggregate JSON MUST be byte-identical for every permutation and +parenthesization of the same finalized contribution set. + +## Ingestion and publication transaction + +Full wire parsing, canonicalization, payload hashing, and every validation that does +not depend on mutable project state happen before acquiring the publication lock. +Publication is then serialized by one shared per-project publication lock used by both +static and runtime publishers. An implementation MUST NOT introduce an independent +runtime lock that permits lost updates to either half of the composite head. + +Within that lock, a runtime publisher: + +1. begins a write transaction with `BEGIN IMMEDIATE`; +2. reads and pins the current composite head and validates compatibility versions; +3. performs state-dependent authorization, quota, producer identity, deduplication, + contribution conflict, and expected-head checks; +4. stages all endpoints, immutable contribution-map entries, and rebuilt aggregates; +5. allocates an opaque runtime generation independent of the pinned static generation; +6. durably finalizes every contribution in the publication; +7. writes the immutable runtime publication and atomically changes only the runtime + half of the composite head, preserving its pinned static generation; +8. commits all changes with `COMMIT`; and +9. releases the per-project publication lock. + +Every validation or write failure before commit uses `ROLLBACK`. There is no visible +partial contribution, aggregate, generation, or head. Finalization and head movement +are in the same transaction, so a success response is sent only after durable commit. + +Static publication follows the same lock discipline and atomically changes only the +static half of the composite head, preserving its pinned runtime generation. It does +not create a new runtime generation merely because source was reindexed. Resolution +records for the new `{static_generation, runtime_generation}` pair are recomputed or +materialized under their pair-and-resolver-version key; records from the old static +generation are never treated as resolutions for the new one. + +## Reader contract + +A request that opts into runtime data pins one composite head at request start. Every +page, endpoint resolution, aggregate, and overlay edge in that request is read from +that pair. Default requests pin only their existing static request scope and do not pay +for or expose the overlay. + +Pagination cursors encode the project, composite head, query shape, overlay mode, and +runtime semantic version. If the current head differs, a subsequent request with that +cursor returns a stale-cursor error; it never resumes against mixed generations. +Readers MAY explicitly request a still-retained historical composite head. Missing or +garbage-collected heads fail explicitly. + +## `RUNTIME_CALL` overlay + +Runtime call projection is an explicit read option. It presents directed overlay edges +derived from one pinned runtime generation. These edges: + +- use the distinct type `RUNTIME_CALL`; +- carry endpoint tags so callers can distinguish symbol, runtime, and unknown ends; +- include deterministic aggregate summaries and the composite head identity; +- never mutate, replace, increase confidence on, or suppress a static `CALLS` edge; +- are omitted from existing search, architecture, and default trace responses unless + the caller requests the overlay; and +- are regenerated when either half of the composite publication changes. + +If a projected runtime call has symbol endpoints, it references their qualified-name +identity within the pinned static generation. Runtime-only and unknown endpoints remain +sidecar objects and are not fabricated as graph nodes. + +## Version boundaries + +Compatibility is divided deliberately: + +- `wire_version`: request and response envelope, field encodings, and retry result; +- `runtime_schema_version`: sidecar tables, indexes, and migration compatibility; +- `runtime_semantic_version`: canonical endpoint/observation keys, mappings, and + aggregation meaning; +- `runtime_artifact_version`: exported/imported sidecar artifact layout; +- `static_semantic_index_version`: source-derived graph semantics; and +- `static_artifact_schema_version`: static artifact representation. + +A change bumps only the boundary it changes. A runtime semantic change can invalidate +runtime generations without changing static semantic index version. A storage-only +migration can change runtime schema version without changing observation identity. +Import rejects incompatible artifact versions before mutation. This document changes +none of these versions. + +## Security, privacy, and cardinality + +Ingestion is project-authorized and producer-authenticated. Producer IDs, contribution +IDs, and project scope are server-validated; caller-provided project or producer fields +do not grant access. Reads apply the same project authorization to runtime and static +data. + +Only allowlisted attributes enter canonical identities or summaries. Credentials, +authorization data, cookies, request/response bodies, query strings, fragments, +arbitrary baggage, and raw exception text are rejected or discarded before durable +storage. Logs and errors use opaque IDs and counts, not rejected payloads. + +The service enforces bounded request bytes, records, endpoints, observations, attribute +lengths, route-template segments, status buckets, and contribution-map growth. Raw URL +paths are not accepted as unbounded route labels. Unknown endpoints are scoped to avoid +both accidental global collapse and attacker-controlled global cardinality. Quotas and +retention apply per project and producer. Limit failure is all-or-nothing and does not +publish a truncated contribution unless a future wire version explicitly defines that +behavior. + +## Failure and recovery matrix + +| Condition | Required disposition | Visible state | +| --- | --- | --- | +| Same contribution ID, same payload hash | Idempotent success; return prior result | Unchanged | +| Same contribution ID, different payload hash | Conflict; reject entire request | Unchanged | +| Unauthorized project or producer | Reject before staging | Unchanged | +| Invalid/unsupported wire or semantic version | Reject before staging | Unchanged | +| Invalid endpoint, attribute, or cardinality limit | Reject entire contribution | Unchanged | +| Ambiguous symbol resolution | Keep runtime/unknown tag; do not guess | Valid sidecar observation only | +| Client-supplied expected composite head differs from the head read under lock | Reject as stale; retry from the newly read head | Unchanged | +| Failure while staging or aggregating | `ROLLBACK` | No partial state | +| Crash before commit | Database recovery rolls back the transaction | Prior head remains | +| Commit succeeds but response is lost | Producer retry resolves idempotently | One finalized contribution | +| Derived aggregate missing or corrupt | Rebuild from finalized contribution map before publish | Prior valid head remains | +| Cursor references a different or removed head | Return stale/missing cursor error | No mixed-generation read | +| Unsupported span kind | Retain only normalized, allowlisted non-call observation metadata; discard unsupported fields and raw payload bytes | No guessed call overlay | +| No mergeable duration distribution | Publish counts/sums; report p99 unavailable | No averaged p99 | + +## Alternatives considered + +### Patch static `CALLS` edges + +Benefit: existing graph queries would see runtime evidence with no overlay option. + +Trade-offs: it conflates observed and source-derived semantics, lets sampled or hostile +telemetry alter the static graph, cannot represent runtime-only endpoints honestly, +couples runtime retention to static indexing, and makes rollback and provenance +ambiguous. It is rejected. + +### Fabricate static nodes for runtime services and unknown peers + +Benefit: all endpoints would fit the existing graph node/edge shape. + +Trade-offs: fabricated nodes look source-derived, create unstable identities and high +cardinality, collide with future symbol resolution, and pollute architecture/search +results. It is rejected. + +### Store persistent runtime edges in the default graph + +Benefit: generic edge uniqueness and `json_patch` upsert are already available. + +Trade-offs: generic uniqueness has no immutable producer-contribution dimension; +`json_patch` is neither the required commutative map union nor a conflict detector; +default readers would unexpectedly mix generations and data classes. It is rejected. + +### Keep runtime observations entirely ephemeral + +Benefit: minimal schema, retention, and migration burden; no durable sensitive data. + +Trade-offs: retries cannot be made idempotent across restarts, results are not +reproducible or pageable, multi-producer aggregation is unstable, and a reader cannot +pin a coherent composite publication. It is rejected as the primary model, though +ephemeral pre-validation remains useful. + +### Durable versioned sidecar with an opt-in overlay (selected) + +Benefits: preserves static semantics, gives runtime-only endpoints an honest identity, +supports deterministic replay and aggregation, isolates retention and authorization, +and provides atomic cross-generation reads. + +Trade-offs: adds schema and version boundaries, a shared publication lock, endpoint +resolution, aggregate rebuilds, overlay-aware APIs, and explicit operational limits. +These costs are accepted because they make provenance, concurrency, failure recovery, +and compatibility testable rather than implicit. + +## Bounded implementation packages + +Implementation remains deferred. The following packages are intentionally ordered and +bounded; each requires its own reproduce-first tests and review. + +1. **Versioned wire validation and canonicalization.** Define authenticated producer + identity and epoch, contribution ID, exact span-identity deduplication, conflicting + duplicate rejection, canonical contribution maps and payload hashes, allowlisted + HTTP/span mapping, limits, and dry-run validation. No store mutation. +2. **Sidecar schema and immutable contributions.** Add runtime schema migrations, + endpoints, contribution records, observation-contribution maps, idempotent retry, + conflict rejection, and transactional rollback. No publication head or overlay. +3. **Deterministic aggregation.** Add canonical intra-contribution grouping, keyed + exact-span idempotent set union/deduplication, associative and commutative exact + integer/status/histogram addition, cross-contribution map-union folds, fixed + mergeable duration distributions, input/worker/merge-order permutation tests, + rebuild verification, and explicit unavailable p99 behavior. +4. **Generation-scoped resolution and publication.** Add the shared per-project lock, + independent runtime generation allocation, durable finalization, pair-and-version + resolution records, atomic composite head updates that preserve the other half, and + crash/retry recovery tests. +5. **Pinned read and `RUNTIME_CALL` overlay.** Add opt-in queries, tagged endpoints, + request-scoped composite pins, stale cursors, historical-head handling, and proofs + that default static tools remain byte-for-byte unchanged. +6. **Artifact, retention, and operational hardening.** Add versioned runtime artifact + import/export, quota/retention policy, authorization audits, corruption rebuild, + migration/rollback tests, metrics, and multi-producer concurrency tests. + +No package may silently absorb a later package's public surface or persistence model. + +## Acceptance invariants + +An implementation of this contract is acceptable only if all of these hold: + +1. Static nodes and `CALLS` edges are byte-identical with runtime ingestion enabled or + disabled for the same source input. +2. Existing search, architecture, and default trace responses exclude runtime data. +3. Runtime data appears only through an explicit, versioned overlay request. +4. Same producer contribution plus same payload is idempotent across retries and + restarts; a different payload under that identity is a no-mutation conflict. +5. Normalized contribution and aggregate bytes are identical for every input, delivery, + worker, and merge order of the same unique normalized span set, including identical + duplicate replay; a conflicting duplicate span identity rejects the whole + contribution without mutation. +6. Aggregate p99 is derived only from a pinned mergeable distribution and is otherwise + unavailable; producer p99 values are never averaged. +7. Symbol resolution is qualified-name based, generation-scoped, and fail-closed on + ambiguity; runtime and unknown endpoints never fabricate static nodes. +8. Runtime generation state contains all of its finalized contributions, endpoints, + and aggregates, and reader visibility requires the separate atomic composite-head + update; the transaction exposes both the complete generation and head update, or + neither. +9. Every overlay read and page uses one immutable + `{static_generation, runtime_generation}` pair; stale cursors fail explicitly. +10. Static and runtime publication serialize through the same per-project lock; each + publisher changes only its own half of the composite head and preserves the other, + and symbol-resolution records are never reused across static generations or + resolver versions. +11. Unsupported span kinds, missing identities, and limit violations never become + guessed calls or partial publications. +12. Authorization, privacy filtering, quotas, and retention are enforced before data + becomes durable or reader-visible. +13. Wire, runtime schema, runtime semantic, runtime artifact, static semantic, and + static artifact versions can change independently according to their boundaries. +14. Documentation-only adoption leaves all six version values and all runtime behavior + unchanged.