Eventing
Structured events and log conventions used by the orchestrator and UI during runs.
Events are what you see in the run log and UI stream.
They explain why CrossWatch did nothing or why it did something scary.
Events worth recognizing
pair:skip→ auth failed or hard gatewrites:skipped→ provider down, so no writessnapshot:suspect→ drop guard treated a snapshot as unsafemass_delete:blocked→ delete wave blockedblocked.counts→ items were blocked by tombstones/blackbox/unresolved
Quick workflow
Scan for
pair:skipandauth_failed.Check for
snapshot:suspectbefore trusting delete plans.If adds repeat, look for
blocked.countsand unresolved.
Related:
Safety model: Guardrails
Provider down/auth behavior: Health
This doc explains how the orchestrator reports progress: events, debug lines, API hit samples, and the conventions the UI expects.
Implementation notes
Core code: cw_platform/orchestrator/_emit.py (Emitter) + usage across _pairs*.py
Also relevant: provider health(..., emit=...) and any provider that emits api:hit.
Two output channels
The orchestrator produces two kinds of output:
Structured events (
Emitter.emit)Human log lines (
Emitter.info/Emitter.warn/Emitter.err)
Both typically go to stdout/stderr, but structured events are JSON-like and machine-consumable.
Emitter: what it is
An Emitter instance is created per run and passed around via ctx.emit and ctx.dbg.
It provides:
emit(event_name, **data)→ structured eventinfo(msg, **data)→ log line + optional structured sidecarwarn(msg, **data)err(msg, **data)dbg(event_name, **data)→ debug-only structured event (when runtime debug enabled)
In practice, most modules use:
emitfor lifecycle + countsdbgfor internal reasoning (suspect snapshot, blocklist composition, alias matches)
Event naming conventions
Event names are string tokens, generally grouped by subsystem:
Run lifecycle
run:startrun:pairrun:done
Health
health(per provider)
Feature lifecycle
feature:startfeature:unsupportedfeature:done
Planning
one:plantwo:plan
Apply
apply:add:startapply:add:progressapply:add:doneapply:remove:startapply:remove:progressapply:remove:doneapply:unresolved
Guardrails / safety
snapshot:suspect/snapshot.guardmass_delete:blockedblocked.countswrites:skipped
Metrics
api:hitapi:totalshttp:overviewstats:overview
These are not enforced by a schema, but the UI expects many of them.
Standard fields (what you’ll usually see)
Most events include some subset of:
pair/pair_keysrc,dstfeaturemode(one-way/two-way)counts:
src_count,dst_countadds,removesattempted,confirmed,unresolved,errors,skipped
timings:
ms(milliseconds)started_at,ended_at(epoch)
Pair context is usually implicit via scope env vars, but many events still include it explicitly for UI clarity.
Debug gating
Debug events are controlled by runtime flags in config, typically under:
config["runtime"]["debug"]config["runtime"]["suspect_debug"]
When debug is off:
dbg(...)emits nothing.
When debug is on:
you’ll see internal events such as:
alias token matches
blocklist breakdown per source
suspect snapshot reasons and checkpoints
API hit samples (api:hit)
api:hit)This is a “soft metric” system: providers can report API calls without the orchestrator needing to know provider internals.
A provider can call:
emit("api:hit", provider="SIMKL", op="GET /sync/activities", ms=123, ok=True, status=200)
The orchestrator aggregates these into:
totals per provider
totals per op group (sometimes)
overall http overview
These totals are emitted at the end:
api:totalshttp:overview
And may be written into state.json metrics as well.
If you want useful performance troubleshooting, emit api:hit aggressively in providers.
Count previews (plan/apply)
For large runs, the orchestrator often emits:
total counts
small “preview” lists of first N items (titles/keys)
This is intentionally limited to avoid:
huge logs
UI stream lag
Typical preview size:
10–30 items
If you need full details, use debug mode or export via state files.
Warnings and errors
Emitter.warn/err produce human-readable lines, but the orchestrator also tries to emit structured events for error contexts.
Common error patterns:
provider exceptions in build_index/add/remove
auth failures
unexpected provider response shapes (missing counts)
mass delete blocks
When errors happen, you’ll usually see:
ok:falsein the feature summaryerrors > 0plus unresolved entries recorded if applicable
UI integration expectations (practical)
The UI generally expects:
A run start event quickly (to show spinner)
Feature start / plan / apply / done sequences
Periodic progress during large apply batches
A final run done event with totals
If you add new event names, keep them:
short
lower-case with
:separatorsgrouped by subsystem (e.g.,
cache:*,provider:*)
Tips for provider authors
If you’re implementing provider ops:
Emit
api:hiton every external request (include op + ms + ok).Emit
healthdetails that include per-feature flags.When returning unresolved items, include IDs so unresolved can be tracked.
If you do these three, debugging becomes 10x easier.
Related pages
High-level orchestrator flow: Orchestrator
Apply events and result normalization: Applier
Snapshot debug events: Snapshots
Safety events: Guardrails
Last updated