Health

Provider health checks and how they gate pairs, features, and safe write behavior.

Health checks decide what CrossWatch is allowed to do.

If health is bad, CrossWatch skips risky work.

What you’ll see

auth_failed: the pair is skipped.
down: writes are skipped to that provider.
feature unsupported: a feature becomes a no-op.

If your run is skipping everything

Re-auth the provider.
Check server URL and tokens (media servers).
Retry the run after fixing auth.

Safety model: Guardrails
Provider expectations: Provider contract

This doc describes how provider health is gathered and how it gates orchestrator behavior.

Core code: cw_platform/orchestrator/_pairs.py (_collect_health_for_run) Provider contract: InventoryOps.health(cfg, emit=...)

Why health exists

Health is the orchestrator’s “don’t do something dumb” preflight.

It prevents:

running sync with invalid auth (guaranteed failure + noise)
treating a down provider as “everything removed”
making delete propagation decisions on missing data

Health is best-effort:

if a provider doesn’t implement health(), the orchestrator assumes it’s OK (less safe).

When health is collected

At the start of _run_pairs(ctx):

The orchestrator determines which providers are used by enabled pairs.
For each provider, it calls ops.health(cfg, emit=ctx.emit) (or ops.health(cfg) fallback).
It stores the result in ctx.health_map.

This happens once per run, not per feature.

Scope during health

During health, the orchestrator sets a temporary scope:

CW_PAIR_MODE=health
CW_PAIR_SCOPE=health
CW_PAIR_FEATURE=health

This ensures any provider-level state files created during health don’t collide with real pair scopes.

Expected health response shape

There is no strict schema, but the orchestrator expects these fields if present:

ok (bool)
status (string):
- ok
- down
- auth_failed
- degraded (optional)
features (dict[str,bool]) (optional)
api (dict) (optional)
latency_ms (int) (optional)

Example:

{
  "ok": true,
  "status": "ok",
  "features": { "watchlist": true, "ratings": true, "history": true, "playlists": false },
  "latency_ms": 120,
  "api": { "calls": 6, "errors": 0 }
}

If fields are missing, the orchestrator falls back conservatively.

How health gates runs

Auth failed

If either side of a pair has:

status == "auth_failed"

Then:

the pair is skipped entirely
features do not run
emits: pair:skip reason=auth_failed

Down

If a provider is:

status == "down"

Then behavior depends on mode:

One-way

If source is down:
- feature run is skipped (no meaningful plan)
If destination is down:
- plan is computed, but writes are skipped and items are recorded unresolved

Two-way

Writes to down side are skipped (unresolved)
Observed deletions are disabled for both sides
- because missing snapshots can look like deletes

Feature support

If health contains features[feature] == False:

feature run becomes a no-op and emits feature:unsupported

This is in addition to provider-declared ops.features().

Emitting health events

After calling health, the orchestrator emits a structured event:

health with provider name + status + basic details

This lets the UI show:

“SIMKL auth failed”
“TRAKT rate limited”
“PLEX ok”

Providers can also emit their own sub-events during health, especially:

api:hit samples (requests made during health)

Practical tips for provider authors

A good health implementation should:

verify credentials (fast, one request if possible)
detect rate-limits separately from auth errors
return per-feature capability flags if certain features are unavailable
emit api:hit samples for every network request

Also: don’t do heavy snapshot calls in health. That belongs in build_index.

Troubleshooting patterns

“Everything skipped with auth_failed”

provider token invalid or expired
connected app revoked
wrong base URL (for Jellyfin/Emby)
wrong protocol switch (HTTP/HTTPS) causing cookie invalidation in UI, etc.

“Two-way suddenly wants to delete everything”

health didn’t mark provider as down, but snapshot was empty
drop guard should catch it if checkpoints exist
if checkpoints missing, implement provider activities() and return an updated timestamp

“Feature says unsupported but provider supports it”

health features map is returning false accidentally
or ops.features() says false

Provider health contract: Provider contract
Drop guard and delete suppression: Guardrails
Observed deletions gating: Two-way sync
Health event format: Eventing

PreviousBlocklists NextChunking

Last updated 4 months ago

Was this helpful?