CW Orchestrator

The Orchestrator is the brain of the operation.

The Orchestrator reads your settings, asks each provider what you have, computes what needs to change, and applies those changes safely. The orchestrator is provider-neutral: it coordinates sync logic without hard-coding behavior for any specific service.

Responsibilities

• Load and validate configuration (pairs, features, runtime flags).

• Build sync pairs with feature scopes (watchlist, history, ratings)

• Query provider modules/adapters for present snapshots (what exists right now).

• Compute a plan (adds/removes per side) by diffing normalized items.

• Apply the plan in chunks with retries and safety checks.

• Emit insights and logs (changes, failures, durations, API hits).

Key concepts

• Pair - A source and a target (one-way or two-way).

• Feature - Watchlist, history, ratings, playlists.

• Present snapshot - The provider’s current state used for planning.

• Plan - The adds/removes needed to reconcile A and B.

• Shadow / Tombstone - Local notes to avoid duplicates and remember intentional removals.

• Chunking - Batch writes to avoid timeouts and rate limits.

How a run works

1. Initialize - Read settings; apply runtime flags (debug, batching, cache).

2. Health checks - Validate provider connectivity/auth and feature availability.

3. Index - For each enabled feature, fetch current snapshots from providers.

4. Plan - Build deltas: what to add/remove on each side; resolve conflicts in two-way mode.

5. Apply - Write changes in chunks; update shadows/tombstones for stability.

6. Summarize - Emit a final summary (+X / -Y per feature, plus unresolved/blocked counts).

Modes

• One-way (A to B) - A is the source of truth; B follows.

• Two-way (A ↔ B) - Both sides contribute; source wins conflicts.

Runtime settings

These live under runtime in config.json.

"runtime": {
  "debug": true,
  "debug_http": false,
  "debug_mods": true,
  "state_dir": "",
  "telemetry": { "enabled": true },
  "snapshot_ttl_sec": 300,
  "apply_chunk_size": 100,
  "apply_chunk_pause_ms": 50,
  "apply_chunk_size_by_provider": { "SIMKL": 500 },
  "suspect_min_prev": 20,
  "suspect_shrink_ratio": 0.1
}

What each setting does

• debug: Master debug switch for extra diagnostics in logs.

• debug_http: Logs HTTP details (endpoints/statuses). Keep this off unless you’re troubleshooting.

• debug_mods: More module/provider debug output (snapshots, planning, decisions).

• state_dir: Where CrossWatch stores state (shadows, tombstones, caches).

  • Empty ("") means “use the default app state directory”.

• telemetry.enabled: Enables anonymous telemetry (not used yet)

Snapshot caching

• snapshot_ttl_sec: Cache TTL (seconds) for “present snapshot” reads inside the orchestrator. Lower = fresher reads; higher = fewer API calls.

Apply chunking (write batching)

• apply_chunk_size: Default write batch size (items per apply chunk).

• apply_chunk_pause_ms: Small pause between chunks to avoid rate limits and keep providers happy.

• apply_chunk_size_by_provider: Per-provider override for chunk size. Example: SIMKL supports larger payloads, so you can do:

  • SIMKL: 500 while keeping the global default at 100.

  You can add more providers the same way, but its untested. Normally the global default of 100 is safe.

  "apply_chunk_size_by_provider": {
    "SIMKL": 500,
    "TRAKT": 200,
    "MDBLIST": 500
  }

“Suspect” detection (safety net)

These are guardrails for weird runs where a provider suddenly returns far fewer items than usual (API hiccup, auth glitch, etc.).

• suspect_min_prev: Only run “suspect” checks if the previous snapshot had at least this many items.

• suspect_shrink_ratio: If the new snapshot is smaller than prev * ratio, treat it as suspicious and avoid doing destructive actions.

Performance & reliability

• Chunking avoids rate limits/timeouts and reduces “giant request” failures.

• Snapshots are cached briefly to speed up re-runs and reduce API pressure.

• Retries/backoff handle transient errors.

• Shadows/tombstones stabilize providers that lag or lack external IDs.

What you’ll see during a run

• Clear start/finish markers for each feature/pair.

• Planned vs applied counts.

• Warnings for items that couldn’t be matched or written (with reasons).

• A simple end summary.