Blackbox

Quarantine noisy “flapping” items so runs stay safe, quiet, and cheap.

Blackbox is “stop retrying this item for a while”.

It quarantines items that keep failing writes.

Use it to reduce:

repeated apply failures
API churn and rate-limit pressure
noisy logs and long runs

What to do if something is stuck

Wait for cooldown to expire, or
remove the key from the blackbox file

Similar mechanism: Unresolved
Watchlist-specific add issues: Phantom Guard

Blackbox is the orchestrator’s “stop touching this item for a while” mechanism.

Overview

What blackbox blocks today

Current wiring is intentionally narrow:

Blackbox is applied as a blocklist for planned ADDs.
It applies for features except watchlist.
It is applied via apply_blocklist(...).

Notably:

watchlist is excluded from apply_blocklist (if feature != "watchlist": ...).
- Watchlist “ghost adds” are handled mainly by PhantomGuard (_phantoms.py).
Blackbox is not applied to removals in the current call sites.

So blackbox currently means:

“don’t try adding this again” (non-watchlist features)

Technical reference

Data model

Blackbox uses two files per destination + feature.

Flap counters (*.flap.json)

Tracks consecutive failures per key.

Path template (scoped):

/config/.cw_state/{dst}_{feature}.{SCOPE}.flap.json

Shape:

{
  "imdb:tt1234567": {
    "consecutive": 2,
    "last_reason": "apply:add:failed",
    "last_op": "add",
    "last_attempt_ts": 1738100000,
    "last_success_ts": 1738090000
  }
}

Blackbox entries (*.blackbox.json)

The “blocked” set, with a timestamp and reason.

Path template:

If pair is provided: /config/.cw_state/{dst}_{feature}.{PAIR}.blackbox.json
Otherwise (scoped): /config/.cw_state/{dst}_{feature}.{SCOPE}.blackbox.json

Shape:

{
  "imdb:tt1234567": {
    "reason": "flapper:consecutive>=3",
    "since": 1738100000
  }
}

Legacy migration

Both *.flap.json and *.blackbox.json support a one-time copy from old unscoped filenames:

/config/.cw_state/{dst}_{feature}.flap.json
/config/.cw_state/{dst}_{feature}.blackbox.json

If the scoped file doesn’t exist yet but the legacy one does, it copies it forward.

Keys (what is actually blocked)

Blackbox stores string keys.

In normal operation, the orchestrator uses canonical keys:

imdb:tt...
tmdb:123
tvdb:456
simkl:...

The blocklist match is broader. A stored “key” can also be:

an ID token: tmdb:123 / imdb:tt... (case-insensitive)
a title token: movie|title:the thing|year:1982

Why: the shared filter (filter_with(...) in _tombstones.py) checks:

canonical key
any ids.* tokens
a type|title|year token

So a manually added token can still match.

Promotion (how an item gets blackboxed)

Promotion is counter-based:

A failed write attempt increments the flap counter:
- inc_flap(dst, feature, key, reason=..., op=...)
If consecutive >= promote_after, it is promoted:
- maybe_promote_to_blackbox(...)
- _promote(...) writes the *.blackbox.json entry

Promotion reason stored:

flapper:consecutive>=N

Important nuance: “ambiguous partial” protection

Blackbox updates are skipped when the orchestrator can’t map outcomes to exact keys.

Example: a provider reports count=5 but no confirmed_keys.

In that case:

_pairs_oneway / _pairs_twoway mark it as ambiguous_partial
blackbox counters are not updated

This avoids poisoning counters.

How the orchestrator uses blackbox

Loading keys

load_blackbox_keys(dst, feature, pair=...) returns a union of:

keys in the scoped file, plus
keys in the pair file (if pair is passed)

In _pairs_oneway / _pairs_twoway, the orchestrator passes:

pair_key = "-".join(sorted([src, dst])) (example: "PLEX-SIMKL")

That means entries are shared across all pairs using the same provider combo.

Applying the blocklist

For non-watchlist features, planned adds are filtered:

adds = apply_blocklist(state_store, adds, dst=dst, feature=feature, pair_key=pair_key, ...)

apply_blocklist merges:

tombstones (pair)
unresolved keys
blackbox keys

Then it removes matching items from the add list.

It also emits counts:

blocked.counts (per-source breakdown)

Success and reset behavior

record_success(dst, feature, keys) resets flap counters for successful keys:

consecutive = 0
last_reason = "ok"
last_success_ts = now

Important:

A success does not remove a blackbox entry.
Once a key is blackboxed, it stays blocked until pruned or manually removed.

Pruning (cooldown decay)

Blackbox entries decay via prune_blackbox(cooldown_days=...):

scans /config/.cw_state/*.blackbox.json
removes any entry where now - since > cooldown_days * 86400

It is invoked once per run from _pairs.py via _bb_prune_once(cfg) using:

sync.blackbox.cooldown_days (default typically 30)

Flap counter files are not pruned. They just reset on success.

Configuration

Config path:

config["sync"]["blackbox"]

Keys the code uses:

promote_after (int): consecutive failures before promotion
pair_scoped (bool): store entries in a pair file vs scoped-only
cooldown_days (int): pruning window

Also present in config defaults:

enabled
unresolved_days
block_adds
block_removes

Reality check (current wiring)

As of the current code:

_blackbox.py does not enforce enabled (it still records/promotes).
block_adds / block_removes are not consulted by apply_blocklist.
unresolved_days exists in maybe_promote_to_blackbox(...), but the orchestrator does not pass unresolved_map.
- That promotion path is effectively unused.

Treat those as “future knobs” unless you wire them.

Operations (playbook)

See what’s blocked

In the container:

ls -lah /config/.cw_state/*blackbox.json
open the relevant file:
- {dst}_{feature}.{PAIR}.blackbox.json (PAIR like PLEX-SIMKL)
- or {dst}_{feature}.{SCOPE}.blackbox.json (scope like one-way:PLEX-SIMKL:0)

Unblock one item immediately

Edit the relevant *.blackbox.json file. Remove the key entry. Save the file.

Reset blackbox for a feature

Delete:

/config/.cw_state/{dst}_{feature}.*.blackbox.json

Optional:

/config/.cw_state/{dst}_{feature}.*.flap.json

Let it decay naturally

Wait for cooldown_days to elapse. Pruning runs once per orchestrator run.

PreviousSnapshots NextOne-way sync

Last updated 4 months ago

Was this helpful?