# Blackbox

{% tabs %}
{% tab title="End users" %}
Blackbox is “stop retrying this item for a while”.

It quarantines items that keep failing writes.

Use it to reduce:

* repeated apply failures
* API churn and rate-limit pressure
* noisy logs and long runs

#### What to do if something is stuck

* Wait for cooldown to expire, or
* remove the key from the blackbox file

Related:

* Similar mechanism: [Unresolved](/blueprint-architecture/orchestrator/unresolved.md)
* Watchlist-specific add issues: [Phantom Guard](/blueprint-architecture/orchestrator/phantom-guard.md)
  {% endtab %}

{% tab title="Power users" %}
Blackbox is the orchestrator’s “stop touching this item for a while” mechanism.

***

### Overview

#### What blackbox blocks today

Current wiring is intentionally narrow:

* Blackbox is applied as a **blocklist for planned ADDs**.
* It applies for features **except** `watchlist`.
* It is applied via `apply_blocklist(...)`.

Notably:

* `watchlist` is excluded from `apply_blocklist` (`if feature != "watchlist": ...`).
  * Watchlist “ghost adds” are handled mainly by **PhantomGuard** (`_phantoms.py`).
* Blackbox is **not applied to removals** in the current call sites.

So blackbox currently means:

* “don’t try adding this again” (non-watchlist features)

***

### Technical reference

#### Data model

Blackbox uses two files per destination + feature.

**Flap counters (`*.flap.json`)**

Tracks consecutive failures per key.

Path template (scoped):

* `/config/.cw_state/{dst}_{feature}.{SCOPE}.flap.json`

Shape:

```json
{
  "imdb:tt1234567": {
    "consecutive": 2,
    "last_reason": "apply:add:failed",
    "last_op": "add",
    "last_attempt_ts": 1738100000,
    "last_success_ts": 1738090000
  }
}
```

**Blackbox entries (`*.blackbox.json`)**

The “blocked” set, with a timestamp and reason.

Path template:

* If `pair` is provided: `/config/.cw_state/{dst}_{feature}.{PAIR}.blackbox.json`
* Otherwise (scoped): `/config/.cw_state/{dst}_{feature}.{SCOPE}.blackbox.json`

Shape:

```json
{
  "imdb:tt1234567": {
    "reason": "flapper:consecutive>=3",
    "since": 1738100000
  }
}
```

**Legacy migration**

Both `*.flap.json` and `*.blackbox.json` support a one-time copy from old unscoped filenames:

* `/config/.cw_state/{dst}_{feature}.flap.json`
* `/config/.cw_state/{dst}_{feature}.blackbox.json`

If the scoped file doesn’t exist yet but the legacy one does, it copies it forward.

***

#### Keys (what is actually blocked)

Blackbox stores string keys.

In normal operation, the orchestrator uses canonical keys:

* `imdb:tt...`
* `tmdb:123`
* `tvdb:456`
* `simkl:...`

The blocklist match is broader. A stored “key” can also be:

* an ID token: `tmdb:123` / `imdb:tt...` (case-insensitive)
* a title token: `movie|title:the thing|year:1982`

Why: the shared filter (`filter_with(...)` in `_tombstones.py`) checks:

1. canonical key
2. any `ids.*` tokens
3. a `type|title|year` token

So a manually added token can still match.

***

#### Promotion (how an item gets blackboxed)

Promotion is counter-based:

1. A failed write attempt increments the flap counter:
   * `inc_flap(dst, feature, key, reason=..., op=...)`
2. If `consecutive >= promote_after`, it is promoted:
   * `maybe_promote_to_blackbox(...)`
   * `_promote(...)` writes the `*.blackbox.json` entry

Promotion reason stored:

* `flapper:consecutive>=N`

**Important nuance: “ambiguous partial” protection**

Blackbox updates are skipped when the orchestrator can’t map outcomes to exact keys.

Example: a provider reports `count=5` but no `confirmed_keys`.

In that case:

* `_pairs_oneway` / `_pairs_twoway` mark it as `ambiguous_partial`
* blackbox counters are not updated

This avoids poisoning counters.

***

#### How the orchestrator uses blackbox

**Loading keys**

`load_blackbox_keys(dst, feature, pair=...)` returns a union of:

* keys in the scoped file, plus
* keys in the pair file (if `pair` is passed)

In `_pairs_oneway` / `_pairs_twoway`, the orchestrator passes:

* `pair_key = "-".join(sorted([src, dst]))` (example: `"PLEX-SIMKL"`)

That means entries are shared across all pairs using the same provider combo.

**Applying the blocklist**

For non-watchlist features, planned adds are filtered:

* `adds = apply_blocklist(state_store, adds, dst=dst, feature=feature, pair_key=pair_key, ...)`

`apply_blocklist` merges:

* tombstones (pair)
* unresolved keys
* blackbox keys

Then it removes matching items from the add list.

It also emits counts:

* `blocked.counts` (per-source breakdown)

***

#### Success and reset behavior

`record_success(dst, feature, keys)` resets flap counters for successful keys:

* `consecutive = 0`
* `last_reason = "ok"`
* `last_success_ts = now`

Important:

* A success does **not** remove a blackbox entry.
* Once a key is blackboxed, it stays blocked until pruned or manually removed.

***

#### Pruning (cooldown decay)

Blackbox entries decay via `prune_blackbox(cooldown_days=...)`:

* scans `/config/.cw_state/*.blackbox.json`
* removes any entry where `now - since > cooldown_days * 86400`

It is invoked once per run from `_pairs.py` via `_bb_prune_once(cfg)` using:

* `sync.blackbox.cooldown_days` (default typically 30)

Flap counter files are not pruned. They just reset on success.

***

### Configuration

Config path:

* `config["sync"]["blackbox"]`

Keys the code uses:

* `promote_after` (int): consecutive failures before promotion
* `pair_scoped` (bool): store entries in a pair file vs scoped-only
* `cooldown_days` (int): pruning window

Also present in config defaults:

* `enabled`
* `unresolved_days`
* `block_adds`
* `block_removes`

#### Reality check (current wiring)

As of the current code:

* `_blackbox.py` does not enforce `enabled` (it still records/promotes).
* `block_adds` / `block_removes` are not consulted by `apply_blocklist`.
* `unresolved_days` exists in `maybe_promote_to_blackbox(...)`, but the orchestrator does not pass `unresolved_map`.
  * That promotion path is effectively unused.

Treat those as “future knobs” unless you wire them.

***

### Operations (playbook)

#### See what’s blocked

In the container:

* `ls -lah /config/.cw_state/*blackbox.json`
* open the relevant file:
  * `{dst}_{feature}.{PAIR}.blackbox.json` (PAIR like `PLEX-SIMKL`)
  * or `{dst}_{feature}.{SCOPE}.blackbox.json` (scope like `one-way:PLEX-SIMKL:0`)

#### Unblock one item immediately

Edit the relevant `*.blackbox.json` file. Remove the key entry. Save the file.

#### Reset blackbox for a feature

Delete:

* `/config/.cw_state/{dst}_{feature}.*.blackbox.json`

Optional:

* `/config/.cw_state/{dst}_{feature}.*.flap.json`

#### Let it decay naturally

Wait for `cooldown_days` to elapse. Pruning runs once per orchestrator run.

***

{% endtab %}
{% endtabs %}


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://wiki.crosswatch.app/blueprint-architecture/orchestrator/blackbox.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.