# Tombstones

{% tabs %}
{% tab title="End users" %}
Tombstones are deletion memory.

They stop “delete → re-add → delete → re-add” loops in two-way sync.

#### When tombstones matter

* You deleted an item on one side.
* Two-way would otherwise re-add it from the other side.

#### If something won’t re-add

It may still be tombstoned.

Wait for tombstone TTL to expire, or remove it manually.

Related:

* Two-way deletes: [Two-way sync](/blueprint-architecture/orchestrator/two-way-sync.md)
  {% endtab %}

{% tab title="Power users" %}
Tombstones are the orchestrator’s “deletion memory”. They exist mainly to make two-way sync safe by preventing delete/re-add ping-pong and by allowing deletion propagation only when there is strong evidence.

**Code:** `cw_platform/orchestrator/_tombstones.py`\
**Used by:** `orchestrator/_pairs_oneway.py`, `orchestrator/_pairs_twoway.py`, `orchestrator/_pairs_blocklist.py`

***

### What a tombstone is (in this codebase)

A tombstone is a record that a specific *token* was deleted in a given **feature + provider pair** context.

A token can be:

* a canonical key (e.g., `imdb:tt0111161`)
* an ID token from `item["ids"]` (e.g., `tmdb:278`, `tvdb:12345`, `simkl:...`)
* a typed token for seasons/episodes (if providers put those into `ids` or derived tokens)

Tombstones are stored in a single JSON map file:

* `/config/.cw_state/tombstones.json`

***

### Why tombstones exist

Without tombstones, two-way sync has a classic failure mode:

1. Item is removed on A.
2. Sync sees item still on B and re-adds it back to A.
3. User removes it again.
4. Repeat until someone throws the server out the window.

Tombstones fix this by:

* remembering that “this token was deleted”
* blocking re-adds for a configurable time window (TTL)
* enabling the orchestrator to propagate deletes intentionally (only when safe)

***

### Storage format

Keys in `tombstones.json` are strings:

```
{feature}:{PAIR}|{token}
```

Where:

* `feature`: `watchlist|ratings|history|playlists`
* `PAIR`: providers sorted, e.g. `PLEX-SIMKL`, `JELLYFIN-PLEX`
* `token`: canonical key or ID token

Example key:

* `watchlist:PLEX-SIMKL|imdb:tt1234567`

Value is a small dict:

```json
{
  "at": 1738100000,
  "why": "remove"
}
```

`why` is optional and mainly used for debugging:

* `"remove"` (explicit remove write succeeded)
* `"observed_delete"` (two-way observed deletions)

***

### Reading tombstones

#### `keys_for_feature(state_store, feature, pair) -> dict[str, dict]`

Loads `tombstones.json` and filters by:

* matching `feature`
* matching `pair` (exact string like `PLEX-SIMKL`)

Returns a map:

* `token -> metadata`

So the caller usually takes:

* `tomb = set(map.keys())`

***

### TTL (how long tombstones are “active”)

The orchestrator enforces TTL by filtering tomb entries:

* `sync.tombstone_ttl_days` (default commonly 30)

Rule:

* if `now - meta["at"] > ttl_days * 86400` → ignore it for blocking/planning

Important:

* tombstones may remain in the file even after TTL; they’re just treated as expired.
* some code paths may later prune old entries (or you can nuke them manually).

***

### Writing tombstones on successful removals

#### One-way

After confirmed destination removals:

* tombstones are written for the destination side under `feature:{PAIR}|token`.

Tokens written:

1. canonical key (`canonical_key(item)`)
2. every `ids.*` token in `item["ids"]`

This increases match reliability across providers even when canonical keys differ.

#### Two-way

Same, but removals can happen on either side. Two-way also writes tombstones for **observed deletions** (see below).

***

### Observed deletions (two-way) → tombstones

Two-way computes “observed deletions” as:

* `obsA = prevA.keys - A_cur.keys`
* `obsB = prevB.keys - B_cur.keys`

These are keys that existed in baseline but disappeared from the latest live snapshot.

When safe (not bootstrapping, not down, not suspect snapshot), two-way:

* writes tombstones for those observed-deleted keys
* also attempts to write ID tokens by looking up the missing item in `prevA/prevB`

This is the core mechanism that turns “it vanished on one side” into a durable “we consider it deleted”.

***

### How tombstones block re-adds (and where)

Tombstones are applied as a blocklist in two main places:

#### 1) Two-way planning

During planning, the orchestrator expands tomb tokens (`tombX`) to match canonical keys via alias maps. If an item matches tombstones strongly, it can:

* be prevented from re-adding, or
* be treated as a deletion that should propagate (if removals enabled)

#### 2) `apply_blocklist(...)` for adds (non-watchlist)

`orchestrator/_pairs_blocklist.py` uses tombstones as one component of its add blocklist:

* tombstones + unresolved + blackbox
* only wired for non-watchlist features in current pipelines

Note:

* watchlist does not go through `apply_blocklist` in current wiring; it relies on PhantomGuard + tombstone handling in two-way logic.

***

### Matching logic: token vs item

A tomb token matches an item if any of these match:

1. canonical key equals token
2. any `ids.*` token equals token (case-insensitive)
3. a normalized `type|title|year` token equals token (used for manual tokens and some edge cases)

This match logic lives in `_tombstones.filter_with(...)` and related helpers.

***

### Manual operations (practical)

#### Inspect tombstones for a pair+feature

Inside the container:

* `grep -n "watchlist:PLEX-SIMKL|" /config/.cw_state/tombstones.json | head`

#### Remove a single tombstone

Edit the JSON and delete the key line, then save.

#### Nuke tombstones for one pair+feature

Delete matching keys from the JSON file.

#### Nuke all tombstones

Delete the file:

* `/config/.cw_state/tombstones.json`

The orchestrator will recreate it when it next writes removals.

Yes, this can re-enable ping-pong deletes if you’re running two-way with removals enabled. Don’t do it casually.

***

### Common “why is X not syncing?” causes

1. Item is tombstoned from a prior delete, so adds are blocked.
2. Tombstone TTL is long and you expected it to expire faster.
3. Provider key mismatch means the token in tombstones doesn’t match the new canonical key.
   * Fix is usually to ensure `ids` are populated (so token matching works).

***

### Related pages

* Two-way deletes and observed deletions: [Two-way sync](/blueprint-architecture/orchestrator/two-way-sync.md)
* Safety model overview: [Guardrails](/blueprint-architecture/orchestrator/guardrails.md)
* Where tombstones live on disk: [State](/blueprint-architecture/orchestrator/state.md)
* Other “stop retrying” systems: [Blackbox](/blueprint-architecture/orchestrator/blackbox.md), [Unresolved](/blueprint-architecture/orchestrator/unresolved.md)
  {% endtab %}
  {% endtabs %}


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://wiki.crosswatch.app/blueprint-architecture/orchestrator/tombstones.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.