Tombstones

Deletion memory that prevents re-add loops and enables safer delete propagation in two-way sync.

Tombstones are deletion memory.

They stop “delete → re-add → delete → re-add” loops in two-way sync.

When tombstones matter

You deleted an item on one side.
Two-way would otherwise re-add it from the other side.

If something won’t re-add

It may still be tombstoned.

Wait for tombstone TTL to expire, or remove it manually.

Two-way deletes: Two-way sync

Tombstones are the orchestrator’s “deletion memory”. They exist mainly to make two-way sync safe by preventing delete/re-add ping-pong and by allowing deletion propagation only when there is strong evidence.

Code: cw_platform/orchestrator/_tombstones.py Used by: orchestrator/_pairs_oneway.py, orchestrator/_pairs_twoway.py, orchestrator/_pairs_blocklist.py

What a tombstone is (in this codebase)

A tombstone is a record that a specific token was deleted in a given feature + provider pair context.

A token can be:

a canonical key (e.g., imdb:tt0111161)
an ID token from item["ids"] (e.g., tmdb:278, tvdb:12345, simkl:...)
a typed token for seasons/episodes (if providers put those into ids or derived tokens)

Tombstones are stored in a single JSON map file:

/config/.cw_state/tombstones.json

Why tombstones exist

Without tombstones, two-way sync has a classic failure mode:

Item is removed on A.
Sync sees item still on B and re-adds it back to A.
User removes it again.
Repeat until someone throws the server out the window.

Tombstones fix this by:

remembering that “this token was deleted”
blocking re-adds for a configurable time window (TTL)
enabling the orchestrator to propagate deletes intentionally (only when safe)

Storage format

Keys in tombstones.json are strings:

{feature}:{PAIR}|{token}

Where:

feature: watchlist|ratings|history|playlists
PAIR: providers sorted, e.g. PLEX-SIMKL, JELLYFIN-PLEX
token: canonical key or ID token

Example key:

watchlist:PLEX-SIMKL|imdb:tt1234567

Value is a small dict:

{
  "at": 1738100000,
  "why": "remove"
}

why is optional and mainly used for debugging:

"remove" (explicit remove write succeeded)
"observed_delete" (two-way observed deletions)

Reading tombstones

`keys_for_feature(state_store, feature, pair) -> dict[str, dict]`

Loads tombstones.json and filters by:

matching feature
matching pair (exact string like PLEX-SIMKL)

Returns a map:

token -> metadata

So the caller usually takes:

tomb = set(map.keys())

TTL (how long tombstones are “active”)

The orchestrator enforces TTL by filtering tomb entries:

sync.tombstone_ttl_days (default commonly 30)

Rule:

if now - meta["at"] > ttl_days * 86400 → ignore it for blocking/planning

Important:

tombstones may remain in the file even after TTL; they’re just treated as expired.
some code paths may later prune old entries (or you can nuke them manually).

Writing tombstones on successful removals

One-way

After confirmed destination removals:

tombstones are written for the destination side under feature:{PAIR}|token.

Tokens written:

canonical key (canonical_key(item))
every ids.* token in item["ids"]

This increases match reliability across providers even when canonical keys differ.

Two-way

Same, but removals can happen on either side. Two-way also writes tombstones for observed deletions (see below).

Observed deletions (two-way) → tombstones

Two-way computes “observed deletions” as:

obsA = prevA.keys - A_cur.keys
obsB = prevB.keys - B_cur.keys

These are keys that existed in baseline but disappeared from the latest live snapshot.

When safe (not bootstrapping, not down, not suspect snapshot), two-way:

writes tombstones for those observed-deleted keys
also attempts to write ID tokens by looking up the missing item in prevA/prevB

This is the core mechanism that turns “it vanished on one side” into a durable “we consider it deleted”.

How tombstones block re-adds (and where)

Tombstones are applied as a blocklist in two main places:

1) Two-way planning

During planning, the orchestrator expands tomb tokens (tombX) to match canonical keys via alias maps. If an item matches tombstones strongly, it can:

be prevented from re-adding, or
be treated as a deletion that should propagate (if removals enabled)

2) `apply_blocklist(...)` for adds (non-watchlist)

orchestrator/_pairs_blocklist.py uses tombstones as one component of its add blocklist:

tombstones + unresolved + blackbox
only wired for non-watchlist features in current pipelines

Note:

watchlist does not go through apply_blocklist in current wiring; it relies on PhantomGuard + tombstone handling in two-way logic.

Matching logic: token vs item

A tomb token matches an item if any of these match:

canonical key equals token
any ids.* token equals token (case-insensitive)
a normalized type|title|year token equals token (used for manual tokens and some edge cases)

This match logic lives in _tombstones.filter_with(...) and related helpers.

Manual operations (practical)

Inspect tombstones for a pair+feature

Inside the container:

grep -n "watchlist:PLEX-SIMKL|" /config/.cw_state/tombstones.json | head

Remove a single tombstone

Edit the JSON and delete the key line, then save.

Nuke tombstones for one pair+feature

Delete matching keys from the JSON file.

Nuke all tombstones

Delete the file:

/config/.cw_state/tombstones.json

The orchestrator will recreate it when it next writes removals.

Yes, this can re-enable ping-pong deletes if you’re running two-way with removals enabled. Don’t do it casually.

Common “why is X not syncing?” causes

Item is tombstoned from a prior delete, so adds are blocked.
Tombstone TTL is long and you expected it to expire faster.
Provider key mismatch means the token in tombstones doesn’t match the new canonical key.
- Fix is usually to ensure ids are populated (so token matching works).

Two-way deletes and observed deletions: Two-way sync
Safety model overview: Guardrails
Where tombstones live on disk: State
Other “stop retrying” systems: Blackbox, Unresolved

PreviousPlanner NextUnresolved

Last updated 4 months ago

Was this helpful?