Tombstones
Deletion memory that prevents re-add loops and enables safer delete propagation in two-way sync.
Tombstones are deletion memory.
They stop “delete → re-add → delete → re-add” loops in two-way sync.
When tombstones matter
You deleted an item on one side.
Two-way would otherwise re-add it from the other side.
If something won’t re-add
It may still be tombstoned.
Wait for tombstone TTL to expire, or remove it manually.
Related:
Two-way deletes: Two-way sync
Tombstones are the orchestrator’s “deletion memory”. They exist mainly to make two-way sync safe by preventing delete/re-add ping-pong and by allowing deletion propagation only when there is strong evidence.
Code: cw_platform/orchestrator/_tombstones.py
Used by: orchestrator/_pairs_oneway.py, orchestrator/_pairs_twoway.py, orchestrator/_pairs_blocklist.py
What a tombstone is (in this codebase)
A tombstone is a record that a specific token was deleted in a given feature + provider pair context.
A token can be:
a canonical key (e.g.,
imdb:tt0111161)an ID token from
item["ids"](e.g.,tmdb:278,tvdb:12345,simkl:...)a typed token for seasons/episodes (if providers put those into
idsor derived tokens)
Tombstones are stored in a single JSON map file:
/config/.cw_state/tombstones.json
Why tombstones exist
Without tombstones, two-way sync has a classic failure mode:
Item is removed on A.
Sync sees item still on B and re-adds it back to A.
User removes it again.
Repeat until someone throws the server out the window.
Tombstones fix this by:
remembering that “this token was deleted”
blocking re-adds for a configurable time window (TTL)
enabling the orchestrator to propagate deletes intentionally (only when safe)
Storage format
Keys in tombstones.json are strings:
Where:
feature:watchlist|ratings|history|playlistsPAIR: providers sorted, e.g.PLEX-SIMKL,JELLYFIN-PLEXtoken: canonical key or ID token
Example key:
watchlist:PLEX-SIMKL|imdb:tt1234567
Value is a small dict:
why is optional and mainly used for debugging:
"remove"(explicit remove write succeeded)"observed_delete"(two-way observed deletions)
Reading tombstones
keys_for_feature(state_store, feature, pair) -> dict[str, dict]
keys_for_feature(state_store, feature, pair) -> dict[str, dict]Loads tombstones.json and filters by:
matching
featurematching
pair(exact string likePLEX-SIMKL)
Returns a map:
token -> metadata
So the caller usually takes:
tomb = set(map.keys())
TTL (how long tombstones are “active”)
The orchestrator enforces TTL by filtering tomb entries:
sync.tombstone_ttl_days(default commonly 30)
Rule:
if
now - meta["at"] > ttl_days * 86400→ ignore it for blocking/planning
Important:
tombstones may remain in the file even after TTL; they’re just treated as expired.
some code paths may later prune old entries (or you can nuke them manually).
Writing tombstones on successful removals
One-way
After confirmed destination removals:
tombstones are written for the destination side under
feature:{PAIR}|token.
Tokens written:
canonical key (
canonical_key(item))every
ids.*token initem["ids"]
This increases match reliability across providers even when canonical keys differ.
Two-way
Same, but removals can happen on either side. Two-way also writes tombstones for observed deletions (see below).
Observed deletions (two-way) → tombstones
Two-way computes “observed deletions” as:
obsA = prevA.keys - A_cur.keysobsB = prevB.keys - B_cur.keys
These are keys that existed in baseline but disappeared from the latest live snapshot.
When safe (not bootstrapping, not down, not suspect snapshot), two-way:
writes tombstones for those observed-deleted keys
also attempts to write ID tokens by looking up the missing item in
prevA/prevB
This is the core mechanism that turns “it vanished on one side” into a durable “we consider it deleted”.
How tombstones block re-adds (and where)
Tombstones are applied as a blocklist in two main places:
1) Two-way planning
During planning, the orchestrator expands tomb tokens (tombX) to match canonical keys via alias maps. If an item matches tombstones strongly, it can:
be prevented from re-adding, or
be treated as a deletion that should propagate (if removals enabled)
2) apply_blocklist(...) for adds (non-watchlist)
apply_blocklist(...) for adds (non-watchlist)orchestrator/_pairs_blocklist.py uses tombstones as one component of its add blocklist:
tombstones + unresolved + blackbox
only wired for non-watchlist features in current pipelines
Note:
watchlist does not go through
apply_blocklistin current wiring; it relies on PhantomGuard + tombstone handling in two-way logic.
Matching logic: token vs item
A tomb token matches an item if any of these match:
canonical key equals token
any
ids.*token equals token (case-insensitive)a normalized
type|title|yeartoken equals token (used for manual tokens and some edge cases)
This match logic lives in _tombstones.filter_with(...) and related helpers.
Manual operations (practical)
Inspect tombstones for a pair+feature
Inside the container:
grep -n "watchlist:PLEX-SIMKL|" /config/.cw_state/tombstones.json | head
Remove a single tombstone
Edit the JSON and delete the key line, then save.
Nuke tombstones for one pair+feature
Delete matching keys from the JSON file.
Nuke all tombstones
Delete the file:
/config/.cw_state/tombstones.json
The orchestrator will recreate it when it next writes removals.
Yes, this can re-enable ping-pong deletes if you’re running two-way with removals enabled. Don’t do it casually.
Common “why is X not syncing?” causes
Item is tombstoned from a prior delete, so adds are blocked.
Tombstone TTL is long and you expected it to expire faster.
Provider key mismatch means the token in tombstones doesn’t match the new canonical key.
Fix is usually to ensure
idsare populated (so token matching works).
Related pages
Two-way deletes and observed deletions: Two-way sync
Safety model overview: Guardrails
Where tombstones live on disk: State
Other “stop retrying” systems: Blackbox, Unresolved
Last updated