Chunking

How write batches are split and throttled to reduce rate limits and failure blast radius.

Chunking splits big writes into smaller batches.

It helps when a provider is strict or rate-limited.

When to enable chunking

Enable it if you see:

rate limit errors
timeouts on large batches
“one bad item breaks the whole batch”

Safe starting point

Start with apply_chunk_size: 25
Add apply_chunk_pause_ms: 200 if you still hit limits

Runtime config keys: Runtime

This doc describes how the orchestrator decides batch sizes for writes, and how it slows down to avoid rate-limit pain.

Code: cw_platform/orchestrator/_chunking.py + orchestrator/_applier.py

Why chunking exists

Providers vary wildly:

Some accept big batches (Plex-ish).
Some want small batches (Trakt-ish).
Some accept batches but rate-limit aggressively (Simkl-ish depending on endpoint).

Chunking gives you:

faster progress visibility
less blast radius per API call
fewer “one bad item ruins 500” failures

Where chunking is applied

Chunking happens in the applier:

_applier._apply_chunked(...)

The pipeline passes:

chunk_size
chunk_pause_ms

If chunking is off, applier calls provider once per add/remove.

How chunk size is chosen

The orchestrator computes chunk size via:

effective_chunk_size(ctx, provider_name) in _chunking.py

Inputs:

ctx.apply_chunk_size (base size)
ctx.apply_chunk_size_by_provider (overrides)
provider name normalization (uppercased)

Resolution rule:

If per-provider override exists and > 0 → use it
Else if base size > 0 → use base size
Else → 0 (no chunking)

Config keys

In config.json:

{
  "runtime": {
    "apply_chunk_size": 50,
    "apply_chunk_pause_ms": 200,
    "apply_chunk_size_by_provider": {
      "TRAKT": 10,
      "SIMKL": 25,
      "PUBLICMETADB": 25
    }
  }
}

Accepted alias keys (same behavior):

apply_chunk_sizes_by_provider
apply_chunk_sizes

Provider keys are uppercased internally.

Chunk pause (throttle between chunks)

If runtime.apply_chunk_pause_ms > 0:

applier sleeps after each chunk

This is a simple “be polite” delay. It’s not adaptive and doesn’t look at headers.

Rate-limit awareness (telemetry warnings)

Chunking itself doesn’t read rate-limit headers. Instead, the orchestrator can emit warnings if rate remaining is low.

At end of run:

ctx.emit_rate_warnings()
uses thresholds from telemetry.warn_rate_remaining

Default thresholds are provider-specific and conservative.

This is a warning system only; it doesn’t auto-throttle.

If you want adaptive rate limiting, you’d implement it in providers (based on 429 / Retry-After, etc.).

Provider side: recommended behavior under chunking

Providers should treat each chunk as independent and return:

confirmed_keys when possible
unresolved as a list of items with IDs for failures

If a provider throws an exception for a chunk:

applier retries that chunk up to 3 times with exponential backoff
if it still fails, it treats that chunk as failed and returns ok:false at aggregate level

This is why chunking can help:

one failing chunk doesn’t nuke the entire run.

Common chunking setups

Conservative (good for Trakt/Simkl)

base 25
Trakt override 10
pause 150–300ms

Aggressive (fast, but riskier)

base 100
no pause

“Debug mode”

base 1
pause 0

Helps isolate a single bad item and see which exact key fails.

Gotchas

Chunking increases total calls → can increase rate-limit pressure if set too low.
If provider doesn’t return confirmed_keys, it can be harder to map successes/failures precisely.
If you chunk too large, one provider error can still fail a lot of items.

How chunking aggregates results: Applier
Where chunk config is parsed: Runtime
Progress events: Eventing

PreviousHealth NextCaching layers

Last updated 20 days ago

Was this helpful?