Rate limiting and retries

Tune provider pacing, retries, chunking, and write delays when APIs return 429s, 5xx errors, or timeouts.

CrossWatch uses several layers of request control.

That reduces API errors and avoids unnecessary load.

Only tune provider limits when logs show repeated 429s, 5xx errors, or timeouts.

Standard users

Most installs should keep the defaults.

Only tune rate limits when a provider returns repeated 429, temporary 5xx, or timeout errors.

Start by reducing request speed.

Change chunk sizes and delays only if needed.

What CW does by default

CW can apply:

• client-side request pacing
• separate read and write limits
• automatic retries
• exponential backoff
• Retry-After handling
• request batching and write delays

Supported providers usually use this shape:

"rate_limit": {
  "get_per_sec": 10,
  "post_per_sec": 1
}

• get_per_sec controls reads.
• post_per_sec controls writes.
• Any non-GET method uses the write bucket.

1 means about one request per second.

0.5 means about one request every two seconds.

3.33 means about one request every 300 milliseconds.

Setting a bucket to 0 disables local pacing for that bucket.

It does not disable requests.

Quick links

Related docs

Config reference

Open

Configuration reference

Related docs

General limitations

Open

Limitations

Related docs

Sync chunking internals

Open

Default provider pacing

• Trakt — 3.33 reads/sec, 1 write/sec, chunks of 100
• SIMKL — 10 reads/sec, 1 write/sec, chunks of 500
• Nuvio — 100 reads/sec, 100 writes/sec
• MDBList — 10 reads/sec, 1 write/sec, chunks of 500, write delays of 600ms
• PublicMetaDB — 20 reads/sec, 3 writes/sec, plus hourly ratings quotas

AniList and TMDb use retry handling, but do not expose configurable per-second pacing.

Plex, Emby, and Jellyfin use feature-specific pacing instead of provider-wide requests-per-second settings.

When to change settings

Change settings when you see:

• HTTP 429
• temporary 500, 502, 503, or 504
• repeated provider timeouts

Most of the time, increasing max_retries is not the first fix.

Reducing pressure works better.

Safe tuning order

When writes hit 429:

1. Reduce post_per_sec.
2. Reduce batch or chunk size.
3. Increase the write delay.

When reads hit 429:

1. Reduce get_per_sec.
2. Increase the schedule interval.
3. Avoid overlapping jobs on one account.

When you see repeated 5xx or timeouts:

1. Reduce batch or chunk size.
2. Add or increase write delays.
3. Reduce request rates.

Conservative examples

Trakt:

"trakt": {
  "timeout": 15,
  "max_retries": 5,
  "rate_limit": {
    "get_per_sec": 2,
    "post_per_sec": 0.5
  },
  "watchlist_batch_size": 50,
  "ratings_chunk_size": 50,
  "history_chunk_size": 50
}

MDBList:

"mdblist": {
  "timeout": 15,
  "max_retries": 3,
  "rate_limit": {
    "get_per_sec": 5,
    "post_per_sec": 0.5
  },
  "watchlist_batch_size": 250,
  "ratings_chunk_size": 250,
  "history_chunk_size": 250,
  "ratings_write_delay_ms": 1000,
  "history_write_delay_ms": 1000,
  "ratings_max_backoff_ms": 10000,
  "history_max_backoff_ms": 10000
}

Use conservative settings when several jobs, profiles, or installs share one account.

Related

• Limitations
• Best practices
• Chunking

Power users

Request control layers

CW uses several layers of request control to reduce API errors and prevent unnecessary load on media servers and tracking providers.

Rate limiting is not limited to a single requests-per-second setting.

Depending on the provider, CW can apply:

• client-side request pacing
• separate limits for reads and writes
• automatic retries
• exponential backoff
• Retry-After handling
• request batching
• write delays
• orchestrator chunk pauses
• scheduler jitter
• rate-limit diagnostics

The default settings should work for most installations.

Changes are normally only required when a provider returns HTTP 429, temporary 5xx, or repeated timeout errors.

Request pacing

Supported providers use a client-side limiter that spaces requests before they are sent.

Two request buckets are available:

• get_per_sec controls GET requests
• post_per_sec controls POST, DELETE, and other non-GET requests

Example:

"rate_limit": {
  "get_per_sec": 3.33,
  "post_per_sec": 1
}

A value of 1 allows about one request per second.

A value of 0.5 allows about one request every two seconds.

A value of 3.33 allows about one request every 300 milliseconds.

Setting a value to 0 disables client-side pacing for that request bucket.

It does not disable the requests themselves.

Providers with configurable request pacing

Trakt

Default settings:

"trakt": {
  "timeout": 10,
  "max_retries": 5,
  "rate_limit": {
    "get_per_sec": 3.33,
    "post_per_sec": 1
  },
  "watchlist_batch_size": 100,
  "ratings_chunk_size": 100,
  "history_chunk_size": 100
}

Trakt uses separate read and write pacing.

The default write limit allows one write request per second.

Watchlist, ratings, and history changes are also split into chunks of 100 items.

SIMKL

Default settings:

"simkl": {
  "rate_limit": {
    "get_per_sec": 10,
    "post_per_sec": 1
  },
  "watchlist_batch_size": 500,
  "ratings_chunk_size": 500,
  "history_chunk_size": 500
}

SIMKL permits faster reads than writes.

Write operations are limited to one request per second by default.

Nuvio

Default settings:

"nuvio": {
  "rate_limit": {
    "get_per_sec": 100,
    "post_per_sec": 100
  }
}

Nuvio allows up to 100 read and write requests per second.

MDBList

Default settings:

"mdblist": {
  "timeout": 10,
  "max_retries": 3,
  "rate_limit": {
    "get_per_sec": 10,
    "post_per_sec": 1
  },
  "watchlist_batch_size": 500,
  "ratings_chunk_size": 500,
  "history_chunk_size": 500,
  "ratings_write_delay_ms": 600,
  "history_write_delay_ms": 600,
  "ratings_max_backoff_ms": 8000,
  "history_max_backoff_ms": 8000
}

MDBList combines provider pacing with feature-specific write delays and backoff limits.

The default 600 millisecond write delay adds extra spacing between ratings and history writes.

PublicMetaDB

Default settings:

"publicmetadb": {
  "timeout": 15,
  "max_retries": 3,
  "rate_limit": {
    "get_per_sec": 20,
    "post_per_sec": 3
  },
  "ratings_submit_per_hour": 200,
  "ratings_update_per_hour": 100
}

PublicMetaDB limits are capped by CW at:

• 20 GET requests per second
• 3 write requests per second
• 200 new rating submissions per hour
• 100 rating updates per hour

Values above these limits are reduced to the supported maximum.

Providers using automatic retry handling

AniList and TMDb do not currently expose configurable requests-per-second controls.

They still use the shared request handler, which provides:

• automatic retries
• exponential backoff
• HTTP 429 handling
• temporary 5xx handling
• connection and timeout retry handling
• rate-limit response header diagnostics

Plex, Emby, and Jellyfin use provider-specific and feature-specific pacing instead of a provider-level requests-per-second limiter.

Automatic retries

CW automatically retries requests for these HTTP responses:

• 429
• 500
• 502
• 503
• 504

Connection errors and request exceptions can also be retried.

The default retry delay starts at about 500 milliseconds.

It increases exponentially after each failed attempt.

Typical retry delays are:

• first retry — 0.5 seconds
• second retry — 1 second
• third retry — 2 seconds
• fourth retry — 4 seconds

The actual number of attempts depends on the provider max_retries setting.

Increasing max_retries does not prevent rate limiting.

It only controls how often CW retries a failed request.

Reducing request speed or batch size is usually more effective.

Retry-After handling

When a provider returns HTTP 429 with a numeric Retry-After header, CW waits for at least that duration before retrying.

CW also reads common provider rate-limit headers:

• X-RateLimit-Limit
• X-RateLimit-Remaining
• X-RateLimit-Reset
• RateLimit-Limit
• RateLimit-Remaining
• RateLimit-Reset

These headers are used for diagnostics.

CW does not automatically change configured requests-per-second values based on remaining request count.

Only Retry-After directly changes the retry wait time.

Batch and chunk sizes

A batch or chunk size controls how many media items are included in one operation.

It does not directly control requests per second.

Examples:

"watchlist_batch_size": 100,
"ratings_chunk_size": 100,
"history_chunk_size": 100

Smaller batches can reduce processing time and lower the chance of provider timeouts.

Larger batches can finish a sync faster.

They can also produce larger payloads and longer provider processing times.

Write delays

Write delays add an extra pause between provider write operations.

They are configured in milliseconds.

Example:

"ratings_write_delay_ms": 1000

A value of 1000 adds a one-second pause between supported ratings writes.

Available write delay settings include:

"plex": {
  "watchlist_write_delay_ms": 100
}

"jellyfin": {
  "watchlist": {
    "watchlist_write_delay_ms": 100
  },
  "history": {
    "history_write_delay_ms": 100
  }
}

"emby": {
  "watchlist": {
    "watchlist_write_delay_ms": 100
  },
  "history": {
    "history_write_delay_ms": 100
  }
}

"mdblist": {
  "ratings_write_delay_ms": 600,
  "history_write_delay_ms": 600
}

Write delays help when a provider accepts the first requests, then starts returning 429, 502, 503, or timeout errors during larger runs.

Orchestrator chunk pacing

CW can pause between groups of planned synchronization changes.

Default settings:

"runtime": {
  "apply_chunk_size": 100,
  "apply_chunk_pause_ms": 50,
  "apply_chunk_size_by_provider": {
    "SIMKL": 500,
    "MDBLIST": 500,
    "PUBLICMETADB": 500
  }
}

apply_chunk_size controls how many planned changes are processed before a pause is applied.

apply_chunk_pause_ms controls the pause between those groups.

Provider-specific values in apply_chunk_size_by_provider override the general chunk size.

This pacing is separate from the provider requests-per-second limiter.

Scheduler jitter

Scheduled synchronizations can be distributed by adding a random delay to the calculated start time.

Example:

"scheduling": {
  "jitter_seconds": 300
}

A value of 300 adds a random delay between zero and five minutes.

Scheduler jitter helps when several jobs, profiles, or installs would otherwise start at the same time.

Avoid starting several large synchronization jobs against the same provider account at the exact same time.

Important limitation

Request pacing is maintained by each active provider client session.

It is not a shared distributed quota across every CW process, container, profile, or component.

For example, two CW containers configured for one Trakt account can each send requests at their configured limit.

Their combined traffic can exceed the provider limit.

The same can happen when several provider profiles or synchronization jobs access the same remote account at once.

Configure lower values when multiple active sessions share one provider account.

Diagnosing rate-limit problems

CW writes structured events for pacing and retry activity.

Relevant events include:

• client_throttle_activity
• rate_limit_retry
• http_retry
• http_exception_retry
• http_exception_giveup
• http_request_failed

A rate_limit_retry event can include:

• provider
• API feature
• HTTP method
• HTTP status
• current attempt
• retry budget
• wait duration
• Retry-After duration
• provider request limit
• remaining requests
• reset value

Client pacing messages are summarized to avoid a log entry for every short sleep.

You can adjust the logging frequency with environment variables:

CW_RATE_LIMIT_LOG_EVERY_S=60
CW_RATE_LIMIT_LOG_MIN_SLEEP_S=0.5

You can enable API request hit events with:

CW_API_HITS=1

API hit logging can generate a large amount of data.

Enable it only while investigating a problem.

Recommended tuning order

When a provider returns HTTP 429 during write operations:

1. Reduce post_per_sec.
2. Reduce the provider batch or chunk size.
3. Increase the relevant write delay.
4. Add scheduler jitter.
5. Prevent simultaneous jobs from using the same provider account.

When a provider returns HTTP 429 during indexing or reading:

1. Reduce get_per_sec.
2. Increase the interval between scheduled synchronizations.
3. Reduce page sizes where supported.
4. Avoid running several pairs against the same provider profile at once.

When a provider returns repeated 5xx or timeouts:

1. Reduce batch or chunk sizes.
2. Add or increase write delays.
3. Reduce request rates.
4. Check the provider service status.
5. Only increase max_retries after pacing has been adjusted.

Conservative Trakt example

"trakt": {
  "timeout": 15,
  "max_retries": 5,
  "rate_limit": {
    "get_per_sec": 2,
    "post_per_sec": 0.5
  },
  "watchlist_batch_size": 50,
  "ratings_chunk_size": 50,
  "history_chunk_size": 50
}

This allows about two read requests per second and one write request every two seconds.

Conservative MDBList example

"mdblist": {
  "timeout": 15,
  "max_retries": 3,
  "rate_limit": {
    "get_per_sec": 5,
    "post_per_sec": 0.5
  },
  "watchlist_batch_size": 250,
  "ratings_chunk_size": 250,
  "history_chunk_size": 250,
  "ratings_write_delay_ms": 1000,
  "history_write_delay_ms": 1000,
  "ratings_max_backoff_ms": 10000,
  "history_max_backoff_ms": 10000
}

Use conservative settings when an account is shared across several profiles, jobs, or CW installations.