Rate limiting and retries
Tune provider pacing, retries, chunking, and write delays when APIs return 429s, 5xx errors, or timeouts.
CrossWatch uses several layers of request control.
That reduces API errors and avoids unnecessary load.
Only tune provider limits when logs show repeated 429s, 5xx errors, or timeouts.
Most installs should keep the defaults.
Only tune rate limits when a provider returns repeated 429, temporary 5xx, or timeout errors.
Start by reducing request speed.
Change chunk sizes and delays only if needed.
What CrossWatch does by default
CrossWatch can apply:
client-side request pacing
separate read and write limits
automatic retries
exponential backoff
Retry-Afterhandlingrequest batching and write delays
Supported providers usually use this shape:
"rate_limit": {
"get_per_sec": 10,
"post_per_sec": 1
}get_per_seccontrols reads.post_per_seccontrols writes.Any non-
GETmethod uses the write bucket.
1 means about one request per second.
0.5 means about one request every two seconds.
3.33 means about one request every 300 milliseconds.
Setting a bucket to 0 disables local pacing for that bucket.
It does not disable requests.
Quick links
Default provider pacing
Trakt —
3.33reads/sec,1write/sec, chunks of100SIMKL —
10reads/sec,1write/sec, chunks of500MDBList —
10reads/sec,1write/sec, chunks of500, write delays of600msPublicMetaDB —
20reads/sec,3writes/sec, plus hourly ratings quotas
AniList and TMDb use retry handling, but do not expose configurable per-second pacing.
Plex, Emby, and Jellyfin use feature-specific pacing instead of provider-wide requests-per-second settings.
When to change settings
Change settings when you see:
HTTP
429temporary
500,502,503, or504repeated provider timeouts
Most of the time, increasing max_retries is not the first fix.
Reducing pressure works better.
Safe tuning order
When writes hit 429:
Reduce
post_per_sec.Reduce batch or chunk size.
Increase the write delay.
When reads hit 429:
Reduce
get_per_sec.Increase the schedule interval.
Avoid overlapping jobs on one account.
When you see repeated 5xx or timeouts:
Reduce batch or chunk size.
Add or increase write delays.
Reduce request rates.
Conservative examples
Trakt:
MDBList:
Use conservative settings when several jobs, profiles, or installs share one account.
Related
Request control layers
CrossWatch uses several layers of request control to reduce API errors and prevent unnecessary load on media servers and tracking providers.
Rate limiting is not limited to a single requests-per-second setting.
Depending on the provider, CrossWatch can apply:
client-side request pacing
separate limits for reads and writes
automatic retries
exponential backoff
Retry-Afterhandlingrequest batching
write delays
orchestrator chunk pauses
scheduler jitter
rate-limit diagnostics
The default settings should work for most installations.
Changes are normally only required when a provider returns HTTP 429, temporary 5xx, or repeated timeout errors.
Request pacing
Supported providers use a client-side limiter that spaces requests before they are sent.
Two request buckets are available:
get_per_seccontrolsGETrequestspost_per_seccontrolsPOST,DELETE, and other non-GETrequests
Example:
A value of 1 allows about one request per second.
A value of 0.5 allows about one request every two seconds.
A value of 3.33 allows about one request every 300 milliseconds.
Setting a value to 0 disables client-side pacing for that request bucket.
It does not disable the requests themselves.
Providers with configurable request pacing
Trakt
Default settings:
Trakt uses separate read and write pacing.
The default write limit allows one write request per second.
Watchlist, ratings, and history changes are also split into chunks of 100 items.
SIMKL
Default settings:
SIMKL permits faster reads than writes.
Write operations are limited to one request per second by default.
MDBList
Default settings:
MDBList combines provider pacing with feature-specific write delays and backoff limits.
The default 600 millisecond write delay adds extra spacing between ratings and history writes.
PublicMetaDB
Default settings:
PublicMetaDB limits are capped by CrossWatch at:
20GETrequests per second3write requests per second200new rating submissions per hour100rating updates per hour
Values above these limits are reduced to the supported maximum.
Providers using automatic retry handling
AniList and TMDb do not currently expose configurable requests-per-second controls.
They still use the shared request handler, which provides:
automatic retries
exponential backoff
HTTP
429handlingtemporary
5xxhandlingconnection and timeout retry handling
rate-limit response header diagnostics
Plex, Emby, and Jellyfin use provider-specific and feature-specific pacing instead of a provider-level requests-per-second limiter.
Automatic retries
CrossWatch automatically retries requests for these HTTP responses:
429500502503504
Connection errors and request exceptions can also be retried.
The default retry delay starts at about 500 milliseconds.
It increases exponentially after each failed attempt.
Typical retry delays are:
first retry —
0.5secondssecond retry —
1secondthird retry —
2secondsfourth retry —
4seconds
The actual number of attempts depends on the provider max_retries setting.
Increasing max_retries does not prevent rate limiting.
It only controls how often CrossWatch retries a failed request.
Reducing request speed or batch size is usually more effective.
Retry-After handling
When a provider returns HTTP 429 with a numeric Retry-After header, CrossWatch waits for at least that duration before retrying.
CrossWatch also reads common provider rate-limit headers:
X-RateLimit-LimitX-RateLimit-RemainingX-RateLimit-ResetRateLimit-LimitRateLimit-RemainingRateLimit-Reset
These headers are used for diagnostics.
CrossWatch does not automatically change configured requests-per-second values based on remaining request count.
Only Retry-After directly changes the retry wait time.
Batch and chunk sizes
A batch or chunk size controls how many media items are included in one operation.
It does not directly control requests per second.
Examples:
Smaller batches can reduce processing time and lower the chance of provider timeouts.
Larger batches can finish a sync faster.
They can also produce larger payloads and longer provider processing times.
Write delays
Write delays add an extra pause between provider write operations.
They are configured in milliseconds.
Example:
A value of 1000 adds a one-second pause between supported ratings writes.
Available write delay settings include:
Write delays help when a provider accepts the first requests, then starts returning 429, 502, 503, or timeout errors during larger runs.
Orchestrator chunk pacing
CrossWatch can pause between groups of planned synchronization changes.
Default settings:
apply_chunk_size controls how many planned changes are processed before a pause is applied.
apply_chunk_pause_ms controls the pause between those groups.
Provider-specific values in apply_chunk_size_by_provider override the general chunk size.
This pacing is separate from the provider requests-per-second limiter.
Scheduler jitter
Scheduled synchronizations can be distributed by adding a random delay to the calculated start time.
Example:
A value of 300 adds a random delay between zero and five minutes.
Scheduler jitter helps when several jobs, profiles, or installs would otherwise start at the same time.
Avoid starting several large synchronization jobs against the same provider account at the exact same time.
Important limitation
Request pacing is maintained by each active provider client session.
It is not a shared distributed quota across every CrossWatch process, container, profile, or component.
For example, two CrossWatch containers configured for one Trakt account can each send requests at their configured limit.
Their combined traffic can exceed the provider limit.
The same can happen when several provider profiles or synchronization jobs access the same remote account at once.
Configure lower values when multiple active sessions share one provider account.
Diagnosing rate-limit problems
CrossWatch writes structured events for pacing and retry activity.
Relevant events include:
client_throttle_activityrate_limit_retryhttp_retryhttp_exception_retryhttp_exception_giveuphttp_request_failed
A rate_limit_retry event can include:
provider
API feature
HTTP method
HTTP status
current attempt
retry budget
wait duration
Retry-Afterdurationprovider request limit
remaining requests
reset value
Client pacing messages are summarized to avoid a log entry for every short sleep.
You can adjust the logging frequency with environment variables:
You can enable API request hit events with:
API hit logging can generate a large amount of data.
Enable it only while investigating a problem.
Recommended tuning order
When a provider returns HTTP 429 during write operations:
Reduce
post_per_sec.Reduce the provider batch or chunk size.
Increase the relevant write delay.
Add scheduler jitter.
Prevent simultaneous jobs from using the same provider account.
When a provider returns HTTP 429 during indexing or reading:
Reduce
get_per_sec.Increase the interval between scheduled synchronizations.
Reduce page sizes where supported.
Avoid running several pairs against the same provider profile at once.
When a provider returns repeated 5xx or timeouts:
Reduce batch or chunk sizes.
Add or increase write delays.
Reduce request rates.
Check the provider service status.
Only increase
max_retriesafter pacing has been adjusted.
Conservative Trakt example
This allows about two read requests per second and one write request every two seconds.
Conservative MDBList example
Use conservative settings when an account is shared across several profiles, jobs, or CrossWatch installations.
Last updated
Was this helpful?