Overload is a scheduling problem before it is a scaling problem
An API that performs document extraction, report generation, or model inference can receive requests faster than its dependencies finish them. Accepting everything initially looks customer-friendly. Soon the queue is full of work whose users have already left, and current requests wait behind it.
Design admission around useful completion. The question is whether the system can finish this request before its deadline without violating other commitments. More workers help only when the constrained dependency can support them.
Google's SRE guidance emphasizes that requests differ in resource cost and that queries per second alone can misrepresent capacity. It also describes degraded responses and explicit rejection as tools for handling overload. Apply those principles to the actual expensive resources in your product. Google SRE: Handling Overload
Estimate concurrency from the bottleneck
For an illustrative workload, suppose sustainable throughput is 20 jobs per second and average time in the system is two seconds. Little's Law gives an average of 40 jobs in the system under stable conditions. This is a planning relationship, not a safe concurrency limit: tail latency, burstiness, and workload variance still require measurement.
Separate CPU work, database connections, provider concurrency, and object-store bandwidth. One universal worker count hides which resource is saturated. An OCR job and a small metadata update should not consume the same admission token merely because both arrive at the same endpoint.
Use measured cost classes where feasible. Reserve capacity for interactive operations and give batch work a separate queue or concurrency pool. Ensure a single customer cannot occupy every slot through a large import.
Bound both queue length and queue age
An admission decision can consider active work, estimated queued work, dependency health, caller quota, and the request deadline. Reject or defer when the work cannot plausibly complete in time. Estimates will be imperfect, so use conservative bounds and measure prediction error.
remaining_budget = deadline - current_time
estimated_wait = queued_work_units / sustainable_work_units_per_second
admit only when:
caller has capacity
queue has capacity
dependency is eligible
estimated_wait + estimated_service_time < remaining_budget
This is illustrative policy logic, not a universal formula. Priority queues and variable work sizes require a scheduler-specific estimate. Once admitted, persist the deadline with the job so workers can discard obsolete work before consuming expensive resources.
For asynchronous jobs, acknowledge only after durable acceptance and provide a status resource. Define cancellation semantics explicitly: cancellation may stop pending work but cannot undo an external side effect that already committed.
Make retries consume a budget
If three layers each make three attempts, one user operation can trigger 27 downstream attempts. A failure that should reduce traffic can instead amplify it. Choose one retry owner per boundary and propagate attempt and deadline information.
Retry only transient, eligible failures. A validation error or an authorization denial will not improve after a delay. A timeout on a write represents uncertainty; use the operation's idempotency contract or reconcile before issuing another effect.
Bound retries by attempts, elapsed time, and a shared retry budget. Add jitter so clients do not synchronize after an outage. Do not launch a retry whose deadline has already expired. Release concurrency permits on every completion path, including cancellation and exceptions.
Degrade only where the product permits it
| Work | Possible degraded mode | Boundary to preserve |
|---|---|---|
| Report preview | Smaller sample with a visible label | Never present it as the complete report |
| Document search | Defer optional enrichment | Preserve access filtering and source identity |
| Model-assisted drafting | Queue for later or offer manual editing | Do not silently substitute unsupported output |
| Payment or access change | Reject before commitment or reconcile | Never guess whether the operation succeeded |
Overload is not a reason to bypass authorization, skip validation, or invent a successful response. Identify optional work ahead of time so the incident path does not require improvising product semantics.
Protect recovery traffic too. Health checks, reconciliation, and cancellation may need a small reserved pool; otherwise saturated normal traffic can prevent the operations required to recover.
Validate the recovery curve
Run a load test beyond sustainable capacity, then reduce demand. The important result is whether latency, queue age, and error rate recover within a bounded interval. A service that accepts the burst but takes hours to drain has shifted the outage rather than avoided it.
Test provider slowdown, worker loss, one customer's burst, and synchronized client retries. Track admitted, rejected, expired-before-start, canceled, and successfully completed work separately. Measure business completion by deadline, not just HTTP acceptance.
Tune the system to preserve a useful subset of work under pressure. Explicit rejection with a clear retry policy is often easier for clients to recover from than an apparently successful request that disappears into an unbounded queue.