429 Too Many Requests Through a Proxy

Find out who returned the 429

Start with the stage that failed. If your client reports 429 in reply to an HTTP CONNECT request, the proxy rejected the tunnel request before the HTTPS call reached the destination. If the tunnel opens and your client then gets an HTTP 429 from the destination, check the destination's API or gateway rules. The application, an API gateway or another service acting for the destination may have applied the limit.

Also check which URL your job was calling. For example, IPMusk documents a 429 rate_limited error with Retry-After in seconds for its own customer API. That limit applies to requests to the IPMusk API. If your job was fetching a page through a proxy and the page request returned 429, look up the destination's rate policy.

Record the timestamp, destination host, which stage returned the status and the Retry-After value. Keep the request ID if the service provides one. Those details let you match the response to a service log or a documented quota. Keep authentication headers and proxy passwords out of the log.

Read the wait time in Retry-After

RFC 6585 defines 429 and allows the server to send a Retry-After header. RFC 9110 defines two forms for the header. Check the raw value before converting it, since an HTTP date needs different handling from a count of seconds.

A value of 30 asks you to wait at least 30 seconds from receipt. A date gives the earliest time for the next attempt. Compare that timestamp with a machine clock you trust. A clock running fast can make the client retry before the service is ready. When the requested wait is longer than the job can afford, save its place and resume later.

Run a local endpoint that returns 429

To see the wait happen, save this Python 3 example as server.py and run python3 server.py. It prints the local port to use in the next step. On the first GET it opens a one-second cooldown, returns 429 and advertises that delay. An early second request still receives 429. A GET after the cooldown receives 200. Restart the server before repeating the demo.

The server listens only on your machine. It sends an empty response body so you can focus on the status and header. In a collection job, checkpoint the last successful record separately so pausing does not discard work you already saved.

from http.server import BaseHTTPRequestHandler, HTTPServer
from math import ceil
from time import monotonic

class Handler(BaseHTTPRequestHandler):
    ready_at = None

    def do_GET(self):
        now = monotonic()
        if Handler.ready_at is None:
            Handler.ready_at = now + 1
        remaining = Handler.ready_at - now
        self.send_response(429 if remaining > 0 else 200)
        if remaining > 0:
            self.send_header("Retry-After", str(ceil(remaining)))
        self.end_headers()

with HTTPServer(("127.0.0.1", 0), Handler) as server:
    print(server.server_port, flush=True)
    server.serve_forever()

Wait and retry one read-only request

Save the client as client.py. In a second terminal, run python3 client.py followed by the port printed by the server. The first request prints 429. The client reads the header, waits and makes one more GET, which prints 200 in this demo.

The demo accepts seconds and the standard GMT date format shown above. It has a five-second wait budget. The client pauses when the wait is missing, unreadable or longer than that budget. It also stops after the second request, whatever status comes back. The server prints each incoming request so you can see whether the client sent an extra one.

from contextlib import closing
from datetime import datetime, timezone
from email.utils import format_datetime, parsedate_to_datetime
from http.client import HTTPConnection
from sys import argv
from time import sleep

def get(port):
    with closing(HTTPConnection("127.0.0.1", port, timeout=3)) as conn:
        conn.request("GET", "/data")
        with conn.getresponse() as response:
            return response.status, response.getheader("Retry-After")

def wait_seconds(header):
    value = (header or "").strip()
    if value.isascii() and value.isdecimal():
        return float(value)
    try:
        deadline = parsedate_to_datetime(value)
        if format_datetime(deadline, usegmt=True) != value:
            return None
        return max(0, (deadline - datetime.now(timezone.utc)).total_seconds())
    except (ValueError, OverflowError):
        return None

port = int(argv[1])
status, header = get(port)
print(status)
if status == 429:
    delay = wait_seconds(header)
    if delay is None or delay > 5:
        print("Paused. Check the retry policy or schedule a later attempt.")
    else:
        sleep(delay)
        print(get(port)[0])

Change the schedule when 429 keeps coming back

One retry can work after a brief burst. Repeated 429s call for a change to the schedule. Check the service's published rate and concurrency limits against the traffic from all workers, including their retries. Six workers using the same API key can exceed a shared quota even when each worker sends requests at a steady pace.

Suppose three workers are fetching product records with the same key and one receives a 429 with Retry-After of 30. Pause that key's queue for the 30 seconds if the service applies the quota to the key. Resume with fewer workers and a lower request rate. If the next attempt still gets a 429, keep the unfinished records queued and review the documented limit before the next run.

When a service allows retries but supplies no wait, use capped backoff and add a small random delay so workers do not all wake together. Set an attempt limit and a time limit for the job. Check the provider's rules before retrying a request that creates or changes anything.

Look for jobs that overlap. An hourly import may still be running when the next one starts, doubling the workers on the same quota. Compare the timestamps of the first 429s with job starts and the total number of requests in the relevant window. Include retries in that count.

A quota may count requests by key, account, endpoint or another scope the service defines. Keep the existing route while you diagnose it. When the permitted rate cannot handle the workload, spread the job out or ask the service about a suitable quota or bulk endpoint.