Running a scraper from a single IP is a one-way ticket to getting blocked. Anti-bot systems track request volume, timing patterns, and geographic consistency, and the moment your IP crosses a threshold, it gets flagged.
Rotating residential proxies fix that by distributing traffic across a pool of real consumer IPs, so your scraper looks like ordinary user traffic instead of one very busy machine.
This guide is for Python developers and data engineers who need their scraping to hold up under pressure. You'll build a working rotation system with curl_cffi and Playwright, learn how to avoid detection, and handle failures without burning your pool.
The first decision is your HTTP client. curl_cffi impersonates browser TLS and HTTP/2 fingerprints at the connection layer, so the server sees traffic that looks like it came from a real Chrome or Safari session. Install it with pip install curl-cffi.
If you're scraping pages that require JavaScript execution, install Playwright too with pip install playwright, then run playwright install to pull the browser binaries.
For the rotation logic itself, you don't need anything extra. Python's built-in itertools and random modules handle sequential and random selection, and time covers interval-based rotation.
The standard requests library isn't the right tool for protected targets. It produces a recognizable non-browser TLS fingerprint that anti-bot systems flag more readily than curl_cffi.
bash
pip install curl-cffi
pip install playwright
playwright install chromium
Python 3.10 or higher is required for curl_cffi. [CONFIRM: version floor and the curl_cffi release it applies to]
Your proxy pool is a list of authenticated proxy URLs in the format http://username:password@host:port. Load the credentials from the environment rather than committing them to source:
python
import os
PROXY_USER = os.environ["PROXY_USER"]
PROXY_PASS = os.environ["PROXY_PASS"]
PROXY_HOSTS = [
"proxy1.host:PORT",
"proxy2.host:PORT",
"proxy3.host:PORT",
]
PROXIES = [
f"http://{PROXY_USER}:{PROXY_PASS}@{host}"
for host in PROXY_HOSTS
]
If you don't have a pool yet, GoProxies gives you access to 30 million residential IPs with pay-as-you-go pricing and no minimum commitment, so you can start small and scale without red tape. [CONFIRM: pool size – four articles say 30M, the rotation-interval article says 80M]
The simplest and most effective strategy is a different proxy on every request. curl_cffi accepts a proxy argument directly on each call, so you don't need a session object to swap between requests:
python
import random
from curl_cffi import requests
urls = [
"https://example.com/page/1",
"https://example.com/page/2",
"https://example.com/page/3",
]
for url in urls:
proxy = random.choice(PROXIES)
response = requests.get(
url,
proxy=proxy,
impersonate="chrome",
timeout=30,
)
print(response.status_code, proxy)
Setting impersonate="chrome" tells curl_cffi to match the TLS fingerprint of the latest supported Chrome release automatically.
One caveat on random.choice: it gives you no distribution guarantee, so the same proxy can come up several times in a row. If even distribution matters, use itertools.cycle for round-robin selection instead, as in the next step.
Some targets treat a rapid string of requests arriving from different IPs as its own signal. Holding one IP for a fixed window before switching produces a traffic pattern closer to a real browsing session, which is often the better trade on session-aware targets.
Here's an interval approach using itertools.cycle, which loops through your list indefinitely without manual index management:
python
import time
import itertools
from curl_cffi import requests
ROTATION_INTERVAL = 10 # seconds
proxy_cycle = itertools.cycle(PROXIES)
current_proxy = next(proxy_cycle)
last_rotation = time.time()
urls = [f"https://example.com/page/{i}" for i in range(20)]
for url in urls:
if time.time() - last_rotation >= ROTATION_INTERVAL:
current_proxy = next(proxy_cycle)
last_rotation = time.time()
response = requests.get(
url,
proxy=current_proxy,
impersonate="chrome",
timeout=30,
)
print(response.status_code)
Note the exact semantics: the check runs only when a request is about to go out, so this rotates on the first request after the interval elapses rather than on a fixed clock. Keep the interval short relative to how long your run takes, or you'll finish the whole job on one IP.
When the target requires JavaScript rendering, curl_cffi alone isn't enough. In Playwright, proxies can't be changed on an existing context, so rotation means creating a new context per proxy. Each context is fully isolated by cookies, storage, and IP:
python
import asyncio
import random
from playwright.async_api import async_playwright
PROXIES = [
{"server": "http://proxy1.host:PORT", "username": "user", "password": "pass"},
{"server": "http://proxy2.host:PORT", "username": "user", "password": "pass"},
{"server": "http://proxy3.host:PORT", "username": "user", "password": "pass"},
]
URLS = [
"https://example.com/page/1",
"https://example.com/page/2",
"https://example.com/page/3",
]
async def scrape_with_context(browser, url, proxy):
context = await browser.new_context(proxy=proxy)
page = await context.new_page()
await page.goto(url, timeout=30000)
content = await page.content()
await context.close()
return content
async def main():
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True)
for url in URLS:
proxy = random.choice(PROXIES)
content = await scrape_with_context(browser, url, proxy)
print(url, len(content))
await browser.close()
asyncio.run(main())
Always close each context after use. Open contexts accumulate memory quickly across a long URL list.
Rotation solves the IP layer. Everything else is still your responsibility.
Anti-bot systems expect cookies and IP to stay consistent within a session. A session cookie set on one IP turning up on a request from a different IP is a clear mismatch signal. In curl_cffi, use a separate Session object per proxy. In Playwright, each browser context handles this automatically with isolated storage.
When a page loads its data via JavaScript after the initial HTML, curl_cffi returns an empty shell. Switch to Playwright for those targets. A Playwright context uses considerably more memory than a curl_cffi request, so default to curl_cffi and escalate only once you've confirmed the target needs rendering.
A US residential IP sending Accept-Language: zh-CN is a mismatch that detection systems catch. Set Accept-Language to reflect the proxy's country. curl_cffi's impersonate parameter handles most header consistency automatically, but check that your Accept-Language is sensible for each proxy's location.
Pair each proxy with a user agent for the duration of a session and rotate them together. Changing the user agent mid-session while holding the same IP creates a detectable inconsistency. Assign one realistic, current user agent per proxy at session start.
Real users don't send requests at perfectly even intervals. Add random delays with time.sleep(random.uniform(1.5, 4.0)) between requests. For well-protected targets, widen the range.
For stateless pages like listings or search results, rotate per request. For multi-step flows – logins, checkouts, paginated sessions tied to cookies – keep the same proxy for the whole session and rotate only when it ends.
When parallelizing with asyncio, distribute requests across your pool so each proxy handles a limited number of concurrent connections. A good starting rule is one active request per proxy. The reasoning is simple: a residential IP handling ten simultaneous requests looks nothing like a person on a laptop, and that pattern is easier to spot than volume alone.
GoProxies rotating residential proxies are built for this kind of distributed workload. With 30 million residential IPs, 99.99% uptime, and country, state, and city-level targeting, you can size your pool to your concurrency needs. Check the residential proxy pricing to find a plan that fits your volume. [CONFIRM: uptime figure, targeting granularity]
Rotation addresses IP-based detection. Modern anti-bot systems layer several methods on top of it.
Every HTTP client produces a TLS fingerprint during the connection handshake. The requests library produces one that doesn't match any real browser, and anti-bot systems recognize it on sight.
curl_cffi solves this by generating fingerprints that match real browser sessions. If you're still seeing blocks after rotating proxies, check whether something in your setup is overriding curl_cffi's automatic headers and breaking the fingerprint.
Anti-bot systems track how you behave: request timing, scroll patterns, resource load sequence, navigation speed. Rotating proxies changes your apparent identity but not your behavior. Randomize request intervals, avoid linear navigation patterns, and for targets that instrument JavaScript-level behavior, use Playwright rather than HTTP requests alone.
Not all residential IPs are equally clean. An IP previously used for spam or credential stuffing can carry a high fraud score and get challenged on first use.
If you're hitting CAPTCHAs on the very first request through a fresh proxy, it's usually one of three things, in this order of likelihood: a high-fraud-score IP, a TLS fingerprint mismatch, or Accept-Language headers that don't match the proxy's geo.
A 429 doesn't mean the IP is permanently banned. Back off, wait, and retry with the same IP or a fresh one. Give rate-limited proxies a cooldown before reintroducing them to rotation rather than discarding them outright.
Stable rotation is what separates a one-off script from a production data pipeline. With it in place, you can collect price data across multiple regions simultaneously, pull localized search results from different countries, and monitor e-commerce listings or stock levels continuously without IP bans breaking the feed.
Geo-restricted content becomes reachable by routing requests through IPs in the target region. Competitive intelligence – pricing, assortment changes, promotional activity – needs the kind of consistent access that rotation enables. Review data, ratings, and user-generated content from platforms that rate-limit heavily also become viable once your pool is large enough to spread the load.
If you'd rather skip the infrastructure overhead, GoProxies gives you the residential pool and support to run these pipelines reliably from day one.
The legality of scraping depends on what you're collecting and what you do with it, not on whether you use proxies. Proxies are a legitimate networking tool used across privacy, security, and content delivery. For publicly accessible data, scraping is generally permissible in many jurisdictions, though case law continues to develop and the picture varies by country.
Every platform's Terms of Service restricts automated access to some degree. Violating a ToS isn't itself illegal in most jurisdictions, but it can expose you to civil action or account termination.
The ethical baseline is to scrape only public data, avoid personal information without a lawful basis, and respect robots.txt. Rate limiting your scraper protects the service for legitimate users, not just your IPs. If your use case is commercial and the target is high-profile, get qualified legal advice before running at scale.
Some providers offer a gateway endpoint that handles rotation server-side. You point your client at a single endpoint and the provider assigns a different residential IP to each request automatically, removing the need to manage a pool or write retry logic yourself.
The trade-off is less control over which IP or geo you get, and dependency on the provider's pool health. Gateway rate limits can also bottleneck at very high volumes.
Manual rotation is better when you need precise geo control, need to debug specific IPs, or require sticky sessions that outlast a single request. A gateway endpoint is better when you want simple code and are happy to delegate pool management entirely.
Free proxy lists rotate IPs, but every IP on a free list has already been used by whoever published it and everyone who downloaded it before you. By the time you make your first request, there's a good chance it's already flagged.
Free proxies are also slow, unstable, and short-lived. They drop connections and produce SSL errors from misconfigured servers. For learning rotation logic, they're fine. For production, they aren't reliable enough – the engineering time spent debugging free proxies and re-running failed jobs usually costs more than a paid plan.
GoProxies residential proxy plans start pay-as-you-go with no minimum commitment, so you can start at low volume and scale as your pipeline grows.
Rotating residential proxies in Python is straightforward once you have the right tools. curl_cffi handles TLS fingerprinting natively and is the right client for most jobs. Playwright is the right escalation for JavaScript-rendered pages.
Pair each proxy with its own session, match headers to its geo, add randomized delays, and build retry logic that reads error codes as signals rather than reasons to burn proxies.
Pool quality is the foundation. Clean residential IPs that haven't already been flagged are what make the rotation logic work at all.
GoProxies Research Team is a group of analysts and writers focused on web data, online privacy, and proxy technology. They produce practical, research-backed content that helps readers understand digital trends, tools, and best practices.
The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.
A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!
Headings, paragraphs, blockquotes, figures, images, and figure captions can all be styled after a class is added to the rich text element using the "When inside of" nested selector system.
For stateless pages like product listings or search results, rotating on every request is safest. For multi-step flows that need session continuity, keep the same proxy for the whole session and rotate when the session ends.
Sticky sessions keep the same IP for the duration of a logical session, which is necessary when the target ties state to your IP. Rotating proxies assign a different IP to each request or session. Most production scrapers use both: sticky within a session, rotating between sessions.
curl_cffi impersonates real browser TLS fingerprints at the connection layer, while the standard requests library produces a non-browser fingerprint that anti-bot systems flag more readily. On protected targets with TLS-based detection, curl_cffi reduces friction significantly.
Yes. Create a new browser context for each proxy. Proxies can't be changed on an existing context, so rotation requires a fresh context per IP. Each context is fully isolated by cookies, storage, and network identity.
