The rotation interval you choose determines whether your pipeline runs clean or collapses quietly behind a wall of 403s. Set it too aggressively, and you'll break sessions, drop cookies, and get flagged for erratic behavior. Hold an IP too long and rate limits kick in before you've collected half your data.
This guide is for Python developers running collection pipelines that mix stateless and stateful work. It covers how to configure both modes, when to use each, and how to run them side by side in the same job.
The tools you reach for depend on what the target requires, not what's easiest to set up.
The requests library handles basic HTTP and is fine for targets that don't inspect TLS fingerprints. For anything more protected – e-commerce sites, search engines, platforms with aggressive bot detection – curl_cffi is the better choice. It impersonates a real browser's TLS fingerprint at the connection level, which removes one of the most common reasons a clean residential IP still gets flagged. The async alternative is httpx, which works well for concurrent pipelines but doesn't offer TLS impersonation.
For JavaScript-rendered pages or stateful flows that need real browser behavior, Playwright is the right tool. Puppeteer is the Node.js equivalent. For HTML parsing, BeautifulSoup with lxml handles most extraction cleanly.
The component this guide actually depends on is a rotating residential proxy with session ID support: a provider that lets you control rotation mode through the proxy username string, switching between per-request and sticky modes by changing a single parameter.
The session ID is just a string appended to your proxy username: username-session-abc123, where abc123 is any value you choose. Change it and you get a new IP assignment.
Per-request rotation is the simplest mode to implement. Point every request at the same gateway endpoint and the provider handles IP selection on each call.
python
import os
import requests
PROXY_HOST = os.environ["PROXY_HOST"]
PROXY_PORT = os.environ["PROXY_PORT"]
PROXY_USER = os.environ["PROXY_USER"]
PROXY_PASS = os.environ["PROXY_PASS"]
proxy_url = f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
proxies = {
"http": proxy_url,
"https": proxy_url,
}
urls = [
"https://example.com/page/1",
"https://example.com/page/2",
"https://example.com/page/3",
]
for url in urls:
response = requests.get(url, proxies=proxies, timeout=30)
print(response.status_code, response.url)
Each call passes through a different residential IP because the gateway rotates on every new connection. Note that no session object is used, and that's deliberate: a requests.Session() reuses the same underlying TCP connection where it can, which interferes with per-request rotation. For this mode, make individual calls.
On targets that inspect TLS fingerprints, requests produces a recognizable non-browser signature. curl_cffi solves this by wrapping libcurl and impersonating a real browser's TLS handshake:
python
import os
from curl_cffi import requests as cffi_requests
proxy_url = (
f"http://{os.environ['PROXY_USER']}:{os.environ['PROXY_PASS']}"
f"@{os.environ['PROXY_HOST']}:{os.environ['PROXY_PORT']}"
)
urls = [
"https://example.com/page/1",
"https://example.com/page/2",
"https://example.com/page/3",
]
for url in urls:
response = cffi_requests.get(
url,
proxies={"http": proxy_url, "https": proxy_url},
impersonate="chrome",
timeout=30,
)
print(response.status_code)
The impersonate="chrome" parameter tells curl_cffi which browser's TLS fingerprint to use. Don't override the default headers it sets automatically – doing so breaks the fingerprint match and reintroduces exactly the detection risk you're trying to avoid.
Sticky sessions need one change: a session identifier added to the proxy username. The gateway reads it and holds the same exit IP for the duration of the session window.
This is what makes authorized access to your own accounts possible across a multi-step flow – pulling your own order history, exporting account data, monitoring your own checkout funnel. Those flows tie state to an IP, so a rotation mid-flow logs you straight back out.
Sign up for GoProxies rotating residential proxies to get your credentials and generate session parameters from the dashboard.
python
import os
import uuid
import requests
PROXY_HOST = os.environ["PROXY_HOST"]
PROXY_PORT = os.environ["PROXY_PORT"]
PROXY_USER = os.environ["PROXY_USER"]
PROXY_PASS = os.environ["PROXY_PASS"]
def make_sticky_proxies(session_id: str) -> dict:
user = f"{PROXY_USER}-sessionid-{session_id}"
proxy_url = f"http://{user}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
return {"http": proxy_url, "https": proxy_url}
session_id = uuid.uuid4().hex[:8]
proxies = make_sticky_proxies(session_id)
session = requests.Session()
session.proxies.update(proxies)
login_response = session.post("https://example.com/login", data={
"username": os.environ["ACCOUNT_USER"],
"password": os.environ["ACCOUNT_PASS"],
})
profile_response = session.get("https://example.com/profile")
print(profile_response.status_code)
uuid4 guarantees uniqueness across concurrent pipelines. Every flow that needs its own IP identity gets its own session_id. Reuse the same ID within a flow, and generate a new one when you want a fresh IP.
For pipelines that need both modes at once, assign a unique session ID to each worker that requires sticky behavior while stateless workers rotate freely.
python
import asyncio
import os
import uuid
import httpx
PROXY_HOST = os.environ["PROXY_HOST"]
PROXY_PORT = os.environ["PROXY_PORT"]
PROXY_USER = os.environ["PROXY_USER"]
PROXY_PASS = os.environ["PROXY_PASS"]
def build_proxy_url(session_id: str = None) -> str:
user = f"{PROXY_USER}-session-{session_id}" if session_id else PROXY_USER
return f"http://{user}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
async def fetch_stateless(url: str) -> int:
# A fresh client per call, so each request opens a new connection
# and the gateway assigns a new exit IP.
async with httpx.AsyncClient(proxy=build_proxy_url()) as client:
response = await client.get(url)
return response.status_code
async def fetch_sticky(session_id: str, urls: list) -> list:
proxy_url = build_proxy_url(session_id=session_id)
async with httpx.AsyncClient(proxy=proxy_url) as client:
results = []
for url in urls:
response = await client.get(url)
results.append(response.status_code)
return results
async def main():
stateless_urls = [
"https://example.com/product/1",
"https://example.com/product/2",
]
sticky_urls = ["https://example.com/account", "https://example.com/orders"]
sticky_session_id = uuid.uuid4().hex[:8]
stateless_results = await asyncio.gather(
*[fetch_stateless(url) for url in stateless_urls]
)
sticky_results = await fetch_sticky(sticky_session_id, sticky_urls)
print("Stateless results:", stateless_results)
print("Sticky results:", sticky_results)
asyncio.run(main())
The stateless fetch omits the session ID and opens its own client per call, so each request gets a fresh IP. The sticky fetch binds one session ID and reuses a single client for the whole authenticated sub-flow, keeping cookies and state consistent throughout.
There's a real trade-off in that stateless pattern: a client per request means no connection reuse, so you pay a handshake on every call. That's the cost of genuine per-request rotation. If you share one client across stateless calls, connection pooling will quietly keep you on the same exit IP.
Rotation handles IP diversity. Headers, timing, and browser behavior are your responsibility.
An IP with a high fraud score gets flagged on the first request regardless of how cleanly it rotates. That makes pool quality a ceiling on what any rotation strategy can achieve – you can't out-engineer a pre-flagged IP.
GoProxies' rotating residential proxies draw from 30 million residential IPs, so addresses enter your rotation clean rather than pre-flagged from prior misuse. See the rotating residential proxy plans for pricing and pool details.
When using sticky sessions, carry cookies explicitly across requests. A requests.Session() or httpx.AsyncClient() handles this automatically when you reuse the same client instance. In per-request mode, don't pass a cookie jar between calls unless you specifically need to simulate a returning user, since doing that while rotating IPs is a detectable signal in itself.
For JS-rendered targets, run Playwright with a persistent browser context rather than spinning up a fresh one per request. Persistent contexts retain cookies and local storage across page loads. Pass your sticky proxy through the browser launch arguments so the same IP backs the whole context.
Align headers with the IP's geography. A US residential IP sending Accept-Language: de-DE is an obvious signal. Match Accept-Language, timezone, and locale to the exit IP's country. Rotate user agents across sessions, not within them – changing mid-session is another flag.
Build retry logic that fetches a new IP on failure rather than retrying through the same one. In per-request mode, this happens automatically. For sticky sessions, catch 403 and 429 responses and generate a new session_id before retrying.
Spread concurrent workers across different session IDs so no single IP carries more than a few parallel requests. A residential IP handling ten simultaneous requests from your scraper looks nothing like genuine user traffic.
Most rotation failures come from a mismatch between rotation mode and what the target expects, not from bad proxies.
If the target tracks IP consistency across page loads – and most login-gated platforms do – switching IPs between requests triggers session invalidation or a re-authentication challenge. The fix is sticky sessions for any flow involving cookies, login state, a shopping cart, or paginated results that depend on session tokens.
A degraded pool produces inconsistent results even with a clean rotation strategy. Shared residential pools accumulate reputation damage over time, and since you can't control exactly which IP you get on any given rotation, the quality floor of the pool sets your baseline success rate.
Rotating the IP doesn't change the TLS fingerprint. If you're using the standard requests library, every request carries the same non-browser signature regardless of which IP it exits from. On targets with fingerprint-based detection, rotation provides no benefit at all. Switch to curl_cffi with a browser impersonation target set, as in Step 3.
Per-request rotation is built for volume: product listings, SERP monitoring, price tracking, and any public data that doesn't require authentication. Each hit looks like a unique visitor, keeping individual IPs well below rate limit thresholds.
Sticky sessions open up the authenticated layer. Account management, order history collection, gated content you have access to, and checkout flow monitoring all require a consistent IP identity across multiple requests. Without them, those flows fail authentication or get flagged for IP inconsistency.
The two modes aren't mutually exclusive. A well-structured pipeline uses per-request rotation for bulk collection and sticky sessions for authenticated sub-flows, assigning a unique session ID per account or logical flow. GoProxies' 30 million IP pool across 200+ locations gives you the depth to run both simultaneously at scale.
Rotating residential proxies are legal in most jurisdictions when collecting publicly accessible data. What matters is what you're collecting and whether the target's terms permit automated access.
Public-facing data – product listings, prices, search results, publicly visible reviews – is generally fair game. Data behind authentication, or explicitly marked private in a site's terms, is a different matter, and accessing it without authorization creates legal exposure under computer access laws in multiple jurisdictions. Sticky sessions against a logged-in account are appropriate when the account is yours or you have permission to access it.
Always read the target's Terms of Service before running automated collection. Respecting stated terms is both an ethical baseline and a practical one, since violations can result in account termination, IP-range blocks, or legal action.
On the ethical side, rate limiting your requests to avoid degrading site performance also reduces your detection footprint. A well-paced pipeline is both more ethical and more sustainable. If your use case is commercial and the target is high-profile, get qualified legal advice before running at scale.
A backconnect gateway gives you a single endpoint. Pass your rotation parameters through the username string and the provider handles IP selection, pool health, and rotation logic. Nothing to maintain, no health checks to run. The trade-off is less visibility into which specific IP you're getting, and rotation behavior partly determined by the provider's infrastructure.
Managing your own proxy list gives you full control: custom rotation logic, your own circuit breaker, weighted selection. The cost is significant – continuous health checks, IP replacement, and every piece of retry and rotation logic built and scaled by you.
For most workloads, a backconnect gateway is the right default. A self-managed list only makes sense when you have requirements a gateway can't meet: fixed IP ranges for allowlisting, custom rotation triggers, or direct IP-level integration with internal infrastructure.
Free proxy lists exist and they do rotate, but the experience is reliably poor for anything beyond casual testing.
The fundamental problem is pool quality. Free lists aggregate IPs from unknown sources, many already flagged by the time you use them. Success rates on any target with basic bot protection are low. And for sticky sessions specifically, most free proxies don't support session IDs at all, which makes that mode impossible to implement correctly.
Reliability is the other issue. Free IPs go offline without notice, and there's no support channel, no SLA, and no pool replenishment. A sticky session bound to an IP that disappears mid-flow means your pipeline stalls with no recourse.
GoProxies offers pay-as-you-go rotating residential proxies with no minimums or contracts, so you pay for the bandwidth you actually use.
Per-request rotation is the right default for stateless, high-volume collection. Sticky sessions are required for any flow involving authentication, cookies, or multi-step navigation. Using the wrong mode is the most common cause of mysterious session failures and unexpected blocks.
Treat rotation mode as a per-flow decision rather than a global setting. Match the mode to what the flow requires, align headers and cookies accordingly, use curl_cffi on targets that inspect TLS fingerprints, and build retry logic that generates fresh session IDs on failure rather than retrying through the same IP.
Gintarė specializes in content related to proxies, web scraping, data collection, and internet infrastructure. With a solid background in information technology, networking, and cybersecurity, she understands both the technical and practical aspects of large-scale data acquisition.
The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.
A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!
Headings, paragraphs, blockquotes, figures, images, and figure captions can all be styled after a class is added to the rich text element using the "When inside of" nested selector system.
Per-request rotation assigns a new IP for every HTTP request your script sends, maximizing IP diversity across a large run. Per-session rotation holds one IP for a defined window, typically 1 to 30 minutes, so cookies and login state stay consistent across multiple requests in the same flow
Long enough to complete the workflow it covers. For most login and navigation flows, 10 to 15 minutes is sufficient. For longer workflows like multi-page form submissions, 20 to 30 minutes is reasonable. Avoid holding a session longer than necessary, since longer windows increase the risk of a single IP accumulating enough requests to trigger rate limiting.
Yes. Assign per-request rotation to workers handling stateless bulk collection and unique session IDs to workers handling authenticated or stateful flows. The two modes operate independently through the same gateway endpoint. The only difference is whether a session ID parameter appears in the proxy username string.
Yes, residential proxies cost more per GB. Their IPs are sourced from real user devices, which makes them significantly harder to detect on protected targets. Datacenter proxies are cheaper and work well on lower-sensitivity targets, but get flagged more often on e-commerce sites, search engines, and platforms with active bot detection. The cost difference is usually justified when success rate matters.
