Scrapy is a fast, extensible Python scraping framework and one of the easiest tools to get blocked with. Its default request fingerprint is recognizable, and a single IP sending hundreds of requests will hit rate limits or be banned fast.
Rotating residential proxies fix this by cycling your requests through IPs tied to real home internet connections. To any target site, each request looks like a different person browsing from their couch.
This guide walks Python developers and data engineers through two integration methods, blocking evasion, the errors you'll actually hit, and the legal basics.
Scrapy is the foundation: a Python framework that handles request scheduling, response parsing, and data pipelines. Install it with pip install Scrapy.
A residential proxy provider gives you either a single rotating gateway endpoint or a list of individual proxy addresses with credentials. Residential IPs are assigned to real households by ISPs, which makes them much harder to detect than datacenter IPs on well-protected targets. Datacenter proxies are cheaper but get flagged more often on sites with active bot detection.
The scrapy-rotating-proxies library (pip install scrapy-rotating-proxies) is useful if your provider supplies individual addresses. It handles health monitoring and removes dead proxies automatically. If you're using a gateway endpoint, you don't need it.
For user agent rotation, pip install scrapy-user-agents or fake-useragent.
For JavaScript-rendered targets, pip install scrapy-playwright, then playwright install chromium. Note that the install alone doesn't do anything: scrapy-playwright needs DOWNLOAD_HANDLERS entries and an asyncio reactor set in settings.py, so follow the project's setup docs before your first run. Use it only when a plain HTTP request doesn't return the content you need.
bash
pip install scrapy
scrapy startproject myproject
cd myproject
This scaffolds the standard structure: settings.py, middlewares.py, pipelines.py, and a spiders directory. All proxy configuration goes into settings.py and middlewares.py.
Sign up at GoProxies and open the rotating residential proxies section of the dashboard. You'll get four things: a gateway hostname, a port, a username, and a password. The dashboard is also where you set your geo targeting and generate the session parameters you'll need later if any of your flows require a sticky IP.
Keep all four out of source control. The next step loads them from environment variables.
With a gateway provider, all IP rotation happens on the provider's side. Add a custom middleware to middlewares.py:
python
import os
class ResidentialProxyMiddleware:
def process_request(self, request, spider):
proxy_user = os.environ.get("PROXY_USER")
proxy_pass = os.environ.get("PROXY_PASS")
proxy_host = os.environ.get("PROXY_HOST")
proxy_port = os.environ.get("PROXY_PORT")
proxy_url = f"http://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port}"
request.meta["proxy"] = proxy_url
Then register it in settings.py. You'll add the user agent middleware to this same dict in Step 5, so treat the block in Step 5 as the final version:
python
DOWNLOADER_MIDDLEWARES = {
"myproject.middlewares.ResidentialProxyMiddleware": 350,
}
If your provider gives you individual proxy addresses, add the list and enable both middlewares in settings.py:
python
ROTATING_PROXY_LIST = [
"http://user:pass@proxy1.example.com:8000",
"http://user:pass@proxy2.example.com:8000",
"http://user:pass@proxy3.example.com:8000",
]
DOWNLOADER_MIDDLEWARES = {
"rotating_proxies.middlewares.RotatingProxyMiddleware": 610,
"rotating_proxies.middlewares.BanDetectionMiddleware": 620,
}
BanDetectionMiddleware marks a proxy dead on any non-200 response or empty body. That default is aggressive, and it's worth understanding before you run it: a target that returns a 302 redirect or a non-200 on a perfectly legitimate response will chew through your pool in minutes. This is the direct cause of the "spider closes because all proxies are dead" failure covered further down, so extend the policy to match your target's actual behavior rather than accepting the default.
You can also extend it in the other direction, to catch site-specific signals like a 200 response containing a CAPTCHA body.
To load proxies from a file instead of the settings dict:
python
ROTATING_PROXY_LIST_PATH = "/path/to/proxies.txt"
Rotating IPs while keeping a fixed user agent is a common oversight – the pattern stays detectable. This is the complete DOWNLOADER_MIDDLEWARES block, replacing the one from Step 3:
python
DOWNLOADER_MIDDLEWARES = {
"myproject.middlewares.ResidentialProxyMiddleware": 350,
"scrapy.downloadermiddlewares.useragent.UserAgentMiddleware": None,
"scrapy_user_agents.middlewares.RandomUserAgentMiddleware": 400,
}
The built-in UserAgentMiddleware has to be disabled so the random one can take over. If you leave both enabled, you'll get inconsistent results depending on priority order.
Create a quick test spider to confirm the proxy is working:
python
import scrapy
class ProxyTestSpider(scrapy.Spider):
name = "proxy_test"
start_urls = ["https://httpbin.org/ip"]
def parse(self, response):
self.logger.info(f"Current IP: {response.json()['origin']}")
Run scrapy crawl proxy_test. The IP in the log should match your proxy pool, not your own machine. Run it several times and confirm the IP changes.
Rotation covers one signal out of many. Here's what else to cover.
IPs with poor reputation or heavy scraping history get flagged fast regardless of your other settings. Residential proxies sourced from real ISP connections read as organic traffic and hold up far better on protected targets.
GoProxies' rotating residential proxies give you a 30 million IP pool with city-level targeting, which is the depth you need to scrape at scale without burning through reputation.
Scrapy's CookiesMiddleware is enabled by default and manages cookie state within a session. For multi-step workflows that need the same IP across a sequence of requests, use your provider's sticky session support. Rotating IPs mid-session can trigger session anomaly detection on stricter targets.
Scrapy makes plain HTTP requests and doesn't execute JavaScript. For targets with dynamic content or JS-based bot challenges, scrapy-playwright integrates a full browser engine into your Scrapy pipeline. It adds significant overhead, so reach for it only when a static request doesn't return what you need.
Replace Scrapy's default user agent string and send a full browser header set including Accept-Language, Accept-Encoding, and Referer. Match Accept-Language to the geographic region of your proxy IP, since a mismatch is a detectable inconsistency.
Use AutoThrottle to vary request timing dynamically based on server latency:
python
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1
AUTOTHROTTLE_MAX_DELAY = 30
AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0
Avoid fixed DOWNLOAD_DELAY values. Even spacing is itself a bot signal.
Scrapy's RetryMiddleware is enabled by default. Configure it explicitly for scraping:
python
RETRY_ENABLED = True
RETRY_TIMES = 5
RETRY_HTTP_CODES = [403, 408, 429, 500, 502, 503, 504]
With rotation active, each retry goes out from a different IP, so a blocked IP doesn't permanently stall a request. Note that 403 isn't in Scrapy's default retry list – adding it only helps if your rotation is genuinely working, otherwise you're retrying the same block five times.
High concurrency amplifies every detection signal. Reduce it when scraping protected targets:
python
CONCURRENT_REQUESTS = 8
CONCURRENT_REQUESTS_PER_DOMAIN = 2
Scrapy's default fingerprint – user agent string, Twisted TLS stack, and header ordering – is well documented by anti-bot vendors. Here are the detection vectors that matter and how to deal with them.
Scrapy's TLS fingerprint (cipher suites, extension order) differs from real browser traffic. Sites with TLS-based detection will catch standard Scrapy requests even behind a realistic user agent. Use scrapy-impersonate or scrapy-playwright to send a genuine browser TLS stack – both require their own DOWNLOAD_HANDLERS configuration, so check each project's docs.
This is where Scrapy differs from browser-based tools. A browser sends a specific set of headers in a specific order, and Scrapy doesn't reproduce either by default. A Chrome user agent arriving without Sec-Ch-Ua or Sec-Fetch-Dest fails consistency checks on sophisticated targets, and header order is checked too.
So when a CAPTCHA appears at the start of a session, work through the causes in this order: incomplete or wrongly ordered headers, then an Accept-Language value inconsistent with the IP's region, then the IP's own fraud score. Set DEFAULT_REQUEST_HEADERS in settings.py to a complete, browser-accurate set before you assume the proxy is the problem.
Residential IPs used heavily for scraping accumulate history that anti-bot systems track. A large, actively refreshed pool reduces the chance of cycling into a flagged IP. If clean-looking IPs are still getting blocked, check your provider's fraud score guarantees.
Scrapy paired with rotating residential proxies suits data-heavy collection at scale. The common use cases:
Ready to collect at scale without fighting blocks? GoProxies offers a 30 million IP residential pool with city-level targeting and 99.99% uptime.
Legality depends on what you scrape and how, not on the tools. A few principles:
Public data that's accessible without logging in or circumventing access controls is generally fair game. Courts in several jurisdictions have found that scraping publicly available data doesn't constitute unauthorized access, though the specifics vary by country.
Terms of Service often prohibit automated access. Violating a ToS is typically a civil matter rather than a criminal one, but it can result in access termination or liability. Check before you scrape.
Scrapy respects robots.txt when ROBOTSTXT_OBEY is True, which is the value set in a freshly generated project. Ignoring it isn't illegal in most jurisdictions, but it raises your ToS risk and is poor practice.
Request rates matter ethically. Keep them reasonable, use AutoThrottle, and don't send volume that degrades anyone's service. If your use case is commercial and the target is high-profile, get qualified legal advice before running at scale.
Scrapy's HttpCacheMiddleware stores responses locally and serves them from disk on repeat requests. It isn't a replacement for live scraping, but it's genuinely useful during development – it stops you hitting a live target every time you adjust your spider.
Enable it while developing:
python
HTTPCACHE_ENABLED = True
HTTPCACHE_EXPIRATION_SECS = 0
HTTPCACHE_DIR = "httpcache"
Set HTTPCACHE_ENABLED = False in production. Cached responses don't reflect current page state, and they do nothing about IP bans or anti-bot detection on live runs.
Free proxy lists exist, but they rarely work well enough to be worth the effort. The core problem is IP reputation: free proxies are shared with no controls on use, so by the time you pull an IP from a public list it's almost certainly been flagged for scraping or spam already. Most well-protected sites have those ranges blocklisted before your first request lands.
They're also unreliable. Free proxies go offline without notice, respond slowly, and fail often. Running a Scrapy spider against them means spending more time managing dead proxies than collecting data.
There's a security angle too: free proxy operators can inspect your traffic, which matters if your workflow touches any authenticated session.
For production, a paid residential pool pays for itself. GoProxies' rotating residential proxies are pay-as-you-go with no minimums – you pay for what you use.
Setting up rotating residential proxies with Scrapy is an intermediate-level task. You need to be comfortable with Python, Scrapy's settings system, and basic middleware concepts. The gateway approach is simpler and covers most use cases. List-based rotation with scrapy-rotating-proxies gives you more control but requires tuning the ban-detection policy for your specific target.
The practices that matter: rotate user agents alongside IPs, send realistic browser headers in a realistic order, use AutoThrottle rather than fixed delays, and reach for scrapy-playwright only when you genuinely need JavaScript rendering. Test your proxy setup against an IP-check endpoint before pointing the spider at a real target.
Gintarė specializes in content related to proxies, web scraping, data collection, and internet infrastructure. With a solid background in information technology, networking, and cybersecurity, she understands both the technical and practical aspects of large-scale data acquisition.
The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.
A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!
Headings, paragraphs, blockquotes, figures, images, and figure captions can all be styled after a class is added to the rich text element using the "When inside of" nested selector system.
Yes. Scrapy's built-in HttpProxyMiddleware handles basic proxy routing via request.meta['proxy']. For production use with rotation, you need either custom middleware or a dedicated library, since the built-in handler does no rotation and no health monitoring.
Gateway rotation routes all requests through one endpoint that handles IP switching on the provider's side. List-based rotation cycles through individual proxy addresses locally, using middleware to monitor health and retire dead proxies. Gateway is simpler; list-based gives you more visibility and control over per-proxy behavior.
No. They reduce blocks substantially but don't eliminate them. A complete setup also needs realistic user agents, consistent browser headers, sensible request timing, and sometimes a real browser engine via scrapy-playwright. Rotation covers the IP; fingerprinting and behavioral detection need separate treatment.
Yes, using Scrapy with proxies is legal. The legality question applies to what you scrape and how, not to the tools. Scraping publicly accessible data is generally permitted, but check a site's Terms of Service and robots.txt before running at scale.
