Amazon is the richest source of product data on the internet. Pricing intelligence, competitor monitoring, review sentiment analysis, inventory tracking. It all flows through Amazon's product pages, and businesses that can collect it reliably have a structural advantage.
Amazon is also the hardest mainstream target to scrape. Its defense stack gets more aggressive every year, and most scrapers get blocked within hours, not because of bad code, but because of the wrong infrastructure.
This guide covers the full setup for developers, e-commerce operators, and data teams: tools, working code, anti-detection techniques, and the proxy infrastructure that makes the difference.
Amazon scraping requires a layered toolkit. Each component solves a different problem, and skipping any one of them is usually what causes blocks.
For static product pages, curl_cffi is the right starting point. Unlike Python's standard requests library, curl_cffi impersonates real browser TLS and HTTP/2 fingerprints, including the correct headers for the browser version you specify.
When you set impersonate="chrome", it handles all of that automatically and Amazon sees a fingerprint identical to a real Chrome browser rather than a Python script.
You can install it with:
///
pip install curl_cffi --upgrade
///
For pages that load prices or availability via JavaScript, you also need a headless browser. Playwright is the current standard, compatible with playwright-stealth, which patches the automation signals headless browsers expose by default.
BeautifulSoup works well for Amazon product pages once you have clean HTML. For higher throughput, lxml is faster.
If you are building on Scrapy, its built-in Parsel parser handles CSS selectors and XPath natively. For large-scale crawls across thousands of ASINs, Scrapy also provides built-in middleware for retries, rate limiting, and proxy rotation.
This is where most operations fail. Amazon pre-flags entire datacenter ASN ranges, so requests from known data center IPs arrive pre-scrutinized before your headers or behavior are even evaluated. You need residential proxies: IPs assigned by real ISPs to real devices.
GoProxies provides a pool of 80 million IPs, 30 million of which are residential, with the lowest fraud scores in the market, which is exactly what Amazon's WAF reputation scoring checks for. Pay-as-you-go pricing means no upfront commitment.
The steps below walk through a complete working setup, from constructing target URLs to exporting data. Each step builds on the previous one.
Amazon product pages follow a predictable structure based on the ASIN (Amazon Standard Identification Number). Constructing direct product URLs is cleaner than crawling search results and avoids the more heavily monitored search result pages.
///
base_url = "https://www.amazon.com/dp/{asin}"
asins = ["B09XYZ1234", "B08ABC5678", "B07DEF9012"] urls = [base_url.format(asin=asin) for asin in asins]
///
Install the core dependencies:
///
pip install curl_cffi beautifulsoup4 lxml
///
Create a dedicated virtual environment to keep dependencies isolated, especially if you are running multiple scrapers side by side.
curl_cffi accepts a proxy via the proxy parameter. Setting impersonate="chrome" matches the latest Chrome TLS fingerprint and headers automatically. Paired with a rotating residential proxy from GoProxies, each request arrives from a different ISP-assigned IP with a clean browser fingerprint.
///
from curl_cffi import requests
response = requests.get( "https://www.amazon.com/dp/B09XYZ1234", proxy="http://user:pass@proxy.goproxies.com:port", impersonate="chrome" )
///
Note: using impersonate="chrome" without a pinned version keeps your fingerprint up to date as curl_cffi releases new browser targets.
See GoProxies rotating residential proxy plans. Pay-as-you-go, no credit card required.
Once you have clean HTML, extract the fields you need using CSS selectors. Always add null checks, as Amazon's page structure varies by product type and region.
The example below is adapted from the amzpy open-source Amazon scraper, which uses curl_cffi under the hood.
///
from bs4 import BeautifulSoup
soup = BeautifulSoup(response.text, "lxml")
title = soup.select_one("#productTitle") price = soup.select_one(".a-price .a-offscreen") rating = soup.select_one("span[data-hook='rating-out-of-text']") review_count = soup.select_one("#acrCustomerReviewText") availability = soup.select_one("#availability span")
product = { "title": title.get_text(strip=True) if title else None, "price": price.get_text(strip=True) if price else None, "rating": rating.get_text(strip=True) if rating else None, "reviews": review_count.get_text(strip=True) if review_count else None, "availability": availability.get_text(strip=True) if availability else None, }
///
Write to newline-delimited JSON for easy streaming and incremental processing. Track which ASINs have already been scraped to avoid duplicate requests on large runs.
///
import json
with open("products.json", "a") as f: json.dump(product, f) f.write("\n")
///
Getting past Amazon's defenses requires more than one fix. The sections below cover the key layers, each of which needs to be in place for consistent results.
The most common reason scrapers fail on Amazon is proxy quality. Amazon's WAF checks IP reputation at the ASN level, and large datacenter IP ranges from well-known providers are flagged.
Requests from those IPs arrive pre-scrutinized regardless of how clean your fingerprint looks.
Fraud score matters too. IPs with history in spam or credential stuffing carry reputation debt that follows them.
GoProxies maintains the lowest fraud scores in the market, which translates directly into higher success rates on Amazon. Check rotating residential proxy pricing here.
When scraping JavaScript-rendered pages, use Playwright with playwright-stealth to patch the signals headless browsers expose, primarily navigator.webdriver, WebGL renderer strings, and canvas fingerprints. The pattern below follows the approach in the luminati-io/curl_cffi-web-scraping reference implementation.
///
from playwright.sync_api import sync_playwright from playwright_stealth import stealth_sync
with sync_playwright() as p: browser = p.chromium.launch(headless=True) page = browser.new_page() stealth_sync(page) page.goto("https://www.amazon.com/dp/B09XYZ1234")
///
Uniform request intervals are a reliable bot signal. Randomize your delays and check for soft blocks before parsing. These are pages that return 200 but contain a robot check:
///
import time, random
def human_delay(): time.sleep(random.uniform(2.0, 5.5))
if "robot check" in response.text.lower() or "captcha" in response.text.lower(): human_delay() # rotate proxy and retry
///
Keep concurrent requests between 3 and 10 per proxy pool. Aggressive parallelization is one of the fastest ways to trigger rate-based blocks.
Amazon's defense stack is more sophisticated than most enterprise applications. Understanding each layer makes every configuration decision easier.
Amazon runs AWS WAF in front of its product pages, evaluating requests against IP reputation databases and continuously updated rule sets.
Requests from known datacenter ASNs are flagged at the network level before your headers or behavior are even evaluated, which is why proxy selection matters more than anything else.
When your client initiates an HTTPS connection, it sends a Client Hello message before encryption begins. The cipher suites, TLS extensions, and elliptic curves it contains form a fingerprint unique to each library.
Python's requests has a measurably different fingerprint from real browsers, and Amazon uses JA3/JA4 hashing to act on this. curl_cffi with impersonate="chrome" resolves it.
Amazon checks over 100 browser environment signals: User-Agent, screen resolution, installed fonts, WebGL renderer, canvas hash, and navigator.webdriver, a boolean that is true in any unpatched headless browser. playwright-stealth patches these before your session starts.
Amazon analyzes mouse movement trajectories, scroll velocity, and click timing. Human movements have natural curves and micro-jitters; scripted movements follow straight paths at constant velocity. For Playwright scrapers, randomize scroll depth, add pauses, and avoid perfectly linear navigation sequences.
When other detection layers flag a session, Amazon escalates to CAPTCHA, ranging from silent JavaScript environment checks to full image-based puzzles.
Prevention is the best strategy: clean IP reputation, correct TLS fingerprint, and realistic behavior means CAPTCHA rarely appears. When it does, rotating to a fresh proxy and clearing cookies is usually enough.
Prices, availability, and add-to-cart states are increasingly loaded via JavaScript after the initial HTML response.
A scraper using curl_cffi alone receives the HTML shell without this content. For these fields, switch to Playwright and wait for the relevant elements to render before extracting.
Even a well-configured scraper will run into errors on Amazon. Here are the most common ones and how to resolve them.
Amazon has flagged your IP or session. The 503 may be a hard HTTP error or a soft block returning 200 with a robot-check body.
Check the response HTML for "robot check" or "captcha" strings, rotate your proxy, clear cookies, and retry after a randomized delay.
The product exists but is not available in the geographic region your proxy IP maps to. Use GoProxies' geo-targeting to route requests through IPs matching your target domain. Use US IPs for amazon.com and UK IPs for amazon.co.uk.
The price is loaded dynamically via JavaScript and is absent from the initial HTML response. Switch to Playwright and wait for the element to render before parsing:
///
page.goto(url) page.wait_for_selector(".a-price", timeout=10000) html = page.content()
///
Your session is burned. Rotate to a completely fresh proxy IP, create a new browser context with fresh cookies and a new fingerprint profile, and restart from scratch. Do not try to solve your way out of a burned session.
Amazon regularly updates its HTML structure, breaking hard-coded CSS selectors. Build selector fallback chains: try multiple known selectors for each field and use the first that matches.
Anchor selectors to stable IDs like #productTitle rather than class names, which change more frequently.
Amazon rate-limits your IP without returning an error. Requests time out or return after unusually long delays with no error code.
Monitor response time as a metric. If it climbs significantly above baseline, pause the session, rotate the proxy, and reduce concurrency.
Amazon's public product pages contain a substantial amount of commercially valuable data, all accessible without a login:
For teams monitoring pricing across thousands of SKUs daily, consistent uptime at the proxy layer is what keeps your dataset complete. GoProxies' rotating residential proxies are built for this kind of sustained, high-volume operation.
The short answer is: it depends on what you scrape and how. Here is what current law and Amazon's own policies say.
Scraping publicly accessible product pages, including prices, titles, ratings, and descriptions, sits in a legally defensible position under current case law.
The HiQ Labs v. LinkedIn case established that scraping non-password-protected, publicly available data does not violate the Computer Fraud and Abuse Act.
Private data is off-limits both legally and ethically, including anything behind a login wall such as order history or extended reviews.
Amazon's Conditions of Use explicitly prohibit automated access to its services. Violating ToS is not the same as breaking the law, but it does expose you to IP bans, account suspensions, and potential civil action.
Stick to public product data and respect rate limits.
If your scraping operation involves data that touches EU or California residents, GDPR and CCPA considerations apply. Stick to product-level data and avoid anything that could constitute personal data.
Amazon offers the Product Advertising API for structured access to product data, but it requires an active Amazon Associates affiliate account and approval is not guaranteed.
Pros: Fully compliant, structured JSON responses, no blocks or CAPTCHAs, stable schema.
Cons: Affiliate account required, one request per second rate limit, limited data fields, no competitor seller data, no BSR history.
Choose the API for low-volume, compliance-critical use cases. Choose scraping for scale, breadth of data, or any field the API does not provide, which covers most serious commercial use cases.
Technically yes. At any meaningful scale, no.
Free proxy IPs are shared across thousands of users, carry burned reputations, and get flagged by Amazon's WAF before the first request completes. Free scraping tools cap requests, lack proxy rotation, and have no CAPTCHA handling.
The real cost is engineering time spent debugging blocks instead of working on the actual data problem. GoProxies offers pay-as-you-go plans with no minimums or contracts, so you only pay for what you use.
Scraping Amazon in 2026 is genuinely hard. The operations that succeed share the same foundation: residential proxies with clean reputation, TLS fingerprint spoofing via curl_cffi, and stealth-patched headless browsers for JavaScript-rendered content.
Realistic request timing and disciplined error handling with session rotation round out the stack.
The proxy layer is where most operations fall down, and it is also the easiest variable to fix.
See GoProxies rotating residential proxy plans: 30 million ethically sourced residential IPs, the lowest fraud scores in the market, and pay-as-you-go pricing with no credit card required.
GoProxies Research Team is a group of analysts and writers focused on web data, online privacy, and proxy technology. They produce practical, research-backed content that helps readers understand digital trends, tools, and best practices.
The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.
A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!
Headings, paragraphs, blockquotes, figures, images, and figure captions can all be styled after a class is added to the rich text element using the "When inside of" nested selector system.
Scraping publicly available Amazon product pages is generally permissible under current case law. HiQ v. LinkedIn established that accessing non-password-protected public data does not violate the CFAA. However, it violates Amazon's Terms of Service, which can result in IP bans or civil action. Limit scraping to public product data and avoid anything behind a login wall.
Residential proxies are the most effective type for scraping Amazon. They use IPs assigned by real ISPs to consumer devices, making them appear as legitimate users to Amazon's WAF. Datacenter proxies are cheaper, but their IP ranges are flagged at the ASN level by Amazon's reputation scoring systems regardless of how clean your headers or behavior look.
The robot check page means your IP or session has been flagged. Rotate to a fresh residential proxy IP, clear all cookies and session data, wait a randomized delay, and retry. If it persists across multiple fresh IPs, the issue is likely your TLS fingerprint. Make sure you are using curl_cffi with impersonate="chrome" rather than Python's standard requests library.
