Playwright runs a real browser, not an approximation of one. Fully rendered pages, real interaction, and the session behavior anti-bot systems expect from an actual user. Paired with rotating residential proxies, it reaches targets that lighter tools can't.
It's also overkill for the ones they can. Playwright earns its place on JavaScript-heavy sites, pages that require interaction, and operations that need real browser fingerprints. For simple scraping, a lighter HTTP library is the better call.
This guide covers installing and configuring Playwright, rotating proxies across browser contexts, avoiding detection, and fixing the errors that turn up in production.
bash
pip install playwright playwright-stealth beautifulsoup4 lxml python-dotenv
python -m playwright install chromium
The second command downloads the Chromium binary. Specifying chromium keeps the install lean.
Store your credentials in a .env file in your project root:
ini
PROXY_HOST=proxy.goproxies.com
PROXY_PORT=10001
PROXY_USER=your-username
PROXY_PASS=your-password
Never commit this file. Add .env to your .gitignore from the start.
Sign up and get your credentials at GoProxies. With 30 million residential IPs across 200 locations and pay-as-you-go pricing, you're not locked into a subscription before you've validated your setup.
Browser-level configuration routes all pages and contexts through the same IP, which is what you want when you need a consistent IP across a multi-page session:
python
import asyncio
import os
from dotenv import load_dotenv
from playwright.async_api import async_playwright
load_dotenv()
PROXY = {
"server": f"http://{os.getenv('PROXY_HOST')}:{os.getenv('PROXY_PORT')}",
"username": os.getenv("PROXY_USER"),
"password": os.getenv("PROXY_PASS"),
}
async def main():
async with async_playwright() as p:
browser = await p.chromium.launch(proxy=PROXY)
page = await browser.new_page()
await page.goto("https://ip.goproxies.com")
print(await page.inner_text("body"))
await browser.close()
asyncio.run(main())
Playwright accepts HTTP, HTTPS, and SOCKS5 proxies in the server field. For GoProxies rotating residential proxies, HTTP is the standard format.
For rotation, context-level configuration is what you want. It assigns a proxy to each isolated browser context while sharing one browser instance, which keeps RAM usage manageable. Each context is completely isolated – separate cookies, local storage, and session state – and websites see each as a distinct browser.
python
import asyncio
import os
from dotenv import load_dotenv
from playwright.async_api import async_playwright
from playwright_stealth import stealth_async
load_dotenv()
PROXY = {
"server": f"http://{os.getenv('PROXY_HOST')}:{os.getenv('PROXY_PORT')}",
"username": os.getenv("PROXY_USER"),
"password": os.getenv("PROXY_PASS"),
}
URLS = [
"https://example.com/page-1",
"https://example.com/page-2",
"https://example.com/page-3",
]
async def scrape(browser, url):
context = await browser.new_context(proxy=PROXY)
page = await context.new_page()
await stealth_async(page)
try:
await page.goto(url, timeout=30000)
return await page.content()
except Exception as e:
print(f"Failed {url}: {e}")
return None
finally:
await context.close()
async def main():
async with async_playwright() as p:
browser = await p.chromium.launch()
results = await asyncio.gather(*[scrape(browser, url) for url in URLS])
await browser.close()
return results
asyncio.run(main())
Because GoProxies rotates the IP on each new connection, each new context gets a fresh IP. No proxy list to maintain, no rotation logic to write.
Navigate to an IP-check endpoint and confirm the returned IP isn't your own:
python
async def verify_proxy(browser):
context = await browser.new_context(proxy=PROXY)
page = await context.new_page()
await page.goto("https://ip.goproxies.com")
body = await page.inner_text("body")
print(f"Current IP: {body.strip()}")
await context.close()
Run this after every configuration change. If the IP matches your real address, the proxy isn't being applied.
With the proxy confirmed, use BeautifulSoup with the lxml parser to extract data from the rendered HTML:
python
from bs4 import BeautifulSoup
async def parse_page(browser, url):
context = await browser.new_context(proxy=PROXY)
page = await context.new_page()
await stealth_async(page)
await page.goto(url, wait_until="networkidle", timeout=30000)
html = await page.content()
await context.close()
soup = BeautifulSoup(html, "lxml")
titles = [el.text.strip() for el in soup.select("h2.product-title")]
prices = [el.text.strip() for el in soup.select("span.price")]
return list(zip(titles, prices))
wait_until="networkidle" tells Playwright to wait until network requests have settled before handing over the HTML, which matters for pages that load content asynchronously.
Proxy configuration is step one. Keeping sessions alive takes a few more layers.
Each new context gets a fresh IP from a real device and ISP, so traffic reads as genuinely organic. Datacenter proxies are cheaper but carry higher reputation risk and get flagged more often on protected targets. If you're already paying the cost of a full browser, the proxy layer should match that investment.
GoProxies gives you 30 million residential IPs across 200 locations with automatic rotation on every new connection. Check the rotating residential proxies pricing to find a plan that fits your volume.
Browser contexts start with no cookies and a clean session state, which prevents session bleed between targets. For sites that require a logged-in session, save the authentication state from one context and reload it into subsequent ones so you're not re-authenticating on every run.
Playwright's default headless mode exposes automation signals. Address them through playwright-stealth and explicit browser arguments rather than switching to headless=False. Headed browsers are slower, resource-heavy, and harder to scale.
Anti-bot systems flag mismatches between browser signals. Keep your user agent, language headers, and timezone consistent with the geo of the IP you're using, and randomize delays between actions rather than firing events at machine speed.
Proxies add failure modes that wouldn't exist on a direct connection. Build retry logic with exponential backoff into every request, and close the context cleanly on failure before retrying with a new one.
asyncio.gather lets you run multiple coroutines concurrently within one browser instance. Start conservative and measure memory before scaling up. Spawning more contexts than your system can handle produces failures that look exactly like proxy issues but are actually resource exhaustion.
Modern anti-bot systems build a fingerprint from dozens of browser properties and compare it against known profiles for real users. Playwright out of the box leaves signals across several layers.
Every default Playwright session sets navigator.webdriver to true, includes a HeadlessChrome marker in the user agent, and produces a fingerprint that doesn't match any real Chrome release. playwright-stealth patches many of these at the JavaScript layer but can't address signals that require binary-level modification.
JavaScript-based challenges verify the browser environment against what a real browser produces: canvas rendering output, WebGL renderer strings, audio fingerprints, font enumeration results. Playwright's Chromium binary produces values that don't match any real Chrome release, which can trigger a challenge before any interaction happens.
Combining playwright-stealth with a clean residential IP removes the most common triggers, but heavily protected targets may need more.
This is the layer most guides skip, and it's the one that catches people who've done everything else right.
Playwright communicates with the browser over the Chrome DevTools Protocol. Some detection systems don't look for automation properties at all – they look for the side effects of CDP being attached: timing anomalies at the protocol layer, and the way certain browser behaviors change when a debugger-class client is connected.
This matters because it isn't patchable at the JavaScript layer. playwright-stealth can overwrite navigator.webdriver because that's a property in the page's own JavaScript environment. It can't change the fact that something is speaking CDP to the browser. If you've applied stealth, matched your headers, verified a clean exit IP, and a specific target still challenges every session, CDP detection is the likely explanation – and the honest answer is that Playwright may be the wrong tool for that particular target.
Even with rotating residential proxies, hitting the same target too fast triggers rate limiting based on volume patterns. Watch for 429 responses (throttled by request rate) and 403 responses (IP flagged or fingerprint blocked), and cap concurrent contexts accordingly.
JavaScript-rendered content is where Playwright earns its place. Any page that loads content asynchronously via API calls, lazy loading, or client-side rendering will return incomplete HTML to a standard HTTP request. Playwright waits for the full render before handing over the content.
Interaction-dependent data – product variant pricing, expandable review sections, infinite scroll feeds, modal content – only appears after a click, scroll, or form submission. Playwright drives those interactions programmatically with page.click(), page.fill(), and page.mouse.wheel() for scroll-triggered content, or locator.scroll_into_view_if_needed() when you need a specific element in view.
Geo-restricted content becomes reachable when you pair Playwright with residential proxies that offer country and city-level targeting. Localized pricing, region-specific listings, and location-filtered search results are all accessible this way.
Structured data from complex pages – product listings, reviews, ratings, pricing tables, view counts, timestamps, paginated datasets – all extract reliably once you have a fully rendered DOM.
GoProxies gives you the geographic reach and IP quality to access all of it at scale. Get started at GoProxies and put your Playwright setup to work.
Legality depends on what you're scraping and how, not on the tool.
Publicly accessible data sits in a different legal category from data behind authentication walls, paywalls, or private APIs. Scraping public data has generally been treated more favorably in legal analysis, though the picture varies by jurisdiction.
Terms of Service aren't laws, but violating them can expose you to civil liability and account termination. If your use case is commercial and involves a high-profile target, get qualified legal advice before running anything at scale.
Ethical scraping means not overloading servers, not extracting personal data without a valid purpose, and respecting rate limits and robots.txt even where they aren't legally binding.
Playwright includes a built-in request API that makes HTTP requests from within the browser context without rendering a full page.
The request API is faster and far less memory-intensive than a full headless browser. The trade-off is that it behaves like an HTTP library: no JavaScript rendering, and a different fingerprint. For targets that serve complete HTML without JavaScript, or expose data via accessible API endpoints, it's the better fit.
Full browser mode is the right choice when the target requires JavaScript rendering, interaction events, or session-state behavior. Combined with rotating residential proxies and stealth configuration, the fingerprint it produces is much harder to distinguish from legitimate traffic.
Start with the request API for any endpoint you can validate with a simple HTTP request, and escalate to full browser mode only where the target actually requires it.
You can configure Playwright to use free proxies, but the combination isn't viable for real work.
Free proxies have high failure rates, inconsistent uptime, and abused IP pools. Anti-bot systems maintain blocklists of known free proxy ranges, and those IPs are typically flagged before your session even begins. Most free proxy lists also don't support authenticated access, which leaves you with no guarantee of IP quality, geo-targeting, or rotation.
For production scraping, a residential network with a verified IP pool and automatic rotation is the right pairing. Start with GoProxies’ rotating residential proxies with no minimums and no contracts. See the full plans on the rotating residential proxies pricing page.
Playwright is a strong scraping tool on the right targets: JavaScript-rendered pages, interaction-dependent content, and sites with serious fingerprint-based detection. Paired with rotating residential proxies, context isolation, stealth configuration, and proper retry logic, it handles real-world scraping at scale.
The proxy configuration itself is straightforward. Getting sessions to survive on protected targets takes more attention: fingerprint consistency, header alignment, and behavioral signals beyond the basic setup. Start with the step-by-step section, verify your proxy at each stage, and add evasion layers as your target requires.
Gintarė specializes in content related to proxies, web scraping, data collection, and internet infrastructure. With a solid background in information technology, networking, and cybersecurity, she understands both the technical and practical aspects of large-scale data acquisition.
The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.
A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!
Headings, paragraphs, blockquotes, figures, images, and figure captions can all be styled after a class is added to the rich text element using the "When inside of" nested selector system.
Yes, with one important limitation. Set the server field to socks5://host:port in the proxy dictionary passed to either launch or new_context. However, Chromium doesn't support username and password authentication over SOCKS5, so authenticated residential proxies need the HTTP or HTTPS endpoint instead. HTTP and HTTPS proxies use the same dict structure, and most rotating residential providers, including GoProxies, use HTTP by default.
Browser-level proxies apply to all contexts and pages from that browser instance. Context-level proxies assign a specific proxy to one isolated context, letting different contexts use different IPs simultaneously. Context-level is the right choice for rotation because it distributes requests across multiple IPs without spawning multiple browser processes.
Yes, for JavaScript-heavy or interaction-dependent targets, but each context consumes significant memory. For targets that serve complete HTML without JavaScript, a lighter library is more cost-effective. Use Playwright where the full browser is genuinely required, and scale concurrency carefully.
No. GoProxies handles IP rotation automatically on each new connection. When you create a new browser context, the endpoint assigns a fresh IP with no rotation logic required on your side.
