< Back
How to rotate proxies in Python?

How to rotate proxies in Python?

In this article we will explain how to rotate proxies for web scraping. Rotating proxies will ensure stable sessions so you can reach your desired targets without issues.

How to start with rotating proxies in Python - installing prerequisites

To get started, you can create a virtual environment by running the following command:


virtualenv venv

This command will help you to install Python, pip, and common libraries in your venv folder.
Use the source command to activate your environment:


source venv/bin/activate

Install requests module in the current virtual environment you are using:


pip install requests

Congratulations! You have finished all the steps for  the installation of the requests module!

Sending GET requests through a proxy

Now, let’s start with the basics. In some cases you might need to connect and use one single IP address or proxy. How do we use a single proxy?These are the essential things that you will need:

  • Scheme (e.g., http);
  • Endpoint;
  • Port (e.g., 10000);
  • Username and password to connect to the proxy.

Here is an example how the proxy request should look in this case:


https://Customer-username:password@proxy.goproxies.com:10001

You can also select multiple protocols, as well as specify domains where you would like to use a separate proxy.
Replace PROXY1, PROXY2, PROXY3 with your proxy format as shown in the example below:


scheme_proxy_map = {
    'http': PROXY1,
    'https': PROXY2,
    'https://example.org': PROXY3,
}

Make a request using requests.get while providing the variables we created previously:


response = requests.get('https://ip.goproxies.com', 
proxies=scheme_proxy_map, timeout=10)
print(response.text)

Your full command should look like this:


import requests
scheme_proxy_map = {
	'http': http://customer-username:password@proxy.goproxies.com:10000,
	'https': https://customer-username:password@proxy.goproxies.com:10001,
  'https://example.org': 
https://customer-username:password@proxy.goproxies.com:10001,
}
response = requests.get('https://ip.goproxies.com',
proxies=scheme_proxy_map, timeout=10)
print(response.text)

The result of this script will provide you with the IP address of your proxy:


% python proxy.py 
45.42.JKL.MNO

You have now taken care of hiding behind a proxy when making requests through the Python script.
Let's learn how to rotate through a list of proxies instead of just using one.

Rotating proxies using proxy pool

You will work with a list of proxy servers saved as a CSV file called proxies.csv, in which you need to list proxy servers as shown below:


http://customer-username:password@proxy.goproxies.com:10000
https://customer-username:password@proxy.goproxies.com:10001
http://customer-username:password@proxy-america.goproxies.com:10000
http://customer-username:password@proxy-asia.goproxies.com:10000
http://customer-username:password@proxy-europe.goproxies.com:10000

If you want to add more proxies in the file, add each of them in a separate line.
After that, create a Python file and specify the file name and the timeout duration for each single proxy response.


TIMEOUT_IN_SECONDS = 10
CSV_FILENAME = 'proxies.csv'

Using the code provided, open the CSV file, read each line of proxy servers into the csv_row variable, and build the scheme_proxy_map configuration.
This is an example of how it should look:


with open(CSV_FILENAME) as open_file:
    reader = csv.reader(open_file)
    for csv_row in reader:
        scheme_proxy_map = {
            'https': csv_row[0],
        }

To ensure that everything runs efficiently, we'll use the same scraping code as before, to access the site with proxies.


with open(CSV_FILENAME) as open_file:
    reader = csv.reader(open_file)
    for csv_row in reader:
        scheme_proxy_map = {
            'https': csv_row[0],
        }
        # Access the website via proxy
        response = requests.get('https://ip.goproxies.com',
        proxies=scheme_proxy_map, timeout=TIMEOUT_IN_SECONDS)
        print(response.text)

If you want to scrape content using any working proxy from the list, just add a break after print to stop going through the proxies in the CSV file:


response = requests.get('https://ip.goproxies.com', 
proxies=scheme_proxy_map, timeout=TIMEOUT_IN_SECONDS)
        print(response.text)
        break  # break here to stop going through the proxies

Your full code should look like this:


import requests
import csv

TIMEOUT_IN_SECONDS = 10
CSV_FILENAME = 'proxies.csv'

with open(CSV_FILENAME) as open_file:
    reader = csv.reader(open_file)
    for csv_row in reader:
        scheme_proxy_map = {
            'https': csv_row[0],
        }
        response = requests.get('https://ip.goproxies.com', 
        proxies=scheme_proxy_map, timeout=TIMEOUT_IN_SECONDS)
        print(response.text)
        # break here to stop going through the proxies

That's it! Congratulations, you have successfully learned how to rotate proxies using Python.

Copywriter

Matas has strong background knowledge of information technology and services, computer and network security. Matas areas of expertise include cybersecurity and related fields, growth, digital, performance, and content marketing, as well as hands-on experience in both the B2B and B2C markets.

FAQ

What Are Rotating Residential Proxies?
Rotating Residential Proxies offer you the best solution for scaling your scraping without getting blocked.

Rotating proxies provide a different IP each time you make a request. With this automated rotation of IPs, you get unlimited scraping without any detection. It provides an extra layer of anonymity and security for higher-demand web scraping needs.

IP addresses change automatically, so after the initial set up you’re ready to scrape as long and much as you need. IPs may shift after a few hours, a few minutes or after each session depending on your configuration. We do this by pulling legitimate residential IPs from our pool.
Why Do You Need Rotating Residential Proxies?
There are a number of use cases for rotating residential proxies. One of the most common ones is bypassing access limitations.

Some websites have specific measures in place to block IP access after a certain number of requests over an extended period of time.

This limits your activity and hinders scalability. With rotating residential IP addresses, it's almost impossible for websites to detect that you are the same user, so you can continue scraping with ease.
When to Use Static Residential Proxies Instead?
There are particular cases where static residential proxies may be more useful for your needs, such as accessing services that require logins.

Rotating IPs might lead to sites not functioning well if they are more optimised for regular use from a single IP.

Learn if our static residential proxies are a better fit for your needs.
Can I choose the IP location by city?
Yes. GoProxies has IPs spread across almost every country and city worldwide.
Can I choose the IP location by country state?
Yes. GoProxies has IPs spread across X countries with localised IPs in every state.

What are rotating proxies?

Rotating proxies are a type of proxy that automatically change or rotate their IP addresses at regular intervals or after each request. They're commonly used in web scraping to avoid detection or bans by websites, as they make it harder for the server to recognize multiple requests coming from the same IP address.

Why rotate proxies?

You rotate proxies to avoid getting blocked or banned when web scraping. By frequently changing your IP address, it's harder for websites to detect and restrict your scraping activities, improving your chances of successful data extraction.

What is the IP rotation library in Python?

The main library that is required for proxy usage in Python overall is called requests.

What’s a Rich Text element?

The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.

Static and dynamic content editing

A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!

How to customize formatting for each rich text

Headings, paragraphs, blockquotes, figures, images, and figure captions can all be styled after a class is added to the rich text element using the "When inside of" nested selector system.

By clicking “Accept All Cookies”, you agree to the storing of cookies on your device to enhance site navigation, analyze site usage, and assist in our marketing efforts. View our Privacy Policy for more information.