Python Selenium: Hide IP Address, Chrome and Firefox – Easy & Efficient Ways

 

Python Selenium tutorial shows a way to automate web application with python Selenium framework

Selenium

Selenium could be a portable framework for testing web applications. Python Selenium runs on Windows, Linux, and macOS.  Selenium are often run during a full mode or in an exceedingly headless mode. within the headless mode, the browser isn’t started.

Selenium drivers

Now download the drivers from https://selenium-python.readthedocs.io/installation.html . The driver must be placed on the  /usr/bin/, /usr/local/bin/ or the present working directory.

 

Python Selenium install

The selenium module is installed with the subsequent command

pip install selenium

This install the selenium module.

 

Python Selenium Firefox example

The Firefox browser, Download the driver (geckodriver.exe for Windows) from https://github.com/mozilla/geckodriver/releases.

from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options

opts = Options()
opts.headless = True

driver = Firefox(options=opts)

try:

    driver.get('http://webcode.me')
    print(browser.title)
    assert 'My html page' == driver.title

finally:

    driver.quit()

For example, we test an online web page title with the Firefox’s driver.

opts = Options()

opts.headless = True

driver = Firefox(options=opts)

We create a driver in a headless mode. The browser won’t start.

try:

    driver.get('http://webcode.me')

    print(driver.title)

    assert 'My html page' == driver.title

 

finally:

    driver.quit()

 

Python Selenium Chrome example

For Chrome Browser, we download the driver (chromedriver.exe for Windows) from https://sites.google.com/a/chromium.org/chromedriver/downloads.

#!/usr/bin/env python

from selenium.webdriver import Chrome
from selenium.webdriver.chrome.options import Options

opts = Options()
opts.headless = True

driver = Chrome(options=opts, executable_path='chromedriver.exe')

try:
    driver.get('http://webcode.me')

    assert 'My html page!' == driver.title

finally:

    driver.quit()

During the example, we use the Chrome Browser.

driver = Chrome(options=opts, executable_path='chromedriver.exe')

We create an instance of the Chrome driver. The executable_path points to the executable; if not specified, it assumes the executable is within the PATH.

assert 'My html page!' == driver.title

We added an extra explanation mark, so the test will fail.

> py chrome_get_title.py

DevTools listening on ws://127.0.0.1:61178/devtools/browser/14d2fd68-eb2a-415a-9bf0-53a0f7b388d6
My html page
Traceback (most recent call last):
File “chrome_get_title.py”, line 19, in <module>
assert ‘My html page!’ == driver.title
AssertionError

Python Selenium page source

The page_source property gets the source of the current page.

#!/usr/bin/env python

from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options

opts = Options()
opts.headless = True

driver = Firefox(options=opts)

try:

    driver.get('http://webcode.me')
    title = driver.title
    content = driver.page_source

    print(content)

    assert title == 'My html page'
    assert 'Today is a beautiful day' in content

finally:

    driver.quit()

For the example, Tets a Website title and for a specific text in the Web Page.

 

Python Selenium find elements

We can use methods such as find_elements_by_tag_name, find_element_by_id, or find_elements_by_class_name to locate the HTML elements and get their content.

#!/usr/bin/env python

from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options

opts = Options()
opts.headless = True

driver = Firefox(options=opts)

try:

    driver.get('http://webcode.me')

    els = driver.find_elements_by_tag_name("p")

    for el in els:
        print(el.text)

finally:

    driver.close()

In the example, we get and print the text of the two paragraphs on the webcode.me home page.

els = driver.find_elements_by_tag_name("p")

We find the p tags by using the find_elements_by_tag_name method.

for el in els:
    print(el.text)

We go through the list of the elements and print their contents with the text property.

> py get_paragraphs.py
Today is a beautiful day. We go swimming and fishing.
Hello there. How are you?

 

Python Selenium alert box

 

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Test page</title>
</head>
<body>

    <button id="mybtn" onclick="alert('Hello there!')">
        Click
    </button>

</body>
</html>

We have an HTML page with a button. When we click on the button, an alert box appears.

#!/usr/bin/env python

import time
from pathlib import Path

from selenium.common.exceptions import TimeoutException
from selenium.webdriver import Firefox
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.support.wait import WebDriverWait

myfile = Path.cwd() / "index.html"
driver = Firefox()

try:
    driver.get(f'file://{myfile}')
    button = driver.find_element_by_id("mybtn")
    button.click()
    time.sleep(2)

    try:
        WebDriverWait(driver, 3).until(ec.alert_is_present(),
                                       'Timed out waiting for confirmation popup to appear')

        alert = driver.switch_to.alert
        assert alert.text == 'Hello there!'

        alert.accept()
        print("alert accepted")

    except TimeoutException:
        print("no alert")

    time.sleep(5)
finally:

    driver.quit()

 

try:
    driver.get(f'file://{myfile}')

    button = driver.find_element_by_id("mybtn")

    button.click()

    time.sleep(2)

We load a file on the disk, find the button element and click on it. In this example, the browser appears; therefore, we sleep for 2 s so that we can see what is happening.

WebDriverWait(driver, 3).until(ec.alert_is_present(),
                               'Timed out waiting for confirmation popup to appear')

With WebDriverWait, we wait 3 seconds for the alert to appear. If the box does not appear, we provide an error message.

alert = driver.switch_to.alert
assert alert.text == 'Hello there!'

We check the text of the alert box.

 

Python Selenium unittest example

 

 

#!/usr/bin/env python

import unittest

from selenium import webdriver
from selenium.webdriver.firefox.options import Options


class WebCode(unittest.TestCase):

    def setUp(self):
        opts = Options()
        opts.headless = True

        self.driver = webdriver.Firefox(options=opts)

    def test_title(self):

        self.driver.get("http://webcode.me")
        self.assertIn("My html page", self.driver.title)

    def test_paragraphs(self):

        self.driver.get("http://webcode.me")

        els = self.driver.find_elements_by_tag_name("p")

        self.assertIn('Today is a beautiful day', els[0].text)
        self.assertIn('Hello there', els[1].text)

    def tearDown(self):
        self.driver.close()


if __name__ == "__main__":
    unittest.main()

We have a test file, where we check for the title and paragraphs of the webcode.me home page.

def setUp(self):
    opts = Options()
    opts.headless = True

    self.driver = webdriver.Firefox(options=opts)

In the setUp method, we set up the Firefox driver.

def test_title(self):

    self.driver.get("http://webcode.me")

    self.assertIn("My html page", self.driver.title)

The test_title method is a single test case, which checks for the title of the specified web page using the assertIn method.

def test_paragraphs(self):

    self.driver.get("http://webcode.me")

    els = self.driver.find_elements_by_tag_name("p")

    self.assertIn('Today is a beautiful day', els[0].text)

    self.assertIn('Hello there', els[1].text)

In the test_paragraphs test case, we check for the content of the two paragraphs.

def tearDown(self):
    self.driver.close()

In the tearDown method, we close the driver.

if __name__ == "__main__":
    unittest.main()

With the main method, we execute the tests.

> py unittest_example.py
..
———————————————————————-
Ran 2 tests in 13.273s

OK

We run the tests. The unittest shows a dot for a successfully executed test case and F for a failed one, and E for an error that occurred during the execution of the test.

 

Python Selenium with pytest

The pytest module is a Python library for testing Python applications. It is an alternative to nose and unittest

pip install pytest

We install the pytest library.

The pytest looks for test_*.py or *_test.py files in directories. In the selected files, pytest looks for test prefixed test functions outside of class and test prefixed test methods inside Test prefixed test classes (without an __init__ method).

#!/usr/bin/env python

import pytest

from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options


@pytest.fixture
def browser():

    opts = Options()
    opts.headless = True
    driver = Firefox(options=opts)

    driver.implicitly_wait(5)

    yield driver

    # For cleanup, quit the driver
    driver.quit()


def test_get_title(browser):
    browser.get("http://webcode.me")

    assert 'My html page' == browser.title

The example shows a simple test case with the pytest module.

@pytest.fixture
def browser():

    opts = Options()

    opts.headless = True

    driver = Firefox(options=opts)

We define a fixture. It sets up the Firefox driver.

driver.implicitly_wait(5)

We wait implicitly for elements to be ready before attempting interactions.

yield driver

With the yield keyword, we return the driver object at the end of setup.

def test_get_title(browser):

    browser.get("http://webcode.me")

    assert 'My html page' == browser.title

We have a test method which checks the title of a web page. It receives the brower fixture as a parameter.

> pytest
============================== test session starts ==============================
platform win32 — Python 3.8.1, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: C:\Users\Jano\Documents\python\SeleniumPytest
collected 1 item

tests\test_web.py .                                                        [100%]

=============================== 1 passed in 6.31s ===============================

We run the tests.
Python Selenium Flask example

In the next example, we create a test case with pytest and Selenium for a Flask web application. We test for a response from an HTML form.

app.py
├───static
│       greet.html
├───templates
│       index.html
└───tests
web_test.py

This is the project structure.

<!doctype html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Greet form</title>
</head>

<body>

    <p>Enter your name:</p>

    <form id="myform" action="greet">

        <input name="name" type="text">
        <button type="submit">Submit</button>

    </form>

</body>

</html>

We have a greet form in the static resources. The form sends a text value to the Flask application.

<!doctype html>
<html lang="en">

<head>
    <meta charset="utf-8">
    <title>Greeting</title>
</head>

<body>
    <p>
        Hello {{ name }}
    </p>
</body>

</html>

This is the Flask template file, which returns a message back to the client.

#!/usr/bin/env python

from flask import Flask, render_template, request

app = Flask(__name__)


@app.route("/")
def home():
    return app.send_static_file('greet.html')


@app.route("/greet")
def greet():
    username = request.args.get('name')
    return render_template('index.html', name=username)


if __name__ == "__main__":
    app.run()

The Flask application has two routes: one for the home page and one for the greeting.

#!/usr/bin/env python

import pytest

from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.support.wait import WebDriverWait


@pytest.fixture
def browser():
    opts = Options()
    opts.headless = True
    driver = Firefox(options=opts)

    driver.implicitly_wait(10)

    yield driver

    # For cleanup, quit the driver
    driver.quit()


def test_greet_form(browser):
    user_name = "John"

    browser.get('http://localhost:5000/')

    form = browser.find_element_by_id("myform")
    name = browser.find_element_by_name("name")
    name.send_keys(user_name)

    form.submit()

    WebDriverWait(browser, 12).until(ec.url_matches('/greet'),
                                     'Timed out waiting for response')

    content = browser.page_source
    print(content)
    assert 'Hello John' in content

The web_test.py contains a test case for the greet form.

def test_greet_form(browser):
    user_name = "John"

    browser.get('http://localhost:5000/')

First, we get the greet form.

form = browser.find_element_by_id("myform")

name = browser.find_element_by_name("name")

name.send_keys(user_name)

form.submit()

We retrieve the elements of the form. We add the test user name to the input tag and submit the form.

WebDriverWait(browser, 12).until(ec.url_matches('/greet'),
                                 'Timed out waiting for response')

We waint for the Flask to redirect to the /greet route.

content = browser.page_source

print(content)

assert 'Hello John' in content

We get the response and check for the message in the response.

> set FLASK_APP=app.py
> flask run

We run the Flask application.

> pytest
========================== test session starts ===========================
platform win32 — Python 3.8.1, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: C:\Users\Jano\PycharmProjects\SeleniumPytestFlask
collected 1 item

tests\web_test.py .                                                 [100%]

=========================== 1 passed in 12.44s ===========================

From a different shell, we execute the pytest.

In this tutorial, we have worked wit the Python Selenium framework.