If you have ever wanted to automate filling forms, clicking on buttons, and getting data from a website, but found Requests and BeautifulSoup too cumbersome, then I have good news for you: there is a simpler way. It is called Selenium.
As advertised on Selenium’s website, Selenium automates browsers. That means your code will run in Chrome, Firefox, or a different browser, and you will be able to easily click on buttons, select input fields, and log into websites. There will be no need to manually deal with cookies and other pesky http internals; Selenium will take care of that.
To get a glimpse at Selenium’s powerful features we will write a short program that logs onto your Facebook profile, finds your best friend on Messenger, and sends him a rude message for no good reason.
Setup
If you don’t already have Python installed, download it from here. Now, let’s get Selenium:
$ pip install selenium
We need to give it the ability to control our browser, so we will install a driver. What follows are instructions for Chrome. If you want to use a different browser, scroll down to the “Drivers” section here.
Chrome:
- Download the driver from this website and unzip it.
- Add the “chromedriver” file to PATH. If you are on macOS or Linux this consists of moving the file to /usr/local/bin and restarting the terminal. If you are on Windows, google it.
Selenium is now ready to go.
Writing the program
from selenium import webdriver
browser = webdriver.Chrome()
browser.get('https://facebook.com')
When you run the code you should see Facebook open in a new Chrome window.
The “webdriver.Chrome()” object is the interface that allows us to execute actions on websites. The “get” function opens the address we give it as the first argument. You can find the full Selenium API here.
To find html objects on the website, we can use css selectors (cheatsheet).
email_field = browser.find_element_by_css_selector("#email")
password_field = browser.find_element_by_css_selector("#pass")
Since they are input fields, we will fill them out:
email_field.send_keys("[email protected]")
password_field.send_keys("mypassword")
We will now log in:
login_button = browser.find_element_by_css_selector("#loginbutton")
login_button.click()
We needed only 9 lines of code to log into Facebook. However, you will see that Facebook asks for agreement to bombard you with desktop notifications and blocks you from doing anything else until you respond. In order to disable that, we will go back in time:
# Replace browser = webdriver.Chrome() with
options = webdriver.ChromeOptions()
prefs = {"profile.default_content_setting_values.notifications" : 2}
options.add_experimental_option("prefs", prefs)
browser = webdriver.Chrome(chrome_options=options)
Now, we can find your friend’s Facebook id by looking at their profile’s url:
If we open up Messenger in the browser we will see that the url scheme for the page where you can message a user is “https://www.facebook.com/messages/t/{profile-id}”. So:
profile_id = "adrian.c"
browser.get("https://www.facebook.com/messages/t/" + profile_id)
msg_field = browser.find_element_by_css_selector("[contenteditable=true]")
msg_field.send_keys("hello!")
msg_field.send_keys("\uE006") # return key
That’s it!
Full program:
from selenium import webdriver
options = webdriver.ChromeOptions()
prefs = {"profile.default_content_setting_values.notifications" : 2}
options.add_experimental_option("prefs", prefs)
browser = webdriver.Chrome(chrome_options=options)
browser.get('https://facebook.com')
email_field = browser.find_element_by_css_selector("#email")
pass_field = browser.find_element_by_css_selector("#pass")
email_field.send_keys("[email protected]")
pass_field.send_keys("mypassword")
submit_button = browser.find_element_by_css_selector("#loginbutton")
submit_button.click()
profile_id = "adrian.c"
browser.get("https://www.facebook.com/messages/t/" + profile_id)
msg_field = browser.find_element_by_css_selector("[contenteditable=true]")
msg_field.send_keys("hello!")
msg_field.send_keys("\uE006") # return key
Notes
If you want to run Selenium without opening a browser window – for example, in order to deploy it on a remote server – you can run it in headless mode:
options = webdriver.ChromeOptions()
options.add_argument('headless')
browser = webdriver.Chrome(chrome_options=options)