Set up Selenium with Chrome (Headless)

August 22, 2016

UPDATE 2017-08-31
Use chromedriver version 2.25. Tried using 2.9 with chrome 60 and could not navigate to URLS.
It throws the exception below. Works fine with 2.25.

WebDriverException: Message: unknown error: Runtime.executionContextCreated has invalid 'context': {"auxData":{"frameId":"18604.1","isDefault":true},"id":1,"name":"","origin":"://"}
  (Session info: chrome=60.0.3112.113)
    (Driver info: chromedriver=2.9.248304,platform=Linux 4.4.0-93-generic x86_64)

Setting up Selenium to work in on a headless server requires the browser to be headless as well.
This involves setting up Google Chrome, Chromedriver, and Selenium.

Install Chromedriver

All instructions that follow are for Ubuntu.

sudo apt-get install unzip
wget http://chromedriver.storage.googleapis.com/2.25/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
rm chromedriver_linux.zip
sudo mv chromedriver /usr/local/bin/

Chromedriver 2.9 is current as of this writing. Check the Chromedriver downloads page for the latest stable release.
That last command moves chromedriver to your local bin folder to put in on the $PATH, so that it’s accessible to the logged-in user.

Install Chrome

I don’t know if this is the best way to install Chrome, but it worked for me.

Get the debian package:

$ wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb

If you try installing it, most likely you’ll get an error for missing dependencies:

$ sudo dpkg -i google-chrome-stable_current_amd64.deb

Errors were encountered while processing:

google-chrome-stable

So force install the dependencies:

$ sudo apt-get install -f

Install Python packages

$ pip install beautifulsoup4 pyvirtualdisplay selenium

BeautifulSoup isn’t critical at this stage, but I usually end up using it to parse page sources.

To Use

from pyvirtualdisplay import Display
from selenium import webdriver

display = Display(visible=0, size=(800, 600))
display.start()
driver = webdriver.Chrome()

# automate_awesome_stuff

# when you're done, stop the display
display.stop()