Want to use Selenium to scrape with Chrome on Windows? Let’s do it!
We’ll need to install a couple things:
- Selenium, which allows you to control browsers from Python
- ChromeDriver, which allows software to control Chrome (like Selenium!)
STEP ONE: Downloading ChromeDriver
First, download ChromeDriver from its terribly ugly site. It looks like a scam or like it was put together by a 12 year old, but I promise it’s good and cool and nice.
chromedriver_win32.zip. That link should download
2.40, but if you want something more recent just go to the page and download the right thing.
STEP TWO: Unzipping ChromeDriver
chromedriver_win32.zip and it will give you a file called
chromedriver.exe. This is the magic software!
STEP THREE: Moving ChromeDriver somewhere sensible
Now we need to move ChromeDriver somewhere that Python and Selenium will be able to find it (a.k.a. in your
The easiest place to put it is in
C:\Windows. So move it there!
If you can’t move chromedriver there, you can always just tell Python where it is when you’re loading it up. See Selenium snippets under “But Python can’t find chromedriver”
Add to the PATH on Windows 10 : https://www.architectryan.com/2018/03/17/add-to-the-path-on-windows-10/
If you google about Selenium, a lot of the time you see things about “Selenium server” and blah blah blah – you don’t need that, you aren’t running a huge complex of automated browser testing machines. You don’t need that. We just need plain ol’ Selenium.
pip3 to install Selenium for Python 3
pip install selenium
Oh, you also need to make sure you have Chrome (or Firefox) installed and it lives in one of the normal places applications do.
If Python can’t find Chrome/Firefox, you can always just tell Python where it is when you’re loading it up. See Selenium snippets under “But Python can’t find Chrome/Firefox”
Selenium snippets: http://jonathansoma.com/lede/foundations-2019/classes/scraping/selenium-snippets/
Want to make sure it works? Run the following to pull all of the headlines from the spyboy blog.
from selenium import webdriver driver = webdriver.Chrome() driver.get("https://www.spyboy.blog") headlines = driver.find_elements_by_class_name("story-heading") for headline in headlines: print(headline.text.strip())