Excel Selenium Element Test

During web scraping sometimes an expected element is not present. This recently happened during a web scraping project when the client input data was not correct causing the page to load with an error and didn’t have the expected data to scrape. The web scraping was being done in Excel using Selenium and Chrome browser. The error handling can be done a couple of different ways. I used the approach of assigning a webelement object to the expected element and then test if the object is Nothing.

Element Test Code

' class that has distance in miles
Set miClass = driver.FindElementByClass("mi", timeout:=0, Raise:=False)
If Not miClass Is Nothing Then
    {Element Present}
    { Element not Present, Error Handling}

This project required loading almost 20,000 web pages. Sometimes the web page url failed to load. Although not directly tied to testing for the present of an element this error case still needed to be handled. For this case Excel VBA error handing is used. Multiple attempts are made to load the url. If the web page loads then the “attempt loop” is exited early. If the maximum number of attempts is reached, then error handling is used.

In an Excel VBA, On Error is used to control the error handing. Before making the web page loading attempt On Error GoTo and Exit For are used to manage the multiple attempts.

Web Page Loading Error handling

' attempt to get website
On Error GoTo tryAgain    ' if error GoTo tryAgain
For try = 1 To Attempts   ' number of attempts to try (const)
    ' get website
    driver.Get url        ' attempt to load web page, if error GoTo tryAgain
    Exit For              ' web page loaded exit For
Next try                  ' next attempt
On Error GoTo 0           ' reset Error handling to stop on error
If (try <> (Attempts + 1)) Then   ' Test if maximum attempts were reached
    {Page Loaded, Do Stuff}
    {Page Didn't Load, Error Handling Stuff}

Excel Selenium Web Scraping Error 33

I’ve done Chrome web scraping in both Python and Excel using Selenium on a Windows 10 platform. I have also web scraped in Python using BeatifulSoup4. A new client had a requirement to find the distance between two zip codes. The client had created an Excel worksheet with a matrix of zip codes (vertical) and cities (horizontal). I found a web site that calculates the distance between two zip codes in a web page. The zip codes are part of the web page URL I converted the cities to city airport zip codes so I just needed to read rows and columns to create the URL and the scrape the results. Easy right?

Error 33

I implemented the VBA code and got a runtime error 33 when opening the Chrome browser. I checked other Excel scrapers that I had written and they all failed. So something was wrong with my Excel environment. An Internet search indicated that the version of Chrome and the chromedriver were not the same. I remembered that I had created a directory for Selenium webdrivers and added it to the system path. I updated the driver to match the Chrome version and I still got the same runtime error. I then tried one of my Python scrapers and they worked! So my problem was with Excel. I continued my search and couldn’t find a resolution until….

Excel Selenium Environment

To setup the Excel environment you need to add SeleniumBasic library, which I already had and the version hadn’t changed since 2016. To add the library to your VBA you need to select the Selenium Type Libary (editor Tools > References…). As I re-read the installation instructions I found that the driver needs to be placed in the same directory as the SeleniumBasic library, which for me was not the webdriver directory I had already updated. That was the difference. Excel did not use the system path for the webdriver.

Once I updated the chromdriver in the SeleniumBasic directory very thing started working again. Watch out for Chrome updates. For my environment I need to update the driver in two locations – the system path directory (Python) and the SeleniumBasic directory (Excel). A good description on setting up Excel to work with Selenium can be found at myOnlineTrainingHub.