Getting Started with Web Scraping in Python

“Web scraping” or “data scraping” is simply the process of extracting data from a website. This can, of course, be done manually: you could go to a website, find the relevant data or information, and enter that information into some data file that you have stored locally. But imagine that you want to pull a very large dataset or data from hundreds or thousands of individual URLs. In this case, extracting the data manually sounds overwhelming and time-consuming.

In this article, we’ll see how to automate this process with Python, using the BeautifulSoup and Selenium libraries. BeautisulSoup is a Python library for extracting and parsing data from HTML files and the Selenium package is used to automate web browser interaction between Python and your browser, despite differences in browser design (e.g., Chrome versus FireFox).

We’ll start by (1) downloading and installing the relevant libraries and web drivers. Next, we’ll (2) fetch and render a web page, (3) parse the HTML content, and inspect the web page to figure out what class names and IDs are relevant to the data we want to extract from the HTML. Finally, we’ll (4) save the data to a comma-separated varibale (csv) file locally, for later use.

At the end of this article, we also demonstrate how the tools built here can be converted to Python functions, automated, and used to scrape data from several hundred different URLs very quickly!

We assume that you have already installed Python, but if not, you can find a detailed set of install instructions here. You’ll also need to have either Google Chrome or FireFox installed.

Downloading Libraries

Make sure you install the BeautifulSoup and Selenium libraries in your Python environment. To do so, from your Terminal, run:

pip install beautifulsoup4

pip install selenium

At this point, you also need to install ChromeDriver. You have two options here: you can use Homebrew to install directly from your Terminal (if using macOS), or follow the instructions to download a zip file from the ChromeDriver website (Windows or macOS). Using Homebrew, in your Terminal, just run:

brew install --cask chromedriver

This may take a few minutes, but once the install is complete, to check that the install was successful, you can run:

chromedriver --version

The output should match the version of Chrome that you have installed locally (check Chrome > About Google Chrome > Version).

Whether you installed via Homebrew or direct download, we want to confirm that the install was successful and confirm the path to chromedriver. In your Terminal, run:

which chromedriver

Remember this path; we’ll need it later.

If using Windows, download the chromedriver_win32.zip file and unzip it. You should have a single exe file called “chromedriver.exe” in a folder called “chromedriver_win32”. Again remember the path to this folder and file.

Now we’re ready to start web scraping!

Fetching and Rendering a Web Page

First, let’s try rendering a web page *without* using Selenium and ChromeDriver, and try to parse the HTML. For this demo, we’re going to try to scrape the information contained in the Summary table from a Weather Underground page for weather observations taken at the Charlottesville Albemarle Airpot (CHO) on December 31, 2020. Here’s the URL: https://www.wunderground.com/history/daily/KCHO/date/2020-12-31 and a screenshot of that webpage:

Before we try rendering the web page, we want to inspect the HTML elements to determine what class we need to select:

We can see that the entire Summary table is contained under lib-city-history-summary. Let’s use that tag to grab just the Summary table elements from the HTML. We’ll also use the .prettify() method to print out the selected HTML to make it just a little more readable.

import os

from bs4 import BeautifulSoup
import requests

search_url = 'http://www.wunderground.com/history/daily/KCHO/date/2020-12-31'

wunderground_page = requests.get(search_url)

wunderground_soup = BeautifulSoup(wunderground_page.text, 'html.parser')

soup_container = wunderground_soup.find('lib-city-history-summary')

print(soup_container.prettify())

    <lib-city-history-summary _ngcontent-sc301="" _nghost-sc237="">
     <div _ngcontent-sc237="">
      <div _ngcontent-sc237="" class="summary-title">
       Summary
      </div>
      <div _ngcontent-sc237="" class="summary-table">
       No data recorded
       <!-- -->
       <!-- -->
      </div>
     </div>
    </lib-city-history-summary>

The HTML returned by BeautifulSoup shows “No data recorded”, but when we go to the Summary table on the web page, the table is obviously populated. What happened?

In this case, the requests.get() method ran faster than the web page could load, so the Summary table wasn’t available by the time BeautifulSoup parsed the web page text. Here’s where the Selenium library and ChromeDriver will come in and allow the page content to load completely before we try to parse it.

To do this, we’re going to write a quick function rendering that will run the ChromeDriver. This will open a Chrome window, load the web page fully, return the page HTML, and quit the ChromeDriver, closing the Chrome window. Be sure to update the path in the webdriver.Chrome function to the path where you installed the ChromeDriver.

def rendering(url):
    
        # change '/usr/local/bin/chromedriver' to the path from you got when you ran 'which chromedriver'
        driver = webdriver.Chrome('/usr/local/bin/chromedriver') # run ChromeDriver
        driver.get(url)                                          # load the web page from the URL
        time.sleep(3)                                            # wait for the web page to load
        render = driver.page_source                              # get the page source HTML
        driver.quit()                                            # quit ChromeDriver
        return render                                            # return the page source HTML

Now, using this function, let’s try again to get the Summary table elements from the HTML. Following the code written above, where, instead of using requests.get(), we’ll use our new rendering function:

import os

from bs4 import BeautifulSoup
from selenium import webdriver

import time

search_url = 'http://www.wunderground.com/history/daily/KCHO/date/2020-12-31'

wunderground_page = rendering(search_url)

wunderground_soup = BeautifulSoup(wunderground_page, 'html.parser')

soup_container = wunderground_soup.find('lib-city-history-summary')

print(soup_container.prettify())

<lib-city-history-summary _ngcontent-app-root-c234="" _nghost-app-root-c204="">
     <div _ngcontent-app-root-c204="">
      <div _ngcontent-app-root-c204="" class="summary-title">
       Summary
      </div>
      <div _ngcontent-app-root-c204="" class="summary-table">
       <table _ngcontent-app-root-c204="" aria-labelledby="History summary" class="ng-star-inserted">
        <thead _ngcontent-app-root-c204="" class="ng-star-inserted">
         <tr _ngcontent-app-root-c204="">
          <th _ngcontent-app-root-c204="">
           Temperature (° F)
          </th>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           Actual
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           Historic Avg.
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           Record
          </td>
          <!-- -->
          <td _ngcontent-app-root-c204="" style="height: 5px; width: 10px;">
           <svg _ngcontent-app-root-c204="" height="5" style="display: block;" width="10">
            <title _ngcontent-app-root-c204="">
             Polygon
            </title>
            <polygon _ngcontent-app-root-c204="" fill="#000000" points="0,5 5,0 10,5">
            </polygon>
           </svg>
          </td>
         </tr>
        </thead>
        <tbody _ngcontent-app-root-c204="" class="ng-star-inserted">
         <tr _ngcontent-app-root-c204="" class="ng-star-inserted">
          <th _ngcontent-app-root-c204="">
           High Temp
          </th>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           55
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           45
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           72
          </td>
          <!-- -->
         </tr>
         <tr _ngcontent-app-root-c204="" class="ng-star-inserted">
          <th _ngcontent-app-root-c204="">
           Low Temp
          </th>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           41
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           27
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           3
          </td>
          <!-- -->
         </tr>
         <tr _ngcontent-app-root-c204="" class="ng-star-inserted">
          <th _ngcontent-app-root-c204="">
           Day Average Temp
          </th>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           46.92
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           36
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           -
          </td>
          <!-- -->
         </tr>
         <!-- -->
        </tbody>
        <!-- -->
        <thead _ngcontent-app-root-c204="" class="ng-star-inserted">
         <tr _ngcontent-app-root-c204="">
          <th _ngcontent-app-root-c204="">
           Precipitation (Inches)
          </th>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           Actual
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           Historic Avg.
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           Record
          </td>
          <!-- -->
          <td _ngcontent-app-root-c204="" style="height: 5px; width: 10px;">
           <svg _ngcontent-app-root-c204="" height="5" style="display: block;" width="10">
            <title _ngcontent-app-root-c204="">
             Polygon
            </title>
            <polygon _ngcontent-app-root-c204="" fill="#000000" points="0,5 5,0 10,5">
            </polygon>
           </svg>
          </td>
         </tr>
        </thead>
        <tbody _ngcontent-app-root-c204="" class="ng-star-inserted">
         <tr _ngcontent-app-root-c204="" class="ng-star-inserted">
          <th _ngcontent-app-root-c204="">
           Precipitation (past 24 hours from 05:53:00)
          </th>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           0.00
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           0.09
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           -
          </td>
          <!-- -->
         </tr>
         <!-- -->
        </tbody>
        <!-- -->
        <thead _ngcontent-app-root-c204="" class="ng-star-inserted">
         <tr _ngcontent-app-root-c204="">
          <th _ngcontent-app-root-c204="">
           Dew Point (° F)
          </th>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           Actual
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           Historic Avg.
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           Record
          </td>
          <!-- -->
          <td _ngcontent-app-root-c204="" style="height: 5px; width: 10px;">
           <svg _ngcontent-app-root-c204="" height="5" style="display: block;" width="10">
            <title _ngcontent-app-root-c204="">
             Polygon
            </title>
            <polygon _ngcontent-app-root-c204="" fill="#000000" points="0,5 5,0 10,5">
            </polygon>
           </svg>
          </td>
         </tr>
        </thead>
        <tbody _ngcontent-app-root-c204="" class="ng-star-inserted">
         <tr _ngcontent-app-root-c204="" class="ng-star-inserted">
          <th _ngcontent-app-root-c204="">
           Dew Point
          </th>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           37.64
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           -
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           -
          </td>
          <!-- -->
         </tr>
         <tr _ngcontent-app-root-c204="" class="ng-star-inserted">
          <th _ngcontent-app-root-c204="">
           High
          </th>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           43
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           -
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           -
          </td>
          <!-- -->
         </tr>
         <tr _ngcontent-app-root-c204="" class="ng-star-inserted">
          <th _ngcontent-app-root-c204="">
           Low
          </th>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           30
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           -
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           -
          </td>
          <!-- -->
         </tr>
         <tr _ngcontent-app-root-c204="" class="ng-star-inserted">
          <th _ngcontent-app-root-c204="">
           Average
          </th>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           37.64
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           -
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           -
          </td>
          <!-- -->
         </tr>
         <!-- -->
        </tbody>
        <!-- -->
        <thead _ngcontent-app-root-c204="" class="ng-star-inserted">
         <tr _ngcontent-app-root-c204="">
          <th _ngcontent-app-root-c204="">
           Wind (MPH)
          </th>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           Actual
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           Historic Avg.
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           Record
          </td>
          <!-- -->
          <td _ngcontent-app-root-c204="" style="height: 5px; width: 10px;">
           <svg _ngcontent-app-root-c204="" height="5" style="display: block;" width="10">
            <title _ngcontent-app-root-c204="">
             Polygon
            </title>
            <polygon _ngcontent-app-root-c204="" fill="#000000" points="0,5 5,0 10,5">
            </polygon>
           </svg>
          </td>
         </tr>
        </thead>
        <tbody _ngcontent-app-root-c204="" class="ng-star-inserted">
         <tr _ngcontent-app-root-c204="" class="ng-star-inserted">
          <th _ngcontent-app-root-c204="">
           Max Wind Speed
          </th>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           14
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           -
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           -
          </td>
          <!-- -->
         </tr>
         <tr _ngcontent-app-root-c204="" class="ng-star-inserted">
          <th _ngcontent-app-root-c204="">
           Visibility
          </th>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           10
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           -
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           -
          </td>
          <!-- -->
         </tr>
         <!-- -->
        </tbody>
        <!-- -->
        <thead _ngcontent-app-root-c204="" class="ng-star-inserted">
         <tr _ngcontent-app-root-c204="">
          <th _ngcontent-app-root-c204="">
           Sea Level Pressure (Hg)
          </th>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           Actual
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           Historic Avg.
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           Record
          </td>
          <!-- -->
          <td _ngcontent-app-root-c204="" style="height: 5px; width: 10px;">
           <svg _ngcontent-app-root-c204="" height="5" style="display: block;" width="10">
            <title _ngcontent-app-root-c204="">
             Polygon
            </title>
            <polygon _ngcontent-app-root-c204="" fill="#000000" points="0,5 5,0 10,5">
            </polygon>
           </svg>
          </td>
         </tr>
        </thead>
        <tbody _ngcontent-app-root-c204="" class="ng-star-inserted">
         <tr _ngcontent-app-root-c204="" class="ng-star-inserted">
          <th _ngcontent-app-root-c204="">
           Sea Level Pressure
          </th>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           29.68
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           -
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           -
          </td>
          <!-- -->
         </tr>
         <!-- -->
        </tbody>
        <!-- -->
        <thead _ngcontent-app-root-c204="" class="ng-star-inserted">
         <tr _ngcontent-app-root-c204="">
          <th _ngcontent-app-root-c204="">
           Astronomy
          </th>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           Day Length
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           Rise
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           Set
          </td>
          <!-- -->
          <td _ngcontent-app-root-c204="" style="height: 5px; width: 10px;">
           <svg _ngcontent-app-root-c204="" height="5" style="display: block;" width="10">
            <title _ngcontent-app-root-c204="">
             Polygon
            </title>
            <polygon _ngcontent-app-root-c204="" fill="#000000" points="0,5 5,0 10,5">
            </polygon>
           </svg>
          </td>
         </tr>
        </thead>
        <tbody _ngcontent-app-root-c204="" class="ng-star-inserted">
         <tr _ngcontent-app-root-c204="" class="ng-star-inserted">
          <th _ngcontent-app-root-c204="">
           Actual Time
          </th>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           9h 33m
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           7:31 AM
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           5:04 PM
          </td>
          <!-- -->
         </tr>
         <tr _ngcontent-app-root-c204="" class="ng-star-inserted">
          <th _ngcontent-app-root-c204="">
           Civil Twilight
          </th>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           7:01 AM
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           5:34 PM
          </td>
          <!-- -->
         </tr>
         <tr _ngcontent-app-root-c204="" class="ng-star-inserted">
          <th _ngcontent-app-root-c204="">
           Nautical Twilight
          </th>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           6:28 AM
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           6:07 PM
          </td>
          <!-- -->
         </tr>
         <tr _ngcontent-app-root-c204="" class="ng-star-inserted">
          <th _ngcontent-app-root-c204="">
           Astronomical Twilight
          </th>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           5:56 AM
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           6:39 PM
          </td>
          <!-- -->
         </tr>
         <tr _ngcontent-app-root-c204="" class="ng-star-inserted">
          <th _ngcontent-app-root-c204="">
           Moon: waning gibbous
          </th>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           6:35 PM
          </td>
          <td _ngcontent-app-root-c204="" class="ng-star-inserted">
           8:52 AM
          </td>
          <!-- -->
         </tr>
         <!-- -->
        </tbody>
        <!-- -->
        <!-- -->
       </table>
       <!-- -->
       <!-- -->
      </div>
     </div>
    </lib-city-history-summary>

Parsing the HTML

The returned HTML might look overwhelming, but you can do a quick search for specific keywords in the HTML. For example, we know the table contains parameters of interest High Temp, Low Temp, Precipitation (past 24 hours), etc. Looking through the HTML for these keywords, we find that all of the data—Temperature (°F), Precipitation (Inches), Dew Point (°F)—in the table are contained under tbody, and that each row (e.g., High Temp) is tagged under tr with each data point (e.g., Actual, Historic Avg., and Record) tagged as td.

Saving HTML Data to CSV File

Now let’s try rendering that web page, selecting all of the temperature and precipitation data from the Summary table, and then writing that data into a csv file. For simplicity, we won’t pull *all* of the data from the web page (i.e., we’re not interested in dew point, wind speed, sea level pressure, or Sun rise/set times), but the code below can easily be altered to select those values as well, if that’s the data you’re interested in.

import os

from bs4 import BeautifulSoup
from selenium import webdriver

import time

search_url = 'http://www.wunderground.com/history/daily/KCHO/date/2020-12-31'

outfile = 'KCHO_2020-12-31.csv'

wunderground_page = rendering(search_url)

wunderground_soup = BeautifulSoup(wunderground_page, 'html.parser')

soup_container = wunderground_soup.find('lib-city-history-summary')
soup_data = soup_container.find_all('tbody', class_='ng-star-inserted')

with open(outfile, 'w') as f:
    # write column headers for each paramter into the file for later use
    f.write('date,'
            'actual_high_temp,histavg_high_temp,record_high_temp,'
            'actual_low_temp,histavg_low_temp,record_low_temp,'
            'actual_avg_temp,histavg_avg_temp,record_avg_temp,'
            'actual_precip,histavg_precip,record_precipitation\n')
    
    row = []
    for i, dat in enumerate(soup_data):
        for j, d in enumerate(dat.find_all('tr', class_='ng-star-inserted')): # loops through High Temp, Low Temp, etc.
            for k in d.find_all('td', class_='ng-star-inserted'): # loops through Actual, Historic Avg., Record
                tmp = k.text
                tmp = tmp.strip('  ') # remove any extra spaces
                        
                row.append(tmp)

    f.write('2020-12-31,') # write the date of the recorded data into the file
    f.write(','.join(row[:12])) # write just the temperature and precipitation data into the file
    f.write('\n') # new line, in case you want to append more rows to the same file later on

You should now have a single csv file containing the data from the Summary table from Weather Underground for weather observations taken at the CHO on December 31, 2020:

Automating for Scraping Multiple URLs

Let’s say we want to scrape the information contained in the Summary table from Weather Underground pages for weather observations taken at *any airport for an entire year*.

To do this, we need to convert the code above to its own function that can render the web page for any airport (given its corresponding airport code) and loop through multiple web pages for a range of input dates:

def scrape_airport_weather(station, start_date, end_date):
    
    # make a new directory for each airport that you scrape data for, to keep files separate and organized   
    if not os.path.exists(station):
        os.mkdir(station) 
        
    # search URL that can be formatted to find the webpage for any airport on any observation date     
    search_url = 'http://www.wunderground.com/history/daily/{}/date/{}-{}-{}' 
    
    # csv file name that can be formatted for any airport and any year
    outfile = '{}/{}_{}.csv'.format(station, station, start_date.year)
    
    # if the csv file does not exist, write, if it does exist, append
    if not os.path.exists(outfile):
        mode = 'w'
    else:
        mode = 'a'
        
    with open(outfile, mode) as f:
        # write column headers for each paramter into the file for later use
        if mode == 'w':
            f.write('date,actual_high_temp,histavg_high_temp,record_high_temp,'
                    'actual_low_temp,histavg_low_temp,record_low_temp,'
                    'actual_avg_temp,histavg_avg_temp,record_avg_temp,'
                    'actual_precip,histavg_precip,record_precipitation\n')
    
        # while loop continues until it reaches the given end date
        while start_date != end_date:
            
            # format the search URL for the given airport and observation date 
            format_search_url = search_url.format('K'+station,
                                                  start_date.year,
                                                  start_date.month,
                                                  start_date.day)
        
            wunderground_page = rendering(format_search_url)
        
            wunderground_soup = BeautifulSoup(wunderground_page, 'html.parser')
            soup_container = wunderground_soup.find('lib-city-history-summary')
            soup_data = soup_container.find_all('tbody', class_='ng-star-inserted')
        
            row = []
            for i, dat in enumerate(soup_data):
                for j, d in enumerate(dat.find_all('tr', class_='ng-star-inserted')): # loops through High Temp, Low Temp, etc.
                    for k in d.find_all('td', class_='ng-star-inserted'): # loops through Actual, Historic Avg., Record
                        tmp = k.text
                        tmp = tmp.strip('  ') # remove any extra spaces
                        
                        row.append(tmp)
                        
            f.write('{}-{}-{},'.format(start_date.year, start_date.month, start_date.day)) # write the observation date into the file
            f.write(','.join(row[:12]))# write just the temperature and precipitation data into the file
            f.write('\n') # new line, in case you want to append more rows to the same file later on
        
            start_date += timedelta(days=1) # go to next date, i.e., next URL
            

Now that we have our new scrape_airport_weather function defined, let’s test it!

In this example, we want to scrape data for weather observations taken at the Charlottesville Albemarle Airport (CHO) for all of 2020. To do this, we simply need to define the station, and start_date, end_date parameters and run the function.

import os

from bs4 import BeautifulSoup
from selenium import webdriver

from datetime import datetime, timedelta

station = 'CHO'
start_date = datetime(year=2020, month=1, day=1)
end_date = datetime(year=2021, month=1, day=1)

scrape_airport_weather(station, start_date, end_date)

This will take a while to run completely, but you should end up with a csv file containing 366 dated rows (2020 was a leap year!), each with temperature and precipitation data for that day. You can find a copy of this file at https://static.lib.virginia.edu/statlab/materials/data/CHO_2020.csv.

Now, you can do some fun visualizations with the data! We’ll use the pandas package to work with the data here. If you need quick tutorial or a refresher on pandas, check out this Getting Started with pandas in Python article. Note, this is just one example visualization that you can produce with these data; we won’t go into detail about working with Weather Underground data here. There are *plenty* of other interesting analyses that you can do with the long time baseline of weather history data provided by Weather Underground though!

from datetime import datetime

import matplotlib as mpl
import matplotlib.pyplot as plt
from matplotlib.dates import DateFormatter
import numpy as np
import pandas as pd

df = pd.read_csv('CHO/CHO_2020.csv', # read the data into a pandas dataframe
                 parse_dates=['date'], # parse and index the columns by the date
                 index_col= ['date'])
    
df = df.drop(['record_avg_temp', 'record_precipitation'], axis=1) # these columns are empty, drop them from the dataframe

df = df.replace('--',np.nan) # replace any other empty cells with NaNs

df = df.dropna(subset=['actual_high_temp']) # due to hardware (i.e., weather station) failures, a few days have all data set as NaNs, drop these rows

df['record_low_temp'] = pd.to_numeric(df['record_low_temp'], errors='coerce') # handles converting strings containing negative numbers (i.e., '-') to floats
    
df.loc[df['actual_low_temp'] == 0] = np.nan # due to hardware (i.e., weather station) failures, a few days have Actual Low Temp set to 0, in the middle of summer, replace with NaN
df['actual_low_temp'] = df['actual_low_temp'].interpolate() # interpolate over NaNs in Actual Low Temp

df['month'] = df.index.month # add month column
month_avg = df.groupby('month').agg(['mean','median','sum']) # calculate mean, median, and total temperatures and precipitation for each month

fig, ax = plt.subplots(2, 1, figsize=(10, 8))

# first subplot
# plot monthly average Actual Avg Temp and average Historical Avg
ax[0].plot(month_avg.index, month_avg['actual_avg_temp']['mean'], color='dodgerblue', linewidth=3.0, label='Actual Avg.')
ax[0].plot(month_avg.index, month_avg['histavg_avg_temp']['mean'], color='salmon', linewidth=3.0, ls='--', label='Historical Avg.')

# second subplot
# plot monthly total Actual Precipitation and total Historical Avg Precipitation
ax[1].plot(month_avg.index, month_avg['actual_precip']['sum'], color='seagreen', linewidth=3.0, label='Actual Total')
ax[1].plot(month_avg.index, month_avg['histavg_precip']['sum'], color='darkorange', linewidth=3.0, ls='--', label='Historical Total')

# set xtick labels as month names on both subplots
ax[0].set_xticks(range(1, 13, 1)) 
ax[0].set_xticklabels([datetime(year=2016, month=x, day=1).strftime('%b')
                                     for x in range(1, 13, 1)], fontsize=12)

ax[1].set_xticks(range(1, 13, 1)) 
ax[1].set_xticklabels([datetime(year=2016, month=x, day=1).strftime('%b')
                                     for x in range(1, 13, 1)], fontsize=12)
# set axis label
ax[1].set_xlabel(r'Month', fontsize=14)

# set ylim and ytick labels on first subplot with degree format
ax[0].set_ylim(10,100)  
ax[0].set_yticks(range(0, 111, 10))
ax[0].set_yticklabels([r'{}$^\circ$'.format(x)
                                     for x in range(0, 111, 10)], fontsize=12)
# set axis label
ax[0].set_ylabel(r'Temperature ($^\circ$F)', fontsize=14)

# set ylim and axis label on second subplot
ax[1].set_ylim(0,7)
ax[1].set_ylabel(r'Precipitation (inches)', fontsize=14)

# include legends
ax[0].legend()
ax[1].legend()

# plot title
plt.suptitle('Charlottesville, VA (CHO), 2020\n', fontsize=16)

# formatting for nice saved figure
plt.tight_layout()

# save figure to same directory as csv file
plt.savefig('CHO/CHO_avgtemp_totalprecip.pdf', dpi=300)

Summary

In this article, we’ve used the Python BeautifulSoup and Selenium libraries to extract and parse data from an example web page. We’ve also shown how to automate this process for scraping data from many web pages at one time. Now that you understand the basics of web scraping with Python, these functions and methods will hopefully be enough to get you *started* working on scraping data from any given website. For more detailed information about the BeautifulSoup and Selenium packages and built-in functions, please see the available documentation.

For questions or clarifications regarding this article, contact the UVA Library StatLab: statlab@virginia.edu

View the entire collection of UVA Library StatLab articles.

Hannah Lewis
StatLab Associate
University of Virginia Library
February 15, 2021