Web Scraping with Python (BeautifulSoup and Pandas)

Imagine that everyday at a certain time of the day you have a task to manually visit a certain website to check the Forex rates of specified currencies against your local currency. For instance, assume that your local currency is UGX, and the foreign currencies are AUD, GBP, EUR, NZD, and USD.

There is no doubt that the above task can be time-consuming even just the way it is. But now imagine that in addition to visiting the website, you have additional tasks to get the rates, save them in a local file on top the historical rates, and plot a chart to see the trend. This certainly becomes more time-consuming, and even complex.

What if I tell you that this series of tasks can be automated and delivered right to your mailbox through web scraping without your intervention? Just imagine how much time you would save everyday, and the time saved can be used more productively elsewhere. The reality of life is that most people, in spite of technological advances, still go through the manual way of doing things. And this is just an example. One excellent use-case is tracking daily COVID-19 rates.

Enter web scraping with Python. I recently developed two similar web scraping systems for Forex rates and Weather (using Open Weather site). Both websites offer very accurate global data, with up to one hour update intervals.Of course, there are considerations to make before you web-scrape a website, otherwise you could be rocking the boat (legally). The bottom line is to check and confirm that web-scraping is allowed on the website before you do anything.

For web scraping, we can use any of the following Python libraries: Selenium, BeautifulSoup, or Pandas. Having used Selenium for interactive web automation before, this time I chose to use both BeautifulSoup and Pandas separately, but obviously they return same results. In this post, we are not using Selenium, but for those who may not know, Selenium allows you to automate actions on a website. For instance, you can write a Selenium script to log into a web page, click a link for reports, click another oink for emailing the report to you ... but that is for another day.

Using BeautifulSoup and Pandas (each in its own right), below are some screenshots of the results. As usual, you can find the code in my GitHub repo.

 

Table of raw Forex rates data scraped from the website

Table of processed Forex rates data ready for plotting

 

Chart showing trends of Forex rates using MatplotLib