Instantly Turn Web Pages into Beautiful Dashboards with Python   

Intro

I recently had the need to monitor multiple web pages and filter information from them. This is a rather simple task but it's time consuming and error prone. Every time I do it, it takes time to find the right data, analyze it and save it. In addition, often I would like to monitor multiple web sources simultaneously without distraction in a homogeneous style.

I did research for this problem but nothing was close enough to my needs. What I need is to build a beautiful dashboard from multiple web sites. In the past, I was using CRON jobs, Python scripts and Jupyter notebooks to collect data in one place.

Finally I found a better solution which extracts data from web pages and turns them into a dashboard. You will also learn how to turn any Jupyter Notebook into a dashboard in seconds.

Setup

Prerequisite

Voilà

Voilà is a Python package which turns Jupyter notebooks into standalone web applications. It can be used as a standalone app with the new Jupyter kernel or inside Jupyter.

It can be installed by:

pip install voila

Voilà provides a JupyterLab extension that displays a Voilà preview of your Notebook in a side-pane. To install the extension from source, run the following command.

jupyter labextension install @voila-dashboards/jupyterlab-preview

voila-gridstack

voila-gridstack is gridstack-based template for Voilà.

pip install voila-gridstack

1. Scraping with Pandas

Our goal is to make quick and easy extracting data from multiple sources into a single dashboard. There are many ways to scrape data with Python. The simplest and easiest way to scrape tabular data is by using Pandas. We will cover two different options:

• basic extraction
• adding user agent

Pandas scrape tables

Pandas offers handy method pandas.read_html which reads HTML tables into a list of DataFrames. By default extracts all tables from a given URL:

import pandas as pd

url_cur = 'https://en.wikipedia.org/wiki/List_of_countries_by_forest_area'
df_ls = pd.read_html(url_cur)
df_ls[0]

Pandas read_html + user agent

Some websites with cause Pandas method read_html() to return:

HTTPError: HTTP Error 403: Forbidden

In order to solve this problem we will add user agent and use package requests:

import pandas as pd
import requests

url_cur = 'https://en.wikipedia.org/wiki/List_of_countries_by_forest_area'

header = {
  "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36",
  "X-Requested-With": "XMLHttpRequest"
}

r = requests.get(url_cur, headers=header)

ls_cur = pd.read_html(r.text)

Adding a user agent as headers solves the HTTPError: HTTP Error 403: Forbidden returned from Pandas.

2. Styling and plotting

Python and Pandas offer multiple ways for styling and creating beautiful visualizations. For simplicity we will mention only two in this article.

DataFrame as heatmap

The first approach is to use method `.style.background_gradient() that can be used create nice looking heatmaps:

df.style.background_gradient(cmap='Greens', subset=str_cols)\
	.background_gradient(cmap='Blues', subset='Price')

To find more about it check out: How to Display Pandas DataFrame As a Heatmap

Quick visualization with seaborn

The second way for creating quick and nice visualizations is by using libraries like seaborn. The advantage of seaborn is simplicity of usage and diversity of plots. To learn more about different visualization options and styles refer to: Pandas Visualization Cheat Sheet

3. Turn Jupyter Notebook into Dashboard

Once we have data collected and all visualizations are ready for use - we can start building our dashboard.

Open voila-gridstack editor

To open voila-gridstack editor we have two options in JupyterLab. The first way is by:

• right clicking the notebook
• Open with
• Voilà Gridstack

Alternatively we can use a button on the right top of an opened notebook.

drag-and-drop cells

We can see the notebook and the editor side by side. We can select a cell and move it to the voila-gridstack editor. Then we can resize or move the cell in the voila-gridstack editor. The image below shows the process.

Open new voila window

Once we are happy with the outlook of the dashboard we can save it. Finally we can open it as a separate window or use it as a standalone application.

Conclusion

There are many options for building a dashboard within the Python ecosystem. Voila offers a quick and easy way to render Jupyter notebooks as a dashboard . In my experience, Voila is the best choice for beginners and people with medium experience.

I hope this article will be a useful guide for people interested in building their own dashboards for their practical problems. Feel free to leave a comment to ask a question.

Resources

• Voila - Rendering of live Jupyter notebooks with interactive widgets.
• voila-gridstack - A gridstack-based template for voila-gridstack