In this tutorial, I'll show you how to compare titles and URLs with Pandas and Python. We are going to use several Pandas functions and techniques in order to find problematic data.
To start, here is the initial Data and what is our final goal:
|A Beginner’s Guide to Word Embedding with Gensim Word2Vec Model||https://towardsdatascience.com/a-beginners-guide-to-word-embedding-with-gensim-word2vec-model-5970fa56cc92|
|Hands-on Graph Neural Networks with PyTorch & PyTorch Geometric||https://towardsdatascience.com/hands-on-graph-neural-networks-with-pytorch-pytorch-geometric-359487e221a8|
|How to Use ggplot2 in Python||https://towardsdatascience.com/how-to-use-ggplot2-in-python-74ab8adec129|
|Databricks: How to Save Files in CSV on Your Local Computer||https://towardsdatascience.com/databricks-how-to-save-files-in-csv-on-your-local-computer-3d0c70e6a9ab|
|A Step-by-Step Implementation of Gradient Descent and Backpropagation||https://towardsdatascience.com/a-step-by-step-implementation-of-gradient-descent-and-backpropagation-d58bda486110|
Usually the slug of the URL is composed from the title after several transformations.
Below you can find example pair of URL and title:
- A Beginner’s Guide to Word Embedding with Gensim Word2Vec Model
Our goal is to find all articles which have mismatch between the title and the slug like:
- Faster Training for Efficient CNNs
The difference above is
In the next section, I’ll review the steps to compare and validate the title against the URL of a given article.
Step 1: Read the data and install slugify
In this tutorial we are going to use data available from Kaggle. You can find it also in the repository related to the notebook(in Resources):
import pandas as pd df = pd.read_csv('../data/medium_data.csv.zip') df
If you like to learn more about how to read Kaggle as a Pandas DataFrame check this article: How to Search and Download Kaggle Dataset to Pandas DataFrame
We need to install Python library called:
pip install slugify
Step 2: Convert title to slug with slugify
Next step is to prepare the titles in order to compare them with the URLs. For this purpose we are going to use library
from slugify import slugify df['title_to_url'] = df['title'].fillna('').apply(lambda x: slugify(x))
After this operation we will have the next data:
|A Beginner’s Guide to Word Embedding with Gensim Word2Vec Model||https://towardsdatascience.com/a-beginners-guide-to-word-embedding-with-gensim-word2vec-model-5970fa56cc92||a-beginners-guide-to-word-embedding-with-gensim-word2vec-model|
|Hands-on Graph Neural Networks with PyTorch & PyTorch Geometric||https://towardsdatascience.com/hands-on-graph-neural-networks-with-pytorch-pytorch-geometric-359487e221a8||hands-on-graph-neural-networks-with-pytorch-pytorch-geometric|
|How to Use ggplot2 in Python||https://towardsdatascience.com/how-to-use-ggplot2-in-python-74ab8adec129||how-to-use-ggplot2-in-python|
|Databricks: How to Save Files in CSV on Your Local Computer||https://towardsdatascience.com/databricks-how-to-save-files-in-csv-on-your-local-computer-3d0c70e6a9ab||databricks-how-to-save-files-in-csv-on-your-local-computer|
|A Step-by-Step Implementation of Gradient Descent and Backpropagation||https://towardsdatascience.com/a-step-by-step-implementation-of-gradient-descent-and-backpropagation-d58bda486110||a-step-by-step-implementation-of-gradient-descent-and-backpropagation|
Step 3: Find all rows where the slugified titles is not in the URL
In this step we are going to compare the slugified titles and the URLs. This is a pretty generic comparison and we might miss some false positives.
We are going to create a new column -
bad_title where all values will be set to
Only if the slugified title is not part of the URL then we are going to set it to
df['bad_title'] = False for row in df[df.title.notna()].iterrows(): title_to_url_temp = row.title_to_url.lower() title_temp = row.title.replace('-', '').replace(' ', '').lower() if not title_to_url_temp in row['url']: df.loc[row, 'bad_title'] = True
In total - 1051 results like:
|<em class="markup--em markup--h3-em">What I Learned from (Two-time) Kaggle Grandmaster Abhishek Thakur</em>||https://towardsdatascience.com/what-i-learned-from-abhishek-thakur-4b905ac0fd55||em-class-markup-em-markup-h3-em-what-i-learned-from-two-time-kaggle-grandmaster-abhishek-thakur-em|
|Faster Training for Efficient CNNs||https://towardsdatascience.com/faster-training-of-efficient-cnns-657953aa080||faster-training-for-efficient-cnns|
|Buyers beware, fake product reviews are plaguing the internet. How Machine Learning can help to spot them.||https://towardsdatascience.com/buyers-beware-fake-product-reviews-are-plaguing-the-internet-cfc599c42b6b||buyers-beware-fake-product-reviews-are-plaguing-the-internet-how-machine-learning-can-help-to-spot-them|
You can use different techniques in order to get better results like:
- replace hyphens from both sides
- convert to lower case
Step 4: Extract the slug from the URL and compare to title
In this step we are going to extract the slug from the URL. This is not a simple operation and might depend on many factors.
We can't replace the domain or extract the slug with simple split:
df['url'].str.split('/', expand = True)
As you can see there are multiple domains and formats.
What we are going to do is extracting the slug by chaining several Pandas operations:
df['url'].str.rsplit('/').str[-1].str.rsplit('-', 1, expand=True)
**How does it work? **
We are using the method
rsplit in order to split on
[https:, , towardsdatascience.com, how-to-use-ggplot2-in-python-74ab8adec129]
str[-1] in order to get the part after the last backslash:
In order to get rid of that final identifier -
-74ab8adec129 we will use
rsplit again -
.rsplit('-', 1, expand=True)
Now we can compare the slug from the URL with the modified title:
df[df['url'].str.rsplit('/').str[-1].str.rsplit('-', 1, expand=True) != df['title_to_url']]
The result is pretty similar to the one from Step 3 - but we have more results - 1119.
But there is a difference between both methods. This step is more accurate and will report cases like:
Living as an Empath — w
while Step 3 will miss this result since the
title_to_url is part of the URL.