How to Create a Bag of Words in Pandas Python

In this short guide, I'll show you how to create a bag of words with Pandas and Python. You can find a example of bag of words using the sklearn library:

from sklearn.feature_extraction.text import CountVectorizer
import pandas as pd

text = ['The fox jumps over the lazy dog.',  'Dog and fox are lazy!']
data = {'text': text}
df = pd.DataFrame(data)

vectorizer = CountVectorizer()
bow = vectorizer.fit_transform(df['text'])
print(bow)

count_array = bow.toarray()
features = vectorizer.get_feature_names()
df = pd.DataFrame(data=count_array, columns=features)

Below you can find the result of the code:

In the next steps I'll explain the process in more detail.

What is a Bag of Words?

A bag of words is a way to represent text data in tabular form as numerical features.

You can also find a quick solution only with Pandas and Python below:

import pandas as pd
from collections import Counter

text = ['Periods of rain', 'Mostly cloudy, a little rain', 'Mostly cloudy', 'Intervals of clouds and sun', 'Sunshine and mild']

df = pd.DataFrame({'text': text})
pd.DataFrame(df['text'].str.split().apply(Counter).to_list())

The input DataFrame is:

The output bag of words represented again as DataFrame:

The diagram below shows how to create a bag of words from multiple documents. Each item in the list is considered as separate document.

Setup

First lets create a sample DataFrame for this example:

from sklearn.feature_extraction.text import CountVectorizer
import pandas as pd

text = ['Periods of rain', 'Mostly cloudy, a little rain', 'Mostly cloudy', 'Intervals of clouds and sun', 'Sunshine and mild']
data = {'text': text}
df = pd.DataFrame(data)

result:

We are going to import CountVectorizer from sklearn.

Step 1: Initialize the vectorizer

Next we are going to initialize the vectorizer of sklearn:

vectorizer = CountVectorizer()

lowercase

At this step we can do customization like on/off of lowercase conversion:

vectorizer = CountVectorizer(lowercase=False)

Stop words

or adding custom stop words for the bag of words:

vectorizer = CountVectorizer(stop_words= ['on', 'off'])

even using stop words based on languages:

coun_vect = CountVectorizer(stop_words='english')

max_df / min_df

The abbreviation df in max_df / min_df stands for document frequency.

• max_df = 0.75 - ignore terms that appear in more than 75% of the documents.
• max_df = 0.5 - ignore terms that appear in less than 50% documents.

When using a float in the range [0.0, 1.0] they refer to the document frequency.

They can be used also in sense of max_df = 10 - which means ignore terms that appear in less than 10 documents

coun_vect = CountVectorizer(max_df=1)

Step 2: Fit and transform the text data

Next step is to fit and transform the text data to create a bag of words:

bow = vectorizer.fit_transform(df['text'])

This creates a bag of words from the DataFrame column like:

(0, 8)    1
(0, 7)    1
(0, 9)    1
(1, 9)    1
(1, 6)    1
(1, 2)    1
(1, 4)    1
(2, 6)    1
(2, 2)    1
(3, 7)    1
(3, 3)    1
(3, 1)    1

This is a sparse matrix, where:

• each row represents a document
• each column represents a word
• the values in the matrix represent the number of times that word appears in that document.

So in (0, 8) 1 we have:

• 0 is the number of the document
  • first document
• 8 number of the feature
  • periods
• 1 - is the count
  • 1 occurrence

Step 3: Get features names and counts

We can get the word list from the vectorizer, by calling the method get_feature_names():

# get count array and features
count_array = bow.toarray()
features = vectorizer.get_feature_names()

# create DataFrame as bag of words
df = pd.DataFrame(data=count_array, columns=features)

This will create a DataFrame where:

• each column is a word
• row represents the documents
• values are number of times each word is present

Normalize the bag of words

Alternatively, we can also use TfidfVectorizer from scikit-learn that creates a bag of words by:

• first counting the frequency of each word in each document
• then normalizing the resulting counts by dividing by the total number of words in the document.

The resulting values are the term frequency-inverse document frequency (TF-IDF) values:

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd

text = ['small dog', 'cute cat', 'cute dog', 'cat']
data = {'Text':text}
df = pd.DataFrame(data)

vectorizer = TfidfVectorizer()
bow = vectorizer.fit_transform(df['Text'])
print(bow)

result:

(0, 2)    0.6191302964899972
(0, 3)    0.7852882757103967
(1, 0)    0.7071067811865475
(1, 1)    0.7071067811865475
(2, 1)    0.7071067811865475
(2, 2)    0.7071067811865475
(3, 0)    1.0

Getting bag of words as a DataFrame with normalized values:

count_array = bow.toarray()
features = vectorizer.get_feature_names()
df = pd.DataFrame(data=count_array, columns=features)

Conclusion

To summarize, in this article, we've seen examples of bags of words. We've briefly covered what a bag of words is and how to create it with Python, Pandas and scikit-learn.

And finally, we've seen how to normalize a bag of words with scikit-learn.