Pandas Exercises - View and Explore Data - Part 1

This is Pandas exercise about data analysis and exploration. It's the first exercise from a series (check the end of the post for more details).

You can test knowledge in several areas:

• initial data analysis
• "viewing data" - as advertised on the Pandas page: Viewing data
• summarizing data

We can think of data exploration as getting to know your food. I think that most of us prefer to see, taste and smell the food - before eating. For me it's similar to data.

Setup

In this tutorial we will work with the following dataset: The Movies Dataset.

The dataset contains data for 45,000 movies. Data points include cast, crew, plot keywords, revenue, release dates, languages, production companies, countries, vote averages and more.

We will focus on the file: movies_metadata.csv. Data from the CSV file can be read as follows:

import pandas as pd
df = pd.read_csv('movies_metadata.csv', low_memory=False)

1: How many rows and columns?

Usually the first thing which I do is to get the DataFrame dimension.

It will give us a rough idea about the size of the DataFrame. Why rough? Because you may have nested hierarchical data - multilevel columns.

df.shape

This will give us:

(45466, 24)

This might save you time and effort. Sometimes it is good to start with smaller data samples.

Bonus question 1

Can you guess how many rows and columns?

df.shape[0]

Bonus Question 2

Can you name synonyms or other names of columns and rows?

2: View the top and bottom rows

How can we see the first N records of the dataset? What about the last N rows?

df.head()

df.tail()

Bonus question 3

How can we see bottom and top rows simultaneously in Pandas?

import numpy as np
df.iloc[np.r_[0:5, -5:0]]

Did you solve that one? Good job, it was a tricky one!

Bonus question 4

Do you know why there are parentheses and sometimes not?
like:

df.head()

and:

df.shape

head()

shape

3: Display the index and columns

Can you display the column names? What about index/row names?

df.columns

df.index

We will get about the columns:

Index(['adult', 'belongs_to_collection', 'budget', 'genres', 'homepage', 'id',
   'imdb_id', 'original_language', 'original_title', 'overview',
   'popularity', 'poster_path', 'production_companies',
   'production_countries', 'release_date', 'revenue', 'runtime',
   'spoken_languages', 'status', 'tagline', 'title', 'video',
   'vote_average', 'vote_count'],
  dtype='object')

and for the index:

RangeIndex(start=0, stop=45466, step=1)

4: Generate descriptive statistics

What about getting a quick statistical summary for a DataFrame in Pandas? This would give us valuable information about:

• count
• mean
• standard deviation
• percentile

df.describe()

This would give us:

The method will analyze both numeric and object series, as well as DataFrame column sets of mixed data types.

5: What are the dtypes?

What are the data types of the columns in a DataFrame. Data types might be several kinds like:

• numeric
• datetime
• string/object
• nested data - dictionary, list or json
• categorical

It can help us when we work with our data. Think of it like knowing the OS of the computer. Working on Linux differs from Windows? At least for me.

So how to get data types?

df.dtypes

This would give us:

adult                     object
belongs_to_collection     object
budget                    object
genres                    object
homepage                  object
id                        object
imdb_id                   object
original_language         object
original_title            object
overview                  object
popularity                object
poster_path               object
production_companies      object
production_countries      object
release_date              object
revenue                  float64
runtime                  float64
spoken_languages          object
status                    object
tagline                   object
title                     object
video                     object
vote_average             float64
vote_count               float64
dtype: object

6: Transpose rows

How can we see the first N records of the dataset - transposed?

I like to transpose rows when I'm working with few rows with many columns.

The first few rows looks like that(transposed):

And transposing rows is possible by:

df.head(2).T

Which is the alias of method transpose(). This is another thing to have in mind working with Pandas - aliases.

7: Sort values

Can you get the top most rated movies from this dataset? To do so we will need to sort data by a column.

So the last exercise is to get the names and number of votes for the most voted movies in that dataset.

df.sort_values(by='vote_count', ascending=False)[['title', 'vote_count']].head()

Bonus question 5

Can you find the top 7 values for the numerical columns? Return the result as a DataFrame:

Can you optimize the proposed solution below?

from pandas.api.types import is_numeric_dtype

dfs = []

for col in df.columns:
    top_values = []
    if is_numeric_dtype(df[col]):
        top_values = df[col].nlargest(n=7)
        dfs.append(pd.DataFrame({col: top_values}).reset_index(drop=True))
pd.concat(dfs, axis=1)

Were you able to solve it? Well done!

Bonus question 6

Method describe() is good for initial exploratory data analysis. For more details like:

• type
• unique values
• missing values
• most frequent values
• histogram and more

Can you research and find Python library which can do "serious exploratory data analysis"?

ProfileReport(df, title="Pandas Profiling Report")

Conclusion

Congratulations! You have passed the first set of Pandas exercises. We've tasted the movie dataset.

Now you will have a better understanding of your data and how to do initial data analysis.

In this post, we learned about initial data analysis in Pandas. Most popular methods in Pandas for "viewing data".

We saw some advanced techniques and tricks very useful when data is touched for the first time.

Pandas exercises for beginners

Pandas exercises will be structured as a sample project plus exercises - a place where we can practice data science.

The goal is to test your knowledge and learn in 3 main areas:

• data cleaning
• data analysis
• data collection

The areas above are often reported as problematic in data science.