# DataScientYst - Data Science Simplified > DataScientYst - Data Science Tutorials, Exercises, Guides, Videos with Python and Pandas Public Ghost content for AI and LLM tooling. Use `/llms-full.txt` for consolidated page and post context. Append `.md` to any post or page URL to get the content in Markdown (for example, `/example-post.md`). ## Pages - [About us.](https://datascientyst.com/about.md) - I'm Johny D K, a software engineer, data scientist and dreamer. Hey there! I'm a data scientist from East Europe. This is my place on the web for data science projects, tutorials and guides. I love facing data challenges and finding hidden insights and relations. I've started as a database administ… - [Data For AI](https://datascientyst.com/data-for-ai.md) - 1. What is Data for AI? High-quality data is the foundation of effective AI. From customer interactions and operational records to images, documents, and sensor data, AI systems rely on diverse and accurate information to learn, make predictions, and generate insights. The better the data, the more… - [Data Science Challenges](https://datascientyst.com/data-science-challenges.md) - Solving Data Science challenges are a great way to practice and improve your analytical skills. Data Science challenges are small projects that exercise your skills exploring, processing and analyzing data. In this article, we'll add Data Science challenges that you can solve to test and develop yo… - [Data Science Tutorial in Python and Pandas](https://datascientyst.com/data-science-tutorial-python-pandas.md) - Pandas is a mature, powerful, open source and highly flexible Python library focused on data analysis and manipulation. Pandas is one of the most popular tools for doing Data Science Some of the key benefits of Pandas are that: * it's easy to start and learn * open source * efficient for variety of… - [Privacy Policy](https://datascientyst.com/privacy.md) - This Privacy Policy explains how information is collected, used and disclosed by DataScientyst with respect to user’s access and use of our service through the application (Referred to below as “DataScientyst”). 1. Information collection When using DataScientyst, we ask certain information from you… - [What can I make for you, today?](https://datascientyst.com/service.md) - Does this sound familiar? * Your data is scattered across multiple systems, making it difficult to trust your reports and make confident business decisions. * You're planning a data migration, but you're worried about data loss, downtime, or costly mistakes. * Poor data quality—duplicates, missing… - [Thank you very much!](https://datascientyst.com/thank-you.md) - All the content on the DataScientYst.com will be available for free for everyone. You can support our work by subscribing for the paid membership. This will allow us creating new content and fresh ideas related to data science. Thank You! P.S. Your opinion is important for us. Feel free to reach ab… ## Posts - [How to Filter a DataFrame for Numeric Values in Pandas](https://datascientyst.com/how-to-filter-a-dataframe-for-numeric-values-in-pandas.md) - To filter a DataFrame for numeric values in Pandas we can: (1) Use str.isnumeric() with boolean indexing df[df['col'].str.isnumeric()] (2) Use pd.to_numeric() with errors='coerce' df[pd.to_numeric(df['col'], errors='coerce').notna()] (3) Use regular expressions with str.match() df[df['col'].str.mat… - [Exploring Economic Data with FRED: A Powerful Source for Data Science Projects](https://datascientyst.com/exploring-economic-data-with-fred-a-powerful-source-for-data-science-projects.md) - FRED (Federal Reserve Economic Data) is a free database run by the Federal Reserve Bank of St. Louis. It holds hundreds of thousands of economic time series from national, international, public, and private sources, plus tools to chart, customize, and export the data. No account needed to browse. �… - [A Beginner's Data Science Project: Olympic Medal Winners' Age by Sport](https://datascientyst.com/a-beginners-data-science-project-olympic-medal-winners-age-by-sport.md) - [How to Get the Last Column After `str.split()` in a Pandas DataFrame](https://datascientyst.com/how-to-get-the-last-column-after-str-split-in-a-pandas-dataframe.md) - In this short post we will see how to split a column into multiple parts and then extract only the last component or the last not null value. This is common with file paths, URLs, codes, delimiter-separated strings and dirty data. For example, if your column contains: * "user/home/file.txt", you mi… - [How to Compare Pandas DataFrames When NaNs Are Present](https://datascientyst.com/how-to-compare-pandas-dataframes-when-nans-are-present.md) - When working with pandas, comparing DataFrames that contain NaN values can be confusing and error prone. By defaultin Python, NaN is not equal to NaN in standard element-wise comparisons, which often leads to unexpected results. Sample data: import numpy as np import pandas as pd df1 = pd.DataFrame… - [How to Extract Capital Words from a Pandas DataFrame](https://datascientyst.com/how-to-extract-capital-words-from-a-pandas-dataframe.md) - If you're working with textual data in a Pandas DataFrame and want to find all words written in uppercase, there are several simple ways to do it using Python. A "capital word" here means a word where every letter is uppercase (like JAVA or PYTHON). This is useful when cleaning data, detecting acro… - [How to Floor a Date to the First Date of That Month in Pandas](https://datascientyst.com/how-to-floor-a-date-to-the-first-date-of-that-month-in-pandas.md) - Working with dates in Pandas often requires precise manipulations, such as flooring a date to the beginning of its month. This is particularly useful when aggregating data monthly or standardizing timestamps for reporting. In this short guide, we'll explore several efficient methods to achieve this… - [How to Compare Each Value in Pandas Column to All Subsequent Values](https://datascientyst.com/how-to-compare-each-value-in-pandas-column-to-all-subsequent-values.md) - Learn how to compare every value in a pandas DataFrame column with all following values efficiently. Sample Data import pandas as pd val = [16, 19, 15, 19, 15] df = pd.DataFrame({'val': val}) val 0 16 1 19 2 15 3 19 4 15 1. Compare with Subsequent Values Using apply Create a new column with lists o… - [How to Insert Item at Beginning of Pandas Series](https://datascientyst.com/how-to-insert-item-at-beginning-of-pandas-series.md) - When working with Pandas Series, you may need to add an item at the beginning rather than at the end. While Pandas doesn't have a built-in prepend method, there are several effective ways to accomplish this task. In this short guide, you'll see how to insert an item at the beginning of a Pandas Ser… - [How to Validate Domain Name in Pandas and Python](https://datascientyst.com/how-to-validate-domain-name-in-pandas-python.md) - When working with user input, web scraping, or data validation, you often need to verify whether a string represents a valid domain name. Python offers several approaches to accomplish this task, from simple regex patterns to specialized libraries. In this guide, you'll learn different methods to v… - [How to Replace a Header with the Top Row in a Pandas DataFrame](https://datascientyst.com/how-to-replace-a-header-with-the-top-row-in-a-pandas-dataframe.md) - In this post you can learn how to replace a header with the top row in a Pandas DataFrame 1: Replace the Header with the First Row If your DataFrame has no header and the first row contains the correct column names, you can promote it to the header: import pandas as pd # Example DataFrame without a… - [Pandas FutureWarning: Passing literal html to 'read_html' is deprecated and will be removed in a future version](https://datascientyst.com/pandas-futurewarning-passing-literal-html-to-read_html-is-deprecated-and-will-be-removed-in-a-future-version.md) - If you're seeing the warning: FutureWarning: Passing literal html to 'read_html' is deprecated and will be removed in a future version this means you're using pandas' read_html() function in a way that will soon be unsupported. Why the Change? The pandas development team is deprecating the direct p… - [How to Convert a MultiIndex to type String or List of Strings in Pandas](https://datascientyst.com/how-to-convert-a-multiindex-to-type-string-or-list-of-strings-in-pandas.md) - To convert Pandas MultiIndex to list of strins or Sting we have several options: (1) Lambda and custom format midx.to_series().apply(lambda x: '{0}-{1}-{1}'.format(*x)).values (2) List comprehension array(['11-21-21', '11-22-22', '12-21-21', '12-22-22'], dtype=object) Data Grade 11 21 31 A 22 32 B… - [Convert Pandas MultiIndex values to New Type -String, Int](https://datascientyst.com/convert-pandas-multiindex-values-to-new-type-string-int.md) - Here are a few common ways to convert Pandas MultiIndex values to string or other data type: (1) Custom conversion per level df.index.set_levels(midx.levels[0].astype(int), level=0) \ .set_levels(midx.levels[1].astype(str), level=1) \ .set_levels(midx.levels[2].astype(int), level=2) (2) Conversion… - [Pandas: Contains Using Case Insensitive Search](https://datascientyst.com/pandas-contains-using-case-insensitive-search.md) - To perform case-insensitive string matching in Pandas, you can use the .str accessor along with regular expressions and the case=False parameter (1) parameter case of str.contains df1['col'].str.contains("MaX", na=False, case=False) (2) Margin only on columns df.query("City.str.lower() == 'new york… - [How to Insert a Row at Top of Pandas DataFrame](https://datascientyst.com/how-to-insert-a-row-at-top-of-pandas-dataframe.md) - To insert a row at the top or a specific index on DataFrame you can achieve it bt using slicing or concat: (1) Using pd.concat() with a list vals = [1, 2] pd.concat([pd.DataFrame([vals], columns=df.columns), df], ignore_index=True) (2) Using df.loc with manual index shift and sort df.loc[-1] = [1,2… - [How to Read CSV Directly from a URL in Pandas and Requests](https://datascientyst.com/how-to-read-csv-directly-from-a-url-in-pandas-and-requests.md) - Pandas can read CSV files directly from a URL by passing the URL to the read_csv() method. This is useful when working with datasets hosted online and for ad hoc tests. We can use the following syntax to read CSV from URL in Pandas: (1) Margin only on rows import pandas as pd url = "https://raw.git… - [Pandas TypeError 'list' object is not callable - rename Pandas columns](https://datascientyst.com/pandas-typeerror-list-object-is-not-callable-rename-pandas-columns.md) - The Pandas error 'list' object is not callable is raised when we try to rename dataframe columns. Usually this means that we try to use list instead of a dict with method: .rename(). df.rename(columns=['A', 'B', 'C']) results into: TypeError: 'list' object is not callable while cols = {'A':'AA', 'B… - [How to Wrap/Break Long Column Names in Pandas Dataframe](https://datascientyst.com/how-to-wrap-break-long-column-names-in-pandas-dataframe.md) - To wrap or break long column names in Pandas we can use module textwrap and map the column names with new line symbols: (1) Wrap DataFrame column names import textwrap cols_wrap = [textwrap.wrap(x, width=20) for x in df.columns] cols_wrap = {' '.join(words) : '
'.join(words) for words in cols_wr… - [How to Create a Pivot Table and Get Percentages in Pandas](https://datascientyst.com/how-to-create-a-pivot-table-and-get-percentages-in-pandas.md) - Pivoting a table and calculating row-wise or column-wise percentages is a common task in data analysis — often used to understand how values in a row contribute to the row total. Here's how to make a pivot table with it with percentage in Pandas: (1) Calculate row-wise percentage pivot_pct = pivot.… - [How to Format Numbers with Commas for Thousands in Pandas](https://datascientyst.com/how-to-format-numbers-with-commas-for-thousands-in-pandas.md) - To display large numbers in a more readable format we can insert commas as thousands separators in Pandas. This is especially useful when preparing data for presentation or reports. Below is a quick solution to format numbers with commas using Pandas: (1) Display Only df.style.format('{:,}') or df.… - [Fixing "ValueError: Cannot mix tz-aware with tz-naive values" in Pandas](https://datascientyst.com/fixing-valueerror-cannot-mix-tz-aware-with-tz-naive-values-in-pandas.md) - When using pd.to_datetime() in Pandas, you might encounter the error: ValueError: Cannot mix tz-aware with tz-naive values This happens when: * timezone-aware (tz-aware) and * timezone-naive (tz-naive) datetime values exist in the same column. Pandas does not allow this combination for operations l… - [How to Split Strings and Extract the N-th Element in a Pandas DataFrame](https://datascientyst.com/how-to-split-strings-and-extract-the-n-th-element-in-a-pandas-dataframe.md) - Working with text data in Pandas, you may need to split strings based on a n-th occuramce of delimiter and extract specific parts. This is useful for parsing URLs, file paths, or structured data. Pandas provides efficient ways to handle such operations with str.split() and expand=True. (1) Split th… - [How to Estimate the Memory Usage of a Pandas DataFrame](https://datascientyst.com/how-to-estimate-the-memory-usage-of-a-pandas-dataframe.md) - When working with large datasets, it's important to estimate how much memory a Pandas DataFrame will consume. This helps optimize performance and prevent memory errors. (1) Calculate memory usage per column df.memory_usage() (2) Interrogating object dtypes for system-level memory consumption df.mem… - [How To Find the Closest Values in a Pandas Series to a Number](https://datascientyst.com/how-to-find-the-closest-values-in-a-pandas-series-to-a-number.md) - In this post we will see how to find the closest values in a pandas Series to a given number. Here you can find two short solutions: (1) Find the single closest value in a Pandas Series closest_value = df['column_name'].iloc[(df['column_name'] - input_value).abs().idxmin()] (2) Find the N closest v… - [How to Calculate Days Elapsed Since a Certain Date in Pandas](https://datascientyst.com/how-to-calculate-days-elapsed-since-a-certain-date-in-pandas.md) - In this guide, I'll show you how to calculate days elapsed since a certain date in Pandas. Calculating the number of days elapsed since a certain date is a common task in data analysis. (1) Calculate days elapsed since today df['days_elapsed'] = (pd.to_datetime('today') - pd.to_datetime(df['date_co… - [How to Count Special Characters in a Column in Pandas](https://datascientyst.com/how-to-count-special-characters-in-a-column-in-pandas.md) - In this guide, I'll show you how to count special characters in a column using Pandas. Whether you want to count special characters row-wise or in the entire column or a single column, these methods will help. (1) Count special characters in each row df['column_name'].str.count(r'[^a-zA-Z0-9\s]') (… - [How to Count the Occurrences of a Specific Value in Pandas DataFrame](https://datascientyst.com/how-to-count-the-occurrences-of-a-specific-value-in-pandas-dataframe.md) - In this short guide, I'll show you how to count occurrences of a specific value in Pandas. Whether you want to count values in a single column or across the whole DataFrame, this guide has you covered. (1) Count occurrences in a single column df['column_name'].value_counts().get('target_value', 0)… - [How To Split Column Data Based on Condition in Pandas](https://datascientyst.com/how-to-split-column-data-based-on-condition-in-pandas.md) - In this short guide, I'll show you how to split a column based on condition in Pandas DataFrame. The example will show how to split a column containing URLs and extract only the last two parts (domain and top-level domain - TLD) (1) Quick Solution Using . as a Separator df['domain'] = df['url'].app… - [Count Characters in a Column and Create a new Length Column in Pandas](https://datascientyst.com/count-characters-in-a-column-and-create-a-new-length-column-in-pandas.md) - In this short guide, I'll show you how to count the number of characters in a string column and store the result as a new column in a Pandas DataFrame. (1) Quick and Fast Solution Using .str.len() df['char_count'] = df['text'].str.len() (2) Using apply(len) as an Alternative df['char_count'] = df['… - [How to Combine Date and Time Columns with Pandas](https://datascientyst.com/how-to-combine-date-and-time-columns-with-pandas.md) - In this short guide, I'll show you how to combine separate Date and Time columns into a single DateTime column in Pandas. When working with datasets, dates and times are often stored separately. Merging them into a single column can help with time-series analysis, sorting, and filtering. (1) Quick… - [How to Compare Two Excel or CSV Files Using Pandas](https://datascientyst.com/compare-two-excel-or-csv-files-using-pandas.md) - When working with data, you may need to compare two Excel or CSV files to: * find differences * detect updates * history changes * mistakes * validate records etc In this guide, we'll explore how to compare two Excel or CSV files using Pandas, with practical examples. Why Compare Files with Pandas… - [How to Read a Compressed CSV or JSON File in Pandas](https://datascientyst.com/read-compressed-csv-json-file-pandas.md) - To read compressed CSV and JSON files directly without manually decompressing them in Pandas use: (1) Read a Compressed CSV pd.read_csv('data.csv.gz', compression='gzip') (2) Read a Compressed JSON pd.read_json('data.json.gz', compression='gzip') This is useful for handling large datasets while sav… - [How to Save a Pandas DataFrame as a Compressed CSV/JSON File](https://datascientyst.com/save-pandas-dataframe-compressed-csv-json-file.md) - To save a DataFrame as a compressed CSV/JSON file using Pandas we can parameter compression='gzip as follows: CSV df.to_csv('data.csv.gz', index=False, compression='gzip') JSON df.to_JSON('data.json.gz', index=False, compression='gzip') Saving a DataFrame as a Compressed CSV You can save a Pandas D… - [How to Open and Convert an SQLite Database to a Pandas DataFrame](https://datascientyst.com/how-to-open-and-convert-an-sqlite-database-to-a-pandas-dataframe.md) - In this article, we’ll explore how to open an SQLite database and convert its tables into Pandas DataFrames with two practical examples. TL;DR To open and convert an SQLite database file like data.bg to a Pandas DataFrame we can use: import sqlite3 import pandas as pd # Connect to SQLite database (… - [How to Use the First Row as the Header in Pandas](https://datascientyst.com/how-to-use-the-first-row-as-the-header-in-pandas.md) - To use the first row as a header in Pandas we can: (1) Convert first row to header - reset index df.columns = df.iloc[0] df = df[1:].reset_index(drop=True) (2) Convert first row to header - keep index headers = df.iloc[0].values df.columns = headers df.drop(index=0, axis=0, inplace=True) (3) Read C… - [Round Pandas date to nearest year, month or week](https://datascientyst.com/round-pandas-date-to-nearest-year-month-or-week.md) - In this post you can find how to: * solve error: ValueError: is a non-fixed frequency * floor, ceil or round to year, month or week in Pandas (1) Get year, month or week in Pandas df['year'] = df['date'].dt.year df['month'] = df['date'].dt.month df['day'] = df['date'].dt.week (2) Round or floor yea… - [Rounding Pandas Timestamps to the Nearest Minute, 15M, 30M](https://datascientyst.com/rounding-pandas-timestamps-to-the-nearest-minute-15m-30m.md) - Below you can find multiple ways to round to the nearest minute or seconds in Pandas: (1) rounding pandas timestamp pd.to_datetime(df['date']).dt.round(freq='30min').dt.time result: 0 11:00:00 1 12:30:00 2 12:00:00 3 15:30:00 Name: date, dtype: object (2) round date to 30 minutes python df['date'].… - [Guide: How To Move a Column to the Front in Pandas DataFrame?](https://datascientyst.com/guide-move-column-front-pandas-dataframe.md) - In this guide, you can learn how to move columns by name to the front in Pandas DataFrame. There are situations where you might need to move a specific column to the front of the DataFrame: * for better visibility * to facilitate further analysis * to fulfill business requirements * export purposes… - [Guide: How to Exclude Rows while Sorting a DataFrame in Pandas](https://datascientyst.com/exclude-rows-while-sorting-dataframe-pandas.md) - In this short how to guide you can learn how to exclude rows from sorting in Pandas. To keep the original order of specific rows is useful when you deal with: * multi-dimensional data * filtering out outliers * excluding specific observations from the sorting process. Let's explore how to sort a Da… - [How to Merge with Missing Values in Pandas](https://datascientyst.com/how-to-merge-with-missing-values-in-pandas.md) - In this article, you can learn how to merge DataFrames in Pandas with handling of missing values. When working with data in Pandas, you often need to merge multiple DataFrames to consolidate information from different sources. However, not all data align perfectly, and missing values are common. Mi… - [How to Read CSV or JSON from URL With Authentication in Pandas](https://datascientyst.com/how-to-read-csv-from-url-with-authentication-in-pandas.md) - Reading a CSV file directly from a URL into Pandas is a common task, especially when dealing with web data. However, sometimes the data you need requires authentication to access. Fortunately, Python and Pandas provide straightforward methods to handle this scenario. Let's explore how to read a CSV… - [ImportError: matplotlib is required for plotting when the default backend "matplotlib" is selected](https://datascientyst.com/pandas-importerror-matplotlib-is-required-for-plotting.md) - Pandas, a powerful data manipulation library in Python, offers a convenient way to analyze and visualize data through its integration with the Matplotlib plotting library. However, users may encounter an ImportError when attempting to use Pandas for plotting, specifically indicating that Matplotlib… - [Importerror missing optional dependency html5lib pandas example](https://datascientyst.com/importerror-missing-optional-dependency-html5lib-pandas-example.md) - In this tutorial, we'll show how to solve a common Pandas error – "importerror missing optional dependency html5lib pandas example". We get this error from the Pandas when we try to use the method read_html() but the library html5lib is not installed. Fix Importerror missing optional dependency htm… - [AttributeError: 'DataFrame' object has no attribute 'append' - Pandas](https://datascientyst.com/fix-attributeerror-dataframe-object-has-no-attribute-append-pandas.md) - In this tutorial, we'll see how to solve a Pandas error: AttributeError: 'DataFrame' object has no attribute 'append'. We will also answer on the questions: * Why is append not working in pandas? * How do I fix pandas attribute error? * How do you append an object to a DataFrame in Python? * How to… - [How to Keep the First Value of Column After Explode in Pandas?](https://datascientyst.com/keep-first-value-column-after-explode-pandas.md) - In this quick tutorial, we're going to look at how to keep the first value of column after explode in Pandas? Suppose we have a DataFrame which has a column with nested data - list or JSON. Let's work with the following DataFrame: import pandas as pd data = {'ID': [1, 2, 3], 'Items': [['A', 'B'], [… - [Pandas vs R - cheat sheet](https://datascientyst.com/pandas-vs-r-cheat-sheet.md) - This is a Python/Pandas vs R cheatsheet for a quick reference for switching between both. The post contains equivalent operations between Pandas and R. The post includes the most used operations needed on a daily baisis for data analysis. Have in mind that some examples might differ due to differen… - [Trim Leading & Trailing White Space in Pandas DataFrame](https://datascientyst.com/trim-leading-trailing-white-space-pandas-dataframe.md) - To trim leading and trailing whitespaces from strings in Pandas DataFrame, you can use the str.strip() function to trim leading and trailing whitespaces from strings. Here's an example: import pandas as pd data = {'col_1': [' Apple ', ' Banana', 'Orange ', ' Grape '], 'col_2': [' Red ', 'Yellow ',… - [How To Map DataFrame Index to Dictionary in Pandas](https://datascientyst.com/map-dataframe-index-to-dictionary-pandas.md) - In this post, we'll explore how to map DataFrame Index values using a dictionary in Pandas. Setup Consider a DataFrame with following data: import pandas as pd data = {'Value': [10, 15, 20, 25]} df = pd.DataFrame(data, index=[1,2,3,4]) result: Value 1 10 2 15 3 20 4 25 This will create a DataFrame… - [Pandas read_csv: Automatic Date Reading from CSV Files](https://datascientyst.com/pandas-read_csv-automatic-date-reading-from-csv-files.md) - In this article, we will see how Pandas handles dates during the CSV reading process and automatic date recognition with method read_csv(). Automatic Date Reading in Pandas Pandas is designed to automatically recognize and parse dates while reading data from a CSV file, provided that dates are form… - [How to Convert Column to Categorical in Pandas DataFrame with Examples](https://datascientyst.com/convert-column-to-categorical-pandas-dataframe-examples.md) - In this article, we'll explore how to convert columns to categorical in a Pandas DataFrame with practical examples. In data analysis, efficient memory usage and improved performance are crucial considerations. Conversion column to categorical is simple as: df['col'].astype('category') Let's dive in… - [How to Deal With Whitespace and Irregular Separators in Pandas Read CSV?](https://datascientyst.com/make-separator-in-pandas-read_csv-more-flexible-wrt-whitespace-for-irregular-separators.md) - In this post, we will try to address how to deal with: * consecutive whitespaces as delimiters * irregular Separators while reading a CSV file with Pandas read_csv() method. Steps work with irregular separators * Inspect the CSV file * Select Pandas method: * read_csv * read_txt * read_fwf * Test r… - [How to split dataframe in Pandas](https://datascientyst.com/how-to-split-dataframe-in-pandas.md) - In this short guide, I'll show you how to split Pandas DataFrame. You can also find how to: * split a large Pandas DataFrame * pandas split dataframe into equal chunks * split DataFrame by percentage * split dataset into training and testing parts To start, here is the syntax to split Pandas Datafr… - [error: nothing to repeat at position 0 - Pandas](https://datascientyst.com/error-nothing-to-repeat-at-position-0-pandas.md) - Common Pandas error - error: nothing to repeat at position 0 can be result of several operations: * bad regular expression * reading CSV file with incorrect separator Fix error: nothing to repeat at position 0 To fix error: error: nothing to repeat at position 0 First we need to identify what is th… - [ValueError: pattern contains no capture groups - pandas](https://datascientyst.com/valueerror-pattern-contains-no-capture-groups-pandas.md) - To solve Pandas Error: Valueerror: Pattern Contains No Capture Groups we need to specify a capture group. Steps to plot 2 variables * Import matplotlib library * Create DataFrame with correlated data * Create the figure and axes object - fig, ax = plt.subplots() * Plot the first variable on x and l… - [How to Extract Everything Before or After with Regex in Pandas](https://datascientyst.com/extract-everything-before-after-regex-pandas.md) - To plot two variables on two sides of Y-axes, we can plot in two steps: * '(.*?)\n' * '.+?(?=\n)' Steps to extract everything until/after Below are the steps which I usually follow for regex extraction in Pandas * analyse the data from which I will extract * clean the data * choose pandas method -… - [How To Split Column by Multiple Characters with Regex in Pandas](https://datascientyst.com/split-column-by-multiple-characters-regex-in-pandas.md) - To split Pandas column by multiple characters we can use complex regex pattern as: * df['address'].str.split('; |, |\n', expand=True) * df['address'].str.extract(r'(.*)\n(.*)') Steps to split column in Pandas * Import matplotlib library * Create DataFrame with correlated data * Create the figure an… - [How To Read Only Specific Columns in Pandas read CSV](https://datascientyst.com/how-to-read-only-specific-columns-in-pandas-read-csv.md) - To read only specific columns from CSV file using Pandas read_csv method we need to use parameter usecols=fields Steps to read specific columns from CSV file * Import pandas * Define columns to be read * usecols=fields - to list columns to be read * Subset of columns to select, denoted either by co… - [How to Round To Nearest Hour in Pandas](https://datascientyst.com/how-to-round-to-nearest-hour-in-pandas.md) - To round to closest hour in Pandas datetime column we can several options: (1) Round to nearest hour df['date'].dt.round('H').dt.hour (2) floor to closest hour df['date'].dt.floor('h') (3) ceil to closest hour df['date'].dt.ceil('h') The image below show the results of 3 options: Let's cover the ca… - [Pandas pivot_table Silently Drops Indices with NaNs](https://datascientyst.com/pandas-pivot_table-silently-drops-indices-with-nans.md) - In this post, we will discuss when pivot_table silently drops indices with NaN-s. We will give an example, expected behavior and many resources. Example Let's have a DataFrame like: import pandas as pd import numpy as np df = pd.DataFrame({'foo': ['one', 'one', 'one', 'two', 'two', 'two'], 'bar': [… - [How To Read Multiple CSV Files into Pandas DataFrame](https://datascientyst.com/how-to-read-multiple-csv-files-into-pandas-dataframe.md) - To read multiple CSV file into single Pandas DataFrame we can use the following syntax: (1) Pandas read multiple CSV files path = r'/home/user/Downloads' all_files = glob.glob(path + "/*.csv") lst = [] for filename in all_files: df = pd.read_csv(filename, index_col=None, header=0) lst.append(df) me… - [How To Margin Only on Single Axis - Column or Row in Pandas](https://datascientyst.com/how-to-margin-only-on-single-axis-column-or-row-in-pandas.md) - We can use the following syntax to margin on a single axis column or row in Pandas: (1) Margin only on rows df.pivot_table(index='foo', columns='bar', values='baz', margins=True).iloc[:, :-1] (2) Margin only on columns df.pivot_table(index='foo', columns='bar', values='baz', margins=True).iloc[:-1,… - [TypeError: DataFrame.pivot() takes 1 positional argument but 4 were given - Pandas](https://datascientyst.com/typeerror-dataframe-pivot-takes-1-positional-argument-but-4-were-given-pandas.md) - In this tutorial, we'll take a closer look at the Pandas error, TypeError: DataFrame.pivot() takes 1 positional argument but 4 were given - Pandas. First, we'll create an example of how to produce it. Next, we'll explain the leading cause of the exception. And finally, we'll see how to fix it. Exam… - [Pandas pivot - ValueError: Index contains duplicate entries, cannot reshape](https://datascientyst.com/pandas-pivot-warning-about-repeated-entries-on-index.md) - In this article we will see how to solve Pandas pivot error: "ValueError: Index contains duplicate entries, cannot reshape". Let's see how to solve this error in different ways depending on the case. Setup Suppose we have a DataFrame like: import pandas as pd df = pd.DataFrame({'foo': ['one', 'one'… - [Football Prediction in Python: Barcelona vs Real Madrid](https://datascientyst.com/football-prediction-in-python-barcelona-vs-real-madrid.md) - In this post, we will Pandas and Python to collect football data and analyse it. We will try to predict probability for the outcome and the result of the fooball game between: Barcelona vs Real Madrid. Today is a great day for football fans - Barcelona vs Real Madrid game will be held tomorrow. Fan… - [How to solve: HTTPError: HTTP Error 403: Forbidden in Pandas](https://datascientyst.com/how-to-solve-httperror-http-error-403-forbidden-in-pandas.md) - In this post you can find how to solve Pandas and Python error: HTTPError: HTTP Error 403: Forbidden HTTPError: HTTP Error 403: Forbidden This error happens when we try to scrape tables with Pandas by using read_html method. For example: import pandas as pd url_cur = 'https://tradingeconomics.com/c… - [How to Round Time to the Nearest Quarter or Hour in Pandas?](https://datascientyst.com/how-to-round-time-to-the-nearest-quarter-or-hour-in-pandas.md) - To round a datetime column to the nearest quarter, minute or hour in Pandas, we can use the method: dt.round(). round datetime column to nearest hour Below you can find an example of rounding to the closest hour in Pandas and Python. We use method dt.round() with parameter H: import pandas as pd da… - [How to Sort by Multiple Columns Ascending and Descending in Pandas?](https://datascientyst.com/how-to-sort-by-multiple-columns-ascending-and-descending-in-pandas.md) - To sort by multiple columns ascending and descending in Pandas we can use syntax like: df.sort_values(by=['name', 'salary'], ascending=[True, False]) Let's cover two examples to explain sorting on multiple columns in more detail. Sort a DataFrame by two or more columns To sort Pandas DataFrame by t… - [Convert Pivot Table to Regular Data Frame in Pandas](https://datascientyst.com/convert-pivot-table-to-regular-data-frame-in-pandas.md) - In this post, we will see how to convert a Pandas pivot table to a regular DataFrame. To convert pivot table to DataFrame we can use: (1) the reset_index() method df_p.set_axis(df_p.columns.tolist(), axis=1).reset_index() (2) to_records() + pd.DataFrame() pd.DataFrame(df_p.to_records()) Let's cover… - [Error: need to escape, but no escapechar set - Pandas](https://datascientyst.com/error-need-to-escape-but-no-escapechar-set-pandas.md) - In this tutorial, we'll see how to solve a common Pandas and Python error – "Error: need to escape, but no escapechar set". We get this error from Pandas when we try to save DataFrame as a CSV file. Let's see several examples of how to reproduce and solve this error. Pandas - Error: need to escape,… - [How to Read Data from Text File Into Pandas?](https://datascientyst.com/load-data-from-text-file-into-pandas.md) - The following step-by-step example shows how to load data from a text file into Pandas. We can use: * read_csv() function * it handles various delimiters, including commas, tabs, and spaces * pd.read_fwf() * read fixed-width formatted lines into DataFrame Let's cover both cases into examples: read_… - [How to Extract Dictionary Value from Column in Pandas](https://datascientyst.com/extract-dictionary-value-from-column-in-pandas.md) - In this short guide, I'll show you how to extract or explode a dictionary value from a column in a Pandas DataFrame. You can use: * list or dict comprehension to extract dictionary values * the apply() function along with a lambda function to extract the value from each dictionary Setup For example… - [How to Convert DataFrame to JSON without Backslash in Pandas](https://datascientyst.com/convert-dataframe-to-json-without-backslash-in-pandas.md) - In this short tutorial, you'll see the steps to convert DataFrame to JSON without backslash escape in Pandas and Python. Note: Read also: How to Export DataFrame to JSON with Pandas Suppose we have the following DataFrame: Name Age site 0 Alice 25 http://example.com/ 1 Bob 30 http://example.com/ 2… - [How to Convert Pandas Column or Row to List](https://datascientyst.com/how-to-convert-pandas-column-or-row-to-list.md) - To convert a DataFrame column or row to a list in Pandas, we can use the Series method tolist(). Here's how to do it: df['A'].tolist() df.B.tolist() Image below shows some of the solutions described in this article: Setup We will use the following DataFrame to convert rows and columns to list: ```p… - [Pandas vs Julia - cheat sheet and comparison](https://datascientyst.com/pandas-vs-julia-comparison-cheat-sheet.md) - This is a Python/Pandas vs Julia cheatsheet and comparison. You can find what is the equivalent of Pandas in Julia or vice versa. You can find links to the documentation and other useful Pandas/Julia resources. The table below show the useful links for both: Pandas Julia data analysis tool high per… - [Pandas random sampling: stratified and weighted](https://datascientyst.com/pandas-random-sampling-stratified-and-weighted.md) - In this quick tutorial, we're going to discuss stratified sampling in Pandas and Python. The following syntax can be used to sample stratified in Pandas: (1) stratified sampling - disproportionated (df .groupby('continent', group_keys=False) .apply(lambda x: x.sample(2)) ) (2) stratified sampling -… - [Random Sample per group in pandas](https://datascientyst.com/random-sample-per-group-in-pandas.md) - Here are several ways to sample random rows per group in Pandas: (1) random selection per group df.groupby('continent').apply(lambda x: x.sample(n=3)) (2) random selection per group - different size (df .groupby('continent') .apply(lambda x: x.sample(n=3, replace=True)) .drop_duplicates() ) (3) sam… - [How to Convert List of Objects to Pandas DataFrame?](https://datascientyst.com/convert-list-of-objects-to-pandas-dataframe.md) - To convert a list of objects to a Pandas DataFrame, we can use the: * pd.DataFrame constructor * method from_records() and list comprehension: (1) Define custom class method pd.DataFrame([p.to_dict() for p in persons]) (2) Use vars() function pd.DataFrame([vars(p) for p in persons]) (3) Use attribu… - [How to apply Formatting and Borders to Pivot Table in Pandas](https://datascientyst.com/how-to-apply-formatting-and-borders-to-pivot-table-in-pandas.md) - To apply formatting and add borders to pivot tables in Pandas we can use style and set_table_styles. You can find basic example on adding borders and formatting: * adding borders * coloring numbers based on values * format NaN and float precision df_pivot.style. \ background_gradient(cmap='Reds', a… - [How To Create a Pivot Table in Pandas?](https://datascientyst.com/how-to-create-a-pivot-table-in-pandas.md) - We can use the following syntax to create a pivot table in Python using Pandas: df_pivot = df.pivot_table(values='D', index=['A', 'B'], columns='C') Next, we'll see the full steps to create pivot tables in Pandas using a simple example. Steps to create pivot table: Step 1: Get data for pivot Suppos… - [334-pivot](https://datascientyst.com/334-pivot.md) - Pivot - [How to Fix: ValueError: Trailing Data - Pandas and JSON](https://datascientyst.com/fix-valueerror-trailing-data-pandas-and-json.md) - In this tutorial, we'll see how to solve a common Pandas error – ValueError: Trailing data. We get this error from the Pandas read_json() method when we try to load a JSON or JSON lines file. To fix ValueError: Trailing data we can try: (1) Add parameter - lines=True pd.read_json('data.json', lines… - [Create Count Column by value_counts in Pandas DataFrame](https://datascientyst.com/create-count-column-value_counts-in-pandas-dataframe.md) - In this short guide, I'll show you how to create a new count column based on value_counts from another column in Pandas DataFrame. There are multiple ways to count values and add them as new column: (1) value_counts and map counts = df['col1'].value_counts() df['col_count'] = df['col1'].map(counts)… - [How to Group by multiple columns, count and map in Pandas](https://datascientyst.com/group-by-multiple-columns-count-and-map-in-pandas.md) - To group by two or multiple columns, count unique combinations and map the result we can chain two Pandas methods: * groupby() * size() df.groupby(['col1', 'col2']).size() The picture below shows all the steps and the final result: Let's create a sample DataFrame and explain all the steps in detail… - [Convert API Response to Pandas Dataframe - Python](https://datascientyst.com/convert-api-response-to-pandas-dataframe-python.md) - In this post, we will learn how to convert an API response to a Pandas DataFrame using the Python requests module. First we will read the API response to a data structure as: * CSV * JSON * XML * list of dictionaries and then we use the: * pd.DataFrame constructor * pd.DataFrame.from_dict(data) etc… - [Pandas cannot merge a series without a name](https://datascientyst.com/pandas-cannot-merge-a-series-without-a-name.md) - In this short guide, I'll show you how to solve Pandas error: ValueError: Cannot merge a Series without a name The error appears when we try to merge two Pandas series without a name. To solve the error we can set name to the series either by: (1) Rename series for the merge pd.merge(s1.rename('old… - [How to Count Na(NaN) and non Na Values in Pandas?](https://datascientyst.com/count-na-nan-and-non-na-values-in-pandas.md) - In this article, we will cover how to count NaN and non-NaN values in Pandas DataFrame or column. Missing values in Pandas are represented by NaN - not a number but sometimes are referred as: * NA * None * null We will see how to count all of them. Here is how to count NaN and non NAN values in Pan… - [How to validate IP address in Pandas](https://datascientyst.com/how-to-validate-ip-address-in-pandas.md) - To validate IP addresses in a Pandas DataFrame, we can use * the `pd.Series.apply() method * custom function or regex Here are the 2 ways to validate IP addresses in Pandas: (1) validate with regex df['ip'].str.contains(r"^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$") (2) custom validation function def val… - [How to Handle Exceptions With ast.literal_eval in a Pandas](https://datascientyst.com/how-to-handle-exceptions-with-ast-literal_eval-in-a-pandas.md) - To handle exceptions and use ast.literal_eval in Pandas we can define new function: import ast import pandas as pd def parse_eval(value): try: return ast.literal_eval(value) except (ValueError, SyntaxError): return value df = pd.DataFrame({'col': ['1', '2', '{"a": 3}', '[[']}) df['col'].apply(parse… - [Pandas Datetime Cheat Sheet](https://datascientyst.com/pandas-datetime-cheat-sheet.md) - Cheat sheet for working with datetime, dates and time in Pandas and Python. The cheat sheet try to show most popular operations in a short form. There is also a visual representation of the cheat sheet. Pandas is a powerful library for working with datetime data in Python. Pandas offer variaty of a… - [How to Create a Bag of Words in Pandas Python](https://datascientyst.com/create-a-bag-of-words-pandas-python.md) - In this short guide, I'll show you how to create a bag of words with Pandas and Python. You can find a example of bag of words using the sklearn library: from sklearn.feature_extraction.text import CountVectorizer import pandas as pd text = ['The fox jumps over the lazy dog.', 'Dog and fox are lazy… - [Instantly Turn Web Pages into Beautiful Dashboards with Python](https://datascientyst.com/instantly-turn-web-pages-into-beautiful-dashboards-python.md) - Intro I recently had the need to monitor multiple web pages and filter information from them. This is a rather simple task but it's time consuming and error prone. Every time I do it, it takes time to find the right data, analyze it and save it. In addition, often I would like to monitor multiple w… - [👋 Getting Started with Data Science Project](https://datascientyst.com/getting-started-with-data-science-project.md) - Are you one of those people who think that learning is difficult? One of the hardest parts about learning a new skill is getting started. Data science can be intimidating and scary at first. Yes, it's true: there are many things to learn - statistics, mathematics, programming - this can be overwhel… - [434-newsletter](https://datascientyst.com/newsletter.md) - Newsletter - [Pandas Visualization Cheat Sheet](https://datascientyst.com/pandas-visualization-cheat-sheet.md) - This visualization cheat sheet is a great resource to explore data visualizations with Python, Pandas and Matplotlib. The Python ecosystem provides many packages for producing high-quality plots, graphs and visualizations. In this guide, we will discuss the basics and a few popular visualization ch… - [FIFA World Cup 2022: Data-Driven Analyze (Twitter)](https://datascientyst.com/data-driven-approach-to-analyze-fifa-world-cup-2022-twitter.md) - Today is the first day of the World Cup 2022 which takes place in Qatar from 20-th of November to 18-th of December. 32 teams will compete in eight groups for the prize. In this post we will use data-driven approach to analyze the teams and what people twit for the new football event. What we can l… - [Style Pandas DataFrame Like a Pro (Examples)](https://datascientyst.com/style-pandas-dataframe-like-pro-examples.md) - In this tutorial, we'll discuss the basics of Pandas Styling and DataFrame formatting. We will also check frequently asked questions for DataFrame styles and formats. We'll start with basic usage, methods, parameters and then see a few Pandas styling examples. Next, we'll learn how to beautify Data… - [Data Science Project for beginners in 15 minutes](https://datascientyst.com/dataisbeautiful-the-absolute-quality-of-breaking-bad.md) - This article shows how to scrape, analyze and visualize movie data from IMDb. We will learn how to use Python and Pandas in order to collect, transform and present data in a beautiful way. Objective The second goal is to follow all steps in order to create popular DataIsBeautiful visualization: * [… - [424-dataisbeautiful](https://datascientyst.com/424-dataisbeautiful.md) - DataIsBeautiful - [How to Extract Domain from URL in Pandas](https://datascientyst.com/extract-domain-from-url-in-pandas.md) - In this short guide, I'll show you how to extract domain from a URL column in Pandas DataFrame. You can also find how to extract netloc, schema, path, params. So at the end you will get: ['https://www.datascientyst.com/cheatsheet','https://www.softhints.com/python'] to: 0 (https, www.datascientyst.… - [Best Data Analysis Libraries for Data Science - Python](https://datascientyst.com/best-python-libraries-for-data-analysis-python.md) - In this tutorial, we'll discuss the best libraries for Exploratory Data Analysis in Python. We will cover these EDA libraries: Library GitHub Stars Contributors Used by pandas-profiling 9700 79 9200 D-Tale 3700 22 501 Sweetviz 2200 5 n/a DataPrep 1400 33 n/a AutoViz 968 13 265 dabl 684 23 n/a klib… - [How to Convert Pandas DataFrame to Dictionary](https://datascientyst.com/convert-a-pandas-dataframe-to-a-dictionary.md) - In this short guide, I'll show you how to convert Pandas DataFrame to dictionary. You can also find how to use Pandas method - to_dict(). So at the end you will get from DataFrame to Python dict: day numeric 0 1 1 1 2 2 2 3 3 3 4 4 4 5 5 to: {'day': {0: 1, 1: 2, 2: 3, 3: 4, 4: 5, 5: 6}, 'numeric':… - [ValueError: DataFrame constructor not properly called! - Pandas](https://datascientyst.com/valueerror-dataframe-constructor-not-properly-called-pandas.md) - In this tutorial, we'll take a look at the Pandas error: ValueError: DataFrame constructor not properly called! First, we'll create examples of how to produce it. Next, we'll explain the reason and finally, we'll see how to fix it. ValueError: DataFrame constructor not properly called Let's try to… - [Free Public Datasets for Data Science Projects](https://datascientyst.com/datasets.md) - In this post we can find free public datasets for Data Science projects. There is a big number of datasets which cover different areas - machine learning, presentation, data analysis and visualization. You can find information for: * Data sources - big datasets collections which has curated data an… - [How to Convert String, DateTime Or TimeStamp to Time in Pandas](https://datascientyst.com/convert-string-datetime-timestamp-time-pandas.md) - In this article we will see how to extract time only from string or datetime in Pandas. First, we'll create an example DataFrame to test it. Next, we'll explain several examples in more detail. (1) extract time with .dt.time - datetime.time df['date'].dt.time (2) get time by .dt.strftime('%H:%M') a… - [KeyError:0 - Create DataFrame in Pandas](https://datascientyst.com/keyerror-0-create-dataframe-pandas.md) - In this tutorial, we'll take a look at the Pandas error: KeyError:0 First, we'll create an example of how to reproduce it. Next, we'll explain the reason and finally, we'll see how to fix it. Example Let's work with the following DataFrame: import pandas as pd data={'day': [1, 2, 3, 4, 5], 'numeric… - [Pandas Cheat Sheet: Data Cleaning](https://datascientyst.com/pandas-cheat-sheet-data-cleaning.md) - A practical Pandas Cheat Sheet: Data Cleaning useful for everyday working with data. This Pandas cheat sheet contains ready-to-use codes and steps for data cleaning. The cheat sheet aggregate the most common operations used in Pandas for: analyzing, fixing, removing - incorrect, duplicate or wrong… - [ValueError: All arrays must be of the same length - Pandas](https://datascientyst.com/valueerror-all-arrays-must-be-of-the-same-length-pandas.md) - In this tutorial, we'll see how to solve Pandas error: ValueError: All arrays must be of the same length First, we'll create an example of how to produce it. Next, we'll explain the reason and finally, we'll see how to fix it. Example Let's try to create the following DataFrame: import pandas as pd… - [416-pandas-error](https://datascientyst.com/416-pandas-error.md) - Pandas Error - [ValueError: If using all scalar values, you must pass an index - Pandas](https://datascientyst.com/valueerror-if-using-all-scalar-values-you-must-pass-an-index-pandas.md) - In this tutorial, we'll take a closer look at the Pandas error: "ValueError: If using all scalar values, you must pass an index" You can find explanation and solution on the image below: Quick fixes: (1) add index pd.DataFrame(dct, index=[0]) (2) use vector values dct = {k:[v] for k,v in dct.items(… - [Extract Day, Night, Morning, Afternoon, Evening from Pandas /Python Datetime](https://datascientyst.com/extract-day-night-morning-afternoon-evening-from-pandas-python-datetime.md) - In this guide, we will see how to extract day, night, morning, afternoon, evening from Pandas DataFrame. We would like to map and return information about part of the day from datetime or string column in DataFrame. Below you can find short answer: (1) Get Day or Night from datetime mask = (pd.to_t… ## Optional - [RSS Feed](https://datascientyst.com/rss/) - [Sitemap](https://datascientyst.com/sitemap.xml) - [Full content of pages and posts](https://datascientyst.com/llms-full.txt)