> ## Content Index
> Fetch the complete content index at: https://datascientyst.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# How to Compare Two Pandas DataFrames and Get Differences
- URL: https://datascientyst.com/compare-two-pandas-dataframes-get-differences/
- Published: 2022-02-18T21:53:15.000Z
- Updated: 2022-10-28T06:11:07.000Z
- Author: John D K
- Tags: DataFrame

## 1\. Overview

In this tutorial, we're going to **compare two Pandas DataFrames side by side and highlight the differences**.

We'll first look into Pandas method `compare()` to find differences between values of two DataFrames, then we will cover some advanced techniques to highlight the values and finally how to compare stats of the DataFrames.

The image below show the final result:

![compare-two-pandas-dataframes-get-differences](https://datascientyst.com/content/images/2022/10/compare-two-pandas-dataframes-get-differences.webp)

## 2\. Setup

Let's have the next DataFrame created by the code below:

```python
import pandas as pd
data = [('11','12','13'),
        ('21','22','23'),
        ('31','32','33')]
df = pd.DataFrame(data,
                 columns = ('col_1', 'col_2', 'col_3' ))

```

DataFrame looks like:

|   | col\_1 | col\_2 | col\_3 |
| - | ------ | ------ | ------ |
| 0 | 11     | 12     | 13     |
| 1 | 21     | 22     | 23     |
| 2 | 31     | 32     | 33     |

Let's create second DataFrame by copying the first one and changing some values:

```python
df2 = df.copy()
df2.iloc[1,1] = 32
df2.iloc[2,1] = 22
df2.iloc[2,2] = 44

```

The second DataFrame looks like:

|   | col\_1 | col\_2 | col\_3 |
| - | ------ | ------ | ------ |
| 0 | 11     | 12     | 13     |
| 1 | 21     | 32     | 23     |
| 2 | 31     | 22     | 44     |

Can you find what are the differences just by watching both DataFrames? In next steps we will **compare two DataFrames in Pandas**.

## 3\. Compare Two Pandas DataFrames to Get Differences

Pandas offers method: [pandas.DataFrame.compare](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.compare.html?ref=datascientyst.com) since version 1.1.0.

It gives the **difference between two DataFrames** \- the method is executed on DataFrame and take another one as a parameter:

```python
df.compare(df2)

```

The default result is new DataFrame which has differences between both DataFrames. The new DataFrame has multi-index - first level is the column name, the second one are the values from the both DataFrames which are compared:

|   | col\_2 | col\_3 |      |       |
| - | ------ | ------ | ---- | ----- |
|   | self   | other  | self | other |
| 1 | 22     | 32     | NaN  | NaN   |
| 2 | 32     | 22     | 33   | 44    |

The method has several parameters which we will cover in next sections:

- `align_axis`
- `keep_shape`
- `keep_equal`

## 4\. Compare Two Pandas DataFrames Side by Side - keeping all values

If you like to keep the original shape and all values (even if they are equal) then you can use - `keep_shape` and `keep_equal`.

Keep the original shape - all equal values are replaced by NaN values:

```python
df.compare(df2, keep_shape=True)

```

|   | col\_1 | col\_2 | col\_3 |       |      |       |
| - | ------ | ------ | ------ | ----- | ---- | ----- |
|   | self   | other  | self   | other | self | other |
| 0 | NaN    | NaN    | NaN    | NaN   | NaN  | NaN   |
| 1 | NaN    | NaN    | 22     | 32    | NaN  | NaN   |
| 2 | NaN    | NaN    | 32     | 22    | 33   | 44    |

Preserving the original values from both DataFrames can be done by both parameters:

```python
df.compare(df2, keep_shape=True, keep_equal=True)

```

The result is:

|   | col\_1 | col\_2 | col\_3 |       |      |       |
| - | ------ | ------ | ------ | ----- | ---- | ----- |
|   | self   | other  | self   | other | self | other |
| 0 | 11     | 11     | 12     | 12    | 13   | 13    |
| 1 | 21     | 21     | 22     | 32    | 23   | 23    |
| 2 | 31     | 31     | 32     | 22    | 33   | 44    |

## 5\. Compare Two Pandas DataFrames Highlighting Differences

Finally let's cover how to highlight the difference between both DataFrames. Again we are going to use the method `compare()` with a combination of styling.

### 5.1\. Compare and Highlight Difference between Two DataFrames

To compare two DataFrames get the difference and highlight them use the code below:

```python
df_mask = df.compare(df2, keep_shape=True).notnull().astype('int')
df_compare = df.compare(df2, keep_shape=True, keep_equal=True)

def apply_color(x):
    colors = {1: 'lightblue', 0: 'white'}
    return df_mask.applymap(lambda val: 'background-color: {}'.format(colors.get(val,'')))

df_compare.style.apply(apply_color, axis=None)

```

### 5.2\. Explanation of the steps

First we are going to build a mask for our styling:

```python
df_diff = df.compare(df2, keep_shape=True)

df_mask = df_diff.notnull().astype('int')

```

The mask will look like:

|   | col\_1 | col\_2 | col\_3 |       |      |       |
| - | ------ | ------ | ------ | ----- | ---- | ----- |
|   | self   | other  | self   | other | self | other |
| 0 | 0      | 0      | 0      | 0     | 0    | 0     |
| 1 | 0      | 0      | 1      | 1     | 0    | 0     |
| 2 | 0      | 0      | 1      | 1     | 1    | 1     |

Next is to build comparison DataFrame:

```python
df_compare = df.compare(df2, keep_shape=True, keep_equal=True)
df_compare

```

the same as the previous step. The last thing to do is apply styling based on the mask to the comparison DataFrame:

```python
def apply_color(x):
    colors = {1: 'lightblue', 0: 'white'}
    return df_mask.applymap(lambda val: 'background-color: {}'.format(colors.get(val,'')))

df_compare.style.apply(apply_color, axis=None)

```

the final result is:

|   | col\_1 | col\_2 | col\_3 |       |      |       |
| - | ------ | ------ | ------ | ----- | ---- | ----- |
|   | self   | other  | self   | other | self | other |
| 0 | 11     | 11     | 12     | 12    | 13   | 13    |
| 1 | 21     | 21     | 22     | 32    | 23   | 23    |
| 2 | 31     | 31     | 32     | 22    | 33   | 44    |

## 6\. Conclusion

To sum up, we looked at how to **compare two DataFrames in Pandas side by side.** We focused on finding differences, keeping the same shape and the equal values.

Finally we saw how to **highlight the differences while showing DataFrames side by side**.