Random Sample per group in pandas

Here are several ways to sample random rows per group in Pandas:

(1) random selection per group

df.groupby('continent').apply(lambda x: x.sample(n=3))

(2) random selection per group - different size

(df
    .groupby('continent')
    .apply(lambda x: x.sample(n=3, replace=True))
    .drop_duplicates()
)

(3) sample based on column

col = 'continent'
categories = list(df[col].dropna().unique())
for cat in categories:
    if df[df[col] == cat].shape[0] > 3:
   	 print(cat, end=' - ')
   	 display(df[df[col] == cat].sample(3, replace=True))

The result is shown below - getting N random samples per each group on:

Setup

In the post, we'll use the following DataFrame, available from library plotly.

To install plotly use pip install plotly. We will create DataFrame getting info for 2007 year:

import plotly.express as px
df = px.data.gapminder().query("year == 2007")
cols = df.columns[:4]
df = df[cols]

DataFrame looks like:

For simplicity we will work only with the first 4 columns.

1: Random selection per group

To do random selection per group in Pandas we can:

• use groupby() on a column(s)
• and use apply and sample methods:

df.groupby('continent').apply(lambda x: x.sample(n=1))

We get random country per each continent:

If we try to use large sample number we will get error:

ValueError: Cannot take a larger sample than population when 'replace=False'

We will solve this error in next section

sample by group - different size

If the groups have different sizes or some groups are smaller than the selected sample number we can use repetitions. To sample with repetition we need to pass replace=True to sample() method:

df.groupby('continent').apply(lambda x: x.sample(n=3, replace=True))

This will solve the error above:

Get unique samples per group

Downside is that we have repetitions in the groups with smaller items. Remove duplicated rows can be done by adding .drop_duplicates():

(df
    .groupby('continent')
    .apply(lambda x: x.sample(n=3, replace=True))
    .drop_duplicates()
)

This code shows the usage of Pandas chaining. It's easier to read and maintain.

Group by multiple columns and sample

To group by multiple columns and sample random values per each group in Pandas we can use similar code:

(df
    .groupby(['continent', 'year'])
    .apply(lambda x: x.sample(n=2, replace=True))
    .drop_duplicates()
)

The result is multi-index with samples from each group:

Random sample per group - for loop

If we need to add filtering or parsing logic we may use a for loop. In this way we may get different DataFrames for each sample group:

col = 'continent'
categories = list(df[col].dropna().unique())

for cat in categories:
    group_size = df[df[col] == cat].shape[0]
    print(cat, '-', group_size)
    if group_size >= 3:
   	 display(df[df[col] == cat].sample(3))
    else:
   	 display(df[df[col] == cat].sample(group_size))

The result is visible on the image below:

Conclusion

In this article, we looked at different ways for sampling random items per group in Pandas and Python. We focused on equal sampling, but also covered sampling from groups with different sizes.

Finally we saw how to exclude, filter or parse some groups when doing random sampling. We briefly introduce Pandas chaining - a nice technique for writing better and more readable Pandas code.

Resources

• Pandas - Random Sample of a subset of a DataFrame
• How to Get Top 10 Highest or Lowest Values in Pandas
• Pandas random sampling: stratified and weighted