In this quick tutorial, we'll cover how to convert Pandas DataFrame to a list of dictionaries.

Below you can find the quick answer of DataFrame to list of dictionaries:

df.to_dict('records')

Let's explain the solution in a practical example.

Suppose we have DataFrame with data like:

import pandas as pd

df = pd.read_csv('https://raw.githubusercontent.com/softhints/Pandas-Tutorials/master/data/population/population.csv')
df
Region 1500 1600 1700 1750 2050 2150
World 585 660 710 791 9725 9746
Africa 86 114 106 106 2478 2308
Asia 282 350 411 502 5267 5561
Europe 168 170 178 190 734 517
Latin America [Note 1] ​ 40 20 10 16 784 912

Convert whole DataFrame to list of dictionaries

Which you would like to convert to list of dictionaries like:

{'Region': 'World',
  '1500': 585,
  '1600': 660,
  '1700': 710,
  '1750': 791,
  '1800': 978,
  '1850': 1262,
  '1900': 1650,
  '1950': 2521,
  '1999': 6008,
  '2008': 6707,
  '2010': 6896,
  '2012': 7052,
  '2050': 9725,
  '2150': 9746},
 {'Region': 'Africa',
  '1500': 86,
  '1600': 114,
  '1700': 106,

In order to achieve this behaviour you can use different approaches. First approach will use Pandas method to_dict('records'):

df.to_dict('records')

This method have several possible options:

  • 'dict’ (default) : dict like {column -> {index -> value}}
  • 'list’ : dict like {column -> [values]}
  • 'series’ : dict like {column -> Series(values)}
  • 'split’ : dict like {'index’ -> [index], 'columns’ -> [columns], 'data’ -> [values]}
  • 'tight’ : dict like {'index’ -> [index], 'columns’ -> [columns], 'data’ -> [values], 'index_names’ -> [index.names], 'column_names’ -> [column.names]}
  • 'records’ : list like [{column -> value}, … , {column -> value}]
  • 'index’ : dict like {index -> {column -> value}}

Convert columns to list of dictionaries

If you want to convert only some columns to a list of dictionaries you can use similar syntax. This can be achieved by selecting the columns and then applying method .to_dict('records'):

df[['Region', '1500', '1600', '1700']].to_dict('records')

Result is subset of the selected columns:

[{'Region': 'World', '1500': 585, '1600': 660, '1700': 710},
 {'Region': 'Africa', '1500': 86, '1600': 114, '1700': 106},
 {'Region': 'Asia', '1500': 282, '1600': 350, '1700': 411},
 {'Region': 'Europe', '1500': 168, '1600': 170, '1700': 178},
 {'Region': 'Latin America [Note 1] \u200b',
  '1500': 40,
  '1600': 20,
  '1700': 10},
 {'Region': 'Northern America [Note 1] \u200b',
  '1500': 6,
  '1600': 3,
  '1700': 2},
 {'Region': 'Oceania', '1500': 3, '1600': 3, '1700': 3}]

Transformation is visible from the image below:

Convert DataFrame to list of dictionaries - column wise

What if you like to get list of dictionaries column wise like:

{'Region': {0: 'World',
  1: 'Africa',
  2: 'Asia',
  3: 'Europe',
  4: 'Latin America [Note 1] \u200b',
  5: 'Northern America [Note 1] \u200b',
  6: 'Oceania'},
 '1500': {0: 585, 1: 86, 2: 282, 3: 168, 4: 40, 5: 6, 6: 3},
 '1600': {0: 660, 1: 114, 2: 350, 3: 170, 4: 20, 5: 3, 6: 3},
 '1700': {0: 710, 1: 106, 2: 411, 3: 178, 4: 10, 5: 2, 6: 3},

You can achieve this transformation by transposing the DataFrame with .T and using option index:

df.T.to_dict('index')

Custom Conversion of DataFrame to list of dictionaries

Finally lets cover the case when there is a custom logic for transformation of DataFrame to list of dictionaries in Pandas. This option is a bit slower than the rest and works fine for small and medium sized data.

We are going to iterate over all rows by:

data_dict = []
for index, row in df[['Region', '1500', '1600', '1700']].iterrows():
    data_dict.append({
            'Region': row['Region'],
            '1500': row['1500'],
            '1600': row['1600'],
            '1700': row['1700'],
            })

the result would be:

[{'Region': 'World', '1500': 585, '1600': 660, '1700': 710},
 {'Region': 'Africa', '1500': 86, '1600': 114, '1700': 106},
 {'Region': 'Asia', '1500': 282, '1600': 350, '1700': 411},
 {'Region': 'Europe', '1500': 168, '1600': 170, '1700': 178},
 {'Region': 'Latin America [Note 1] \u200b',
  '1500': 40,
  '1600': 20,
  '1700': 10},
 {'Region': 'Northern America [Note 1] \u200b',
  '1500': 6,
  '1600': 3,
  '1700': 2},
 {'Region': 'Oceania', '1500': 3, '1600': 3, '1700': 3}]

To convert all columns to list of dicts with custom logic you can use code like:

data_dict = []
for index, row in df[df.columns].iterrows():
    data_dict.append({
            row.to_list()[0] : row.to_list()[1:]
            })
data_dict

result:

[{'World': [585,
   660,
   710,
   791,
   978,
   1262,
   1650,
   2521,
   6008,
   6707,
   6896,
   7052,
   9725,
   9746]},
 {'Africa': [86,
   114,
   106,
   106,
   107,

Resources