How to Drop Bad Lines with read_csv in Pandas   

Here are two approaches to drop bad lines with read_csv in Pandas, when on_bad_lines is the only option, since:

• error_bad_lines
• warn_bad_lines

were removed in pandas 2.0.

(1) Parameter on_bad_lines='skip' — Pandas >= 1.3 (current, recommended)

df = pd.read_csv(csv_file, delimiter=';', on_bad_lines='skip')

(2) error_bad_lines=False — Pandas < 1.3 (removed, do not use)

# ❌ No longer works — raises TypeError on any pandas 2.x install
df = pd.read_csv(csv_file, delimiter=';', error_bad_lines=False)

error_bad_lines and warn_bad_lines were deprecated back in pandas 1.3 and fully removed in pandas 2.0.0. If you still have this in an older script, calling it on a modern pandas version will raise TypeError: read_csv() got an unexpected keyword argument 'error_bad_lines'. There's no fallback or compatibility shim — you need to switch to on_bad_lines.

Suppose we have two files:

• Single separator ;

  Date;Company A;Company A;Company B;Company B
  2021-09-06;1;7.9;2;6
  2021-09-07;1;8.5;2;7
  2021-09-08;2;8;1;8.1
  2021-09-09;2;8;1;"8.3;5.5"
  

• Double separator ;;

  Date;;Company A;;Company A;;Company B;;Company B
  2021-09-06;;1;;7.9;;2;;6
  2021-09-07;;1;;8.5;;2;;7
  2021-09-08;;2;;8;;1;;8.1
  2021-09-09;;2;;8;;1;;"8.3;;5.5"
  

The on_bad_lines parameter

on_bad_lines{'error', 'warn', 'skip'} or callable, default 'error'

Specifies what to do upon encountering a bad line (a line with too many fields). Allowed values are:

• error — raise an exception when a bad line is encountered (this is still the default).
• warn — raise a warning when a bad line is encountered and skip that line.
• skip — skip bad lines without raising or warning when they are encountered.
• callable — since pandas 1.4, you can also pass a function. It receives the bad line as a list of strings and can return a corrected list of fields (to fix the row instead of dropping it), or None to drop it. This only works with the engine='python' parser.

Example of the callable form, useful when you want to repair rather than discard a malformed row:

def fix_row(bad_line):
    # keep only the first 5 fields, drop the rest
    return bad_line[:5]

df = pd.read_csv(csv_file, delimiter=';', engine='python', on_bad_lines=fix_row)

Note that depending on the separator:

• single
• multiple
• regex

the read_csv behavior can be different. You can check this article for more information: How to Use Multiple Char Separator in read_csv in Pandas

The reason for this is described in the pandas documentation:

Note that regex delimiters are prone to ignoring quoted data. Regex example: '\r\t'.

Any separator longer than one character (like our ;; example) is treated as a regex by the parser, which is why quoted fields containing the delimiter get mis-split — this behavior is unchanged in current pandas.

For example, for a single separator ";" this code will work fine:

While if you have a file with two separators you will get an error or warning (depending on on_bad_lines) for the quoted line:

csv_file = '../data/csv/multine_bad_line_multi_sep.csv'
df = pd.read_csv(csv_file, delimiter=';;', engine='python', on_bad_lines='warn')

Skipping line 5: Expected 5 fields in line 5, saw 6. Error could possibly be due to quotes being ignored when a multi-char delimiter is used.

In this case only 3 rows will be read from the CSV file:

A quick note on the C vs Python engine

With a single-character delimiter, pandas defaults to the fast C engine, which supports on_bad_lines='error'|'warn'|'skip' natively. With a multi-character or regex delimiter, pandas automatically falls back to engine='python', which is slower but is also the only engine that supports a callable passed to on_bad_lines. If you need to fix (not just skip) bad lines and you're on a single-character delimiter, you must explicitly pass engine='python' yourself, as shown above.

Resources

• Notebook
• pandas.read_csv