> ## Content Index
> Fetch the complete content index at: https://datascientyst.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# How to Drop Bad Lines with read_csv in Pandas
- URL: https://datascientyst.com/drop-bad-lines-with-read_csv-pandas/
- Published: 2021-11-03T12:37:23.000Z
- Updated: 2026-07-22T22:19:53.000Z
- Author: John D K
- Tags: read_csv()

Here are two approaches to **drop bad lines with `read_csv` in Pandas**, when `on_bad_lines` is the only option, since:

- `error_bad_lines`
- `warn_bad_lines`

were removed in pandas 2.0.

**(1) Parameter `on_bad_lines='skip'` — Pandas >= 1.3 (current, recommended)**

```python
df = pd.read_csv(csv_file, delimiter=';', on_bad_lines='skip')

```

**(2) `error_bad_lines=False` — Pandas < 1.3 (removed, do not use)**

```python
# ❌ No longer works — raises TypeError on any pandas 2.x install
df = pd.read_csv(csv_file, delimiter=';', error_bad_lines=False)

```

`error_bad_lines` and `warn_bad_lines` were deprecated back in pandas 1.3 and fully **removed in pandas 2.0.0**. If you still have this in an older script, calling it on a modern pandas version will raise `TypeError: read_csv() got an unexpected keyword argument 'error_bad_lines'`. There's no fallback or compatibility shim — you need to switch to `on_bad_lines`.

Suppose we have two files:

- Single separator `;`  
```  
Date;Company A;Company A;Company B;Company B  
2021-09-06;1;7.9;2;6  
2021-09-07;1;8.5;2;7  
2021-09-08;2;8;1;8.1  
2021-09-09;2;8;1;"8.3;5.5"  
```
- Double separator `;;`  
```  
Date;;Company A;;Company A;;Company B;;Company B  
2021-09-06;;1;;7.9;;2;;6  
2021-09-07;;1;;8.5;;2;;7  
2021-09-08;;2;;8;;1;;8.1  
2021-09-09;;2;;8;;1;;"8.3;;5.5"  
```

## The `on_bad_lines` parameter

```
on_bad_lines{'error', 'warn', 'skip'} or callable, default 'error'

```

Specifies what to do upon encountering a bad line (a line with too many fields). Allowed values are:

- `error` — raise an exception when a bad line is encountered (this is still the default).
- `warn` — raise a warning when a bad line is encountered and skip that line.
- `skip` — skip bad lines without raising or warning when they are encountered.
- **callable** — since pandas 1.4, you can also pass a function. It receives the bad line as a list of strings and can return a corrected list of fields (to fix the row instead of dropping it), or `None` to drop it. This only works with the `engine='python'` parser.

Example of the callable form, useful when you want to repair rather than discard a malformed row:

```python
def fix_row(bad_line):
    # keep only the first 5 fields, drop the rest
    return bad_line[:5]

df = pd.read_csv(csv_file, delimiter=';', engine='python', on_bad_lines=fix_row)

```

Note that depending on the separator:

- single
- multiple
- regex

the `read_csv` behavior can be different. You can check this article for more information: [How to Use Multiple Char Separator in read\_csv in Pandas](https://datascientyst.com/use-multiple-char-separator-read%5Fcsv-pandas/)

The reason for this is described in the pandas documentation:

> Note that regex delimiters are prone to ignoring quoted data. Regex example: `'\r\t'`.

Any separator longer than one character (like our `;;` example) is treated as a regex by the parser, which is why quoted fields containing the delimiter get mis-split — this behavior is unchanged in current pandas.

For example, for a single separator `";"` this code will work fine:

| Date       | Company A | Company A.1 | Company B | Company B.1 |
| ---------- | --------- | ----------- | --------- | ----------- |
| 2021-09-06 | 1         | 7.9         | 2         | 6           |
| 2021-09-07 | 1         | 8.5         | 2         | 7           |
| 2021-09-08 | 2         | 8.0         | 1         | 8.1         |
| 2021-09-09 | 2         | 8.0         | 1         | 8.3;5.5     |

While if you have a file with two separators you will get an error or warning (depending on `on_bad_lines`) for the quoted line:

```python
csv_file = '../data/csv/multine_bad_line_multi_sep.csv'
df = pd.read_csv(csv_file, delimiter=';;', engine='python', on_bad_lines='warn')

```

> Skipping line 5: Expected 5 fields in line 5, saw 6\. Error could possibly be due to quotes being ignored when a multi-char delimiter is used.

In this case only 3 rows will be read from the CSV file:

| Date       | Company A | Company A.1 | Company B | Company B.1 |
| ---------- | --------- | ----------- | --------- | ----------- |
| 2021-09-06 | 1         | 7.9         | 2         | 6.0         |
| 2021-09-07 | 1         | 8.5         | 2         | 7.0         |
| 2021-09-08 | 2         | 8.0         | 1         | 8.1         |

![](https://datascientyst.com/content/images/2021/11/drop-bad-lines-with-read_csv-pandas.png)

## A quick note on the C vs Python engine

With a single-character delimiter, pandas defaults to the fast C engine, which supports `on_bad_lines='error'|'warn'|'skip'` natively. With a multi-character or regex delimiter, pandas automatically falls back to `engine='python'`, which is slower but is also the only engine that supports a **callable** passed to `on_bad_lines`. If you need to fix (not just skip) bad lines and you're on a single-character delimiter, you must explicitly pass `engine='python'` yourself, as shown above.

## Resources

- [Notebook](https://github.com/softhints/Pandas-Tutorials/blob/master/read%5Fcsv/drop-bad-lines-with-read%5Fcsv-pandas.ipynb?ref=datascientyst.com)
- [pandas.read\_csv](https://pandas.pydata.org/docs/reference/api/pandas.read%5Fcsv.html?ref=datascientyst.com)