#####   ######  #####    ###    #   #  ###  #   #  ######
##  ##  ##      ##  ##  ## ##   #   #   #   #   #  ##
#####   ####    #####   #   #   #   #   #   #   #  ####
##  #   ##      ##      ## ##    # #    #    # #   ##
##   #  ######  ##       ###      #    ###    #    ######

$ curl repovive.com/roadmaps/data-engineer-interview-prep/python-for-data-engineering/pandas-at-scale

░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░█████████████████████████████████████████████████████████████████████████████████████████████████████

#####   ######  #####    ###    #   #  ###  #   #  ######
##  ##  ##      ##  ##  ## ##   #   #   #   #   #  ##
#####   ####    #####   #   #   #   #   #   #   #  ####
##  #   ##      ##      ## ##    # #    #    # #   ##
##   #  ######  ##       ###      #    ###    #    ######

$ curl repovive.com/roadmaps/data-engineer-interview-prep/python-for-data-engineering/pandas-at-scale

Repovive

Pandas at Scale - Python for Data Engineering | Data Engineer Interview Prep | Repovive

Repovive.

Data Engineer Interview Prep16 sections · 140 units8h total

Open in Course

Pandas at Scale

When pandas isn't enough

Pandas works great until it doesn't. It typically needs $2$ - $5$ x the file size in RAM. A $5$ GB CSV might require $10$ - $25$ GB of memory depending on data types.

Solutions:

Chunking: Process in pieces with pd.read_csv(chunksize=10000)
Dtypes: Use efficient types. category for strings, int32 instead of int64
Dask: Parallel pandas. Same API, distributed execution
Polars: Rust-based, faster than pandas for many operations

import dask.dataframe as dd
df = dd.read_parquet('large_file.parquet')
result = df.groupby('category').sum().compute()

For large data, move to Spark.