Pandas - Slicing Large Dataframes into Chunks

When attempting to process oversized dataframes, a common obstacle is the dreaded Memory Error. One effective solution is to divide the dataframe into smaller, manageable chunks. This strategy not only reduces memory consumption but also facilitates efficient processing.

To achieve this, we can leverage either list comprehension or the NumPy array_split function.

List Comprehension

n = 200000  # Chunk row size
list_df = [df[i:i n] for i in range(0, df.shape[0], n)]

NumPy array_split

list_df = np.array_split(df, math.ceil(len(df) / n))

Individual chunks can then be retrieved using:

list_df[0]
list_df[1]
...

To reassemble the chunks into a single dataframe, employ pd.concat:

# Example: Concatenating by chunks
rejoined_df = pd.concat(list_df)

Slicing by AcctName

To split the dataframe by AcctName values, utilize the groupby method:

list_df = []

for n, g in df.groupby('AcctName'):
    list_df.append(g)