fingoldo opened a new issue, #38736:
URL: https://github.com/apache/arrow/issues/38736
### Describe the bug, including details regarding any error messages,
version, and platform.
I've been noticing a memory leak for several years now. When reading a big
parquet file, pyarrow lib or Pandas with pyarrow engine (default one) suffer
from extensive RAM waste, twice the size of data itself. RAM is not freed by
gc.collect(), even after the dataframe in question has been deleted from the
workspace (you can try uncommenting deletion lines in my file).
Finally found time to reproduce and report it.
Let's first create a big file with lots of columns and mixed dtypes:
test_parquet_make_file.py
```python
import pyarrow.parquet as pq
import pandas as pd, numpy as np
df = pd.DataFrame(np.random.random(size=(100_000,10000)))
for i in [10,100,1000,5000]:
df.iloc[:,i]=df.iloc[:,i].astype(np.int32)
df.iloc[:,i+1]=df.iloc[:,i+1].astype(np.float32)
df['cat']='cat'
df['cat']=df['cat'].astype('category')
print(df.info())
df.to_parquet("output.parquet")
```
It gets saved as an approx. 10Gb file.
Let's now open it with pyarrow, report its size, wait a bit, garbage
collect, and report process RAM usage:
test_parquet_leak.py
```python
import pyarrow.parquet as pq
from gc import collect
from time import sleep
import pandas as pd, numpy as np
import psutil, os
#df = pd.read_parquet("output.parquet",engine='fastparquet')
#df = pd.read_parquet("output.parquet",engine='pyarrow')
#print(df.info())
df=pq.read_table("output.parquet")
collect()
sleep(5)
collect()
print(f"Data size: {df.get_total_buffer_size()/1024**3:.2f} Gb")
#del df
#collect()
#sleep(5)
#collect()
process = psutil.Process(os.getpid())
try:
mem=process.memory_info()
except:
mem=process.full_memory_info()
print(f"process RAM used: {mem.rss / 1024**3:.2f} Gb")
```
Output on my Windows x64 station:
>
> Data size: 7.57 Gb
> process RAM used: 21.90 Gb
By uncommenting the lines, you can see that Pandas with Pyarrow engine also
suffers from this, unlike fastparquet engine.
Again, even if data object is deleted, memory stays taken in the process
forever.
For big files, this creates unnecesary OOM problems. I can understand
temporarily usage of more RAM for performance reasons, but IMHO there should be
no excuse for permanent RAM lock up like this in such significant amounts
(2-x/3x of original data).
Thanks!
### Component(s)
Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
