bveeramani opened a new issue, #45493:
URL: https://github.com/apache/arrow/issues/45493
### Describe the bug, including details regarding any error messages,
version, and platform.
Two issues:
1. `pa.infer_type` uses something like O(n^2) memory to infer the type
2. The memory isn't cleaned up
```python
import gc
import numpy as np
import psutil
import pyarrow as pa
array = np.zeros(8 * 1024 * 1024 // 8, dtype="datetime64[s]") # A 8 MiB
array
process = psutil.Process()
for _ in range(30):
before = process.memory_info().rss
pa.infer_type(array)
gc.collect()
pa.default_memory_pool().release_unused()
after = process.memory_info().rss
print(
f"{(after - before) / 1024 / 1024:.02f} MiB ({before / 1024 /
1024:.02f} MiB "
f"-> {after / 1024 / 1024:.02f} MiB)"
)
```
```
190.53 MiB (46.95 MiB -> 237.48 MiB)
146.28 MiB (237.50 MiB -> 383.78 MiB)
146.33 MiB (383.78 MiB -> 530.11 MiB)
146.23 MiB (530.11 MiB -> 676.34 MiB)
146.30 MiB (676.34 MiB -> 822.64 MiB)
146.30 MiB (822.64 MiB -> 968.94 MiB)
146.31 MiB (968.94 MiB -> 1115.25 MiB)
146.30 MiB (1115.25 MiB -> 1261.55 MiB)
146.28 MiB (1261.55 MiB -> 1407.83 MiB)
146.30 MiB (1407.83 MiB -> 1554.12 MiB)
146.69 MiB (1554.12 MiB -> 1700.81 MiB)
146.30 MiB (1700.81 MiB -> 1847.11 MiB)
146.25 MiB (1847.11 MiB -> 1993.36 MiB)
146.33 MiB (1993.36 MiB -> 2139.69 MiB)
146.30 MiB (2139.69 MiB -> 2285.98 MiB)
146.30 MiB (2285.98 MiB -> 2432.28 MiB)
146.30 MiB (2432.28 MiB -> 2578.58 MiB)
146.30 MiB (2578.58 MiB -> 2724.88 MiB)
146.25 MiB (2724.88 MiB -> 2871.12 MiB)
146.30 MiB (2871.12 MiB -> 3017.42 MiB)
146.70 MiB (3017.42 MiB -> 3164.12 MiB)
146.34 MiB (3164.12 MiB -> 3310.47 MiB)
146.25 MiB (3310.47 MiB -> 3456.72 MiB)
146.30 MiB (3456.72 MiB -> 3603.02 MiB)
146.30 MiB (3603.02 MiB -> 3749.31 MiB)
146.27 MiB (3749.31 MiB -> 3895.58 MiB)
146.30 MiB (3895.58 MiB -> 4041.88 MiB)
146.31 MiB (4041.88 MiB -> 4188.19 MiB)
146.31 MiB (4188.19 MiB -> 4334.50 MiB)
146.28 MiB (4334.50 MiB -> 4480.78 MiB)
```
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
