[issue37294] concurrent.futures.ProcessPoolExecutor state=finished raised error

2020-09-28 Thread DanilZ 


DanilZ  added the comment:

After executing a single task inside a process the result is returned with 
state=finished raised error.

Error happens when trying to load a big dataset (over 5 GB). Otherwise the same 
dataset reduced to a smaller nrows executes and returns from result() without 
errors.

with concurrent.futures.ProcessPoolExecutor(max_workers = 1) as executor:
results = executor.submit(pd.read_csv, path)

data = results.result()

--
components: +2to3 (2.x to 3.x conversion tool) -Library (Lib)
nosy: +DanilZ
title: concurrent.futures.ProcessPoolExecutor and multiprocessing.pool.Pool 
fail with super -> concurrent.futures.ProcessPoolExecutor state=finished raised 
error

___
Python tracker 
<https://bugs.python.org/issue37294>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37294] concurrent.futures.ProcessPoolExecutor state=finished raised error

2020-10-01 Thread DanilZ 


DanilZ  added the comment:

.

> On 1 Oct 2020, at 03:11, Kyle Stanley  wrote:
> 
> 
> Kyle Stanley  added the comment:
> 
> DanilZ, could you take a look at the superseding issue 
> (https://bugs.python.org/issue37297) and see if your exception raised within 
> the job is the same?  
> 
> If it's not, I would suggest opening a separate issue (and linking to it in a 
> comment here), as I don't think it's necessarily related to this one. 
> "state=finished raised error" doesn't indicate the specific exception that 
> occurred. A good format for the name would be something along the lines of:
> 
> "ProcessPoolExecutor.submit()  while reading 
> large object (4GB)"
> 
> It'd also be helpful in the separate issue to paste the full exception stack 
> trace, specify OS, and multiprocessing start method used (spawn, fork, or 
> forkserver). This is necessary to know for replicating the issue on our end.
> 
> In the meantime, I workaround I would suggest trying would be to use the  
> *chunksize* parameter (or *Iterator*) in pandas.read_csv(), and split it 
> across several jobs (at least 4+, more if you have additional cores) instead 
> of within a single one. It'd also be generally helpful to see if that 
> alleviates the problem, as it could possibly indicate an issue with running 
> out of memory when the dataframe is converted to pickle format (which often 
> increases the total size) within the process associated with the job.
> 
> --
> nosy: +aeros
> 
> ___
> Python tracker 
> <https://bugs.python.org/issue37294>
> ___

--

___
Python tracker 
<https://bugs.python.org/issue37294>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37294] concurrent.futures.ProcessPoolExecutor state=finished raised error

2020-10-01 Thread DanilZ 

DanilZ  added the comment:

I think you have correctly estimated the problem in the last part of your 
message: "as it could possibly indicate an issue with running out of memory 
when the dataframe is converted to pickle format (which often increases the 
total size) within the process associated with the job”

The function pd.read_csv performs without any problems inside a process, the 
error appears only when I try to extract it from the finished process via:
for f in concurrent.futures.as_completed(results):
data = f.result()

or

data = results.result()

It just does not pass a large file from the results object.

I am sure that inside of a multiprocess everything works correctly for 2 
reasons:
1. If I change in function inside a process to just save the file (that had 
been read in memory) to disk.
2. If I recuse the file size, then it gets extracted from results.result() 
without error.

So I guess then that my question narrows down to: 
1. Can I increase the memory allocated to a process? 
2. Or at least understand what would is the limit.

Regards,
Danil

> On 1 Oct 2020, at 03:11, Kyle Stanley  wrote:
> 
> 
> Kyle Stanley  added the comment:
> 
> DanilZ, could you take a look at the superseding issue 
> (https://bugs.python.org/issue37297) and see if your exception raised within 
> the job is the same?  
> 
> If it's not, I would suggest opening a separate issue (and linking to it in a 
> comment here), as I don't think it's necessarily related to this one. 
> "state=finished raised error" doesn't indicate the specific exception that 
> occurred. A good format for the name would be something along the lines of:
> 
> "ProcessPoolExecutor.submit()  while reading 
> large object (4GB)"
> 
> It'd also be helpful in the separate issue to paste the full exception stack 
> trace, specify OS, and multiprocessing start method used (spawn, fork, or 
> forkserver). This is necessary to know for replicating the issue on our end.
> 
> In the meantime, I workaround I would suggest trying would be to use the  
> *chunksize* parameter (or *Iterator*) in pandas.read_csv(), and split it 
> across several jobs (at least 4+, more if you have additional cores) instead 
> of within a single one. It'd also be generally helpful to see if that 
> alleviates the problem, as it could possibly indicate an issue with running 
> out of memory when the dataframe is converted to pickle format (which often 
> increases the total size) within the process associated with the job.
> 
> --
> nosy: +aeros
> 
> ___
> Python tracker 
> <https://bugs.python.org/issue37294>
> ___

--

___
Python tracker 
<https://bugs.python.org/issue37294>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42245] concurrent.futures.ProcessPoolExecutor freezes depending on complexity

2020-11-02 Thread DanilZ 


New submission from DanilZ :

Note: problem occurs only after performing the RandomizedSearchCV...

When applying a function in a multiprocess using concurrent.futures if the 
function includes anything else other than print(), it is not executed and the 
process freezes.

Here is the code to reproduce.

from xgboost import XGBRegressor
from sklearn.model_selection import KFold
import concurrent.futures
from sklearn.datasets import make_regression
import pandas as pd
import numpy as np
from sklearn.model_selection import RandomizedSearchCV

# STEP 1
# 
# simulate RandomizedSearchCV

data = make_regression(n_samples=500, n_features=100, n_informative=10, 
n_targets=1, random_state=5)
X = pd.DataFrame(data[0])
y = pd.Series(data[1])
kf = KFold(n_splits = 3, shuffle = True, random_state = 5)
model = XGBRegressor(n_jobs = -1)
params = {
'min_child_weight': [0.1, 1, 5],
'subsample':[0.5, 0.7, 1.0],
'colsample_bytree': [0.5, 0.7, 1.0],
'eta':  [0.005, 0.01, 0.1],
'n_jobs':   [-1]
}
random_search = RandomizedSearchCV(
model,
param_distributions =   params,
n_iter =50,
n_jobs =-1,
refit = True, # necessary for 
random_search.best_estimator_
cv =kf.split(X,y),
verbose =   1,
random_state =  5
)
random_search.fit(X, np.array(y))

# STEP 2.0
# 
# test if multiprocessing is working in the first place

def just_print():
print('Just printing')

with concurrent.futures.ProcessPoolExecutor() as executor:
results_temp = [executor.submit(just_print) for i in range(0,12)]
# 


# STEP 2.1
# 
# test on a slightly more complex function

def fit_model():
# JUST CREATING A DATASET, NOT EVEN FITTING ANY MODEL!!! AND IT FREEZES
data = make_regression(n_samples=500, n_features=100, n_informative=10, 
n_targets=1, random_state=5)
# model = XGBRegressor(n_jobs = -1)
# model.fit(data[0],data[1])
print('Fit complete')

with concurrent.futures.ProcessPoolExecutor() as executor:
results_temp = [executor.submit(fit_model) for i in range(0,12)]
# 


Attached this code in a .py file.

--
components: macOS
files: concur_fut_freeze.py
messages: 380220
nosy: DanilZ, bquinlan, ned.deily, pitrou, ronaldoussoren
priority: normal
severity: normal
status: open
title: concurrent.futures.ProcessPoolExecutor freezes depending on complexity
versions: Python 3.7
Added file: https://bugs.python.org/file49562/concur_fut_freeze.py

___
Python tracker 
<https://bugs.python.org/issue42245>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42245] concurrent.futures.ProcessPoolExecutor freezes depending on complexity

2020-11-02 Thread DanilZ 


DanilZ  added the comment:

Hi Ken, thanks for a quick reply.

Here are the requested specs.
System:
Python 3.7.6
OS X 10.15.7

Packages:
XGBoost 1.2.0
sklearn 0.22.2
pandas 1.0.5
numpy 1.18.1

I can see that you have reduced the code, which now excludes the 
RandomizedSearchCV part. This (reduced) code runs without any problems on my 
side as well, but if running it after the RandomizedSearchCV, the last function 
fit_model() freezes in a multiprocess.

I will read through the docs, but at first it looks as the actual problem is in 
the concurrent.futures module, because the easy function just_print() runs 
without issues. So the freeze is triggered by adding minor complexity into the 
fit_model() function running in a multiprocess.

> On 2 Nov 2020, at 17:34, Ken Jin  wrote:
> 
> 
> Ken Jin  added the comment:
> 
> Hello, it would be great if you can you provide more details. Like your 
> Operating System and version, how many logical CPU cores there are on your 
> machine, and lastly the exact Python version with major and minor versions 
> included (eg. Python 3.8.2). Multiprocessing behaves differently depending on 
> those factors.
> 
> FWIW I reduced your code down to make it easier to read, and removed all the 
> unused variables:
> 
> import concurrent.futures
> from sklearn.datasets import make_regression
> 
> def just_print():
>print('Just printing')
> 
> def fit_model():
>data = make_regression(n_samples=500, n_features=100, n_informative=10, 
> n_targets=1, random_state=5)
>print('Fit complete')
> 
> if __name__ == '__main__':
>with concurrent.futures.ProcessPoolExecutor() as executor:
>results_temp = [executor.submit(just_print) for i in range(0,12)]
> 
>with concurrent.futures.ProcessPoolExecutor() as executor:
>results_temp = [executor.submit(fit_model) for i in range(0,12)]
> 
> The problem is that I am *unable* to reproduce the bug you are reporting on 
> Windows 10 64-bit, Python 3.7.6. The code runs till completion for both 
> examples. I have a hunch that your problem lies elsewhere in one of the many 
> libraries you imported.
> 
>>>> Note: problem occurs only after performing the RandomizedSearchCV...
> 
> Like you have noted, I went to skim through RandomizedSearchCV's source code 
> and docs. RandomizedSearchCV is purportedly able to use multiprocessing 
> backend for parallel tasks. By setting `n_jobs=-1` in your params, you're 
> telling it to use all logical CPU cores. I'm unsure of how many additional 
> processes and pools RandomizedSearchCV's spawns after calling it, but this 
> sounds suspicious. concurrent.futures specifically warns that this may 
> exhaust available workers and cause tasks to never complete. See 
> https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor 
> (the docs here are for ThreadPoolExecutor, but they still apply).
> 
> A temporary workaround might be to reduce n_jobs OR even better: use 
> scikit-learn's multiprocessing parallel backend that's dedicated for that, 
> and should have the necessary protections in place against such behavior. 
> https://joblib.readthedocs.io/en/latest/parallel.html#joblib.parallel_backend 
> 
> 
> TLDR: I don't think this is a Python bug and I'm in favor of this bug being 
> closed as `not a bug`.
> 
> --
> nosy: +kj
> 
> ___
> Python tracker 
> <https://bugs.python.org/issue42245>
> ___

--

___
Python tracker 
<https://bugs.python.org/issue42245>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42245] concurrent.futures.ProcessPoolExecutor freezes depending on complexity

2020-11-02 Thread DanilZ 

DanilZ  added the comment:

Here is a gif of what’s going on in my ActivityMonitor on a Mac while this code 
is executed:
https://gfycat.com/unselfishthatgraysquirrel 
<https://gfycat.com/unselfishthatgraysquirrel>

--

___
Python tracker 
<https://bugs.python.org/issue42245>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42245] concurrent.futures.ProcessPoolExecutor freezes depending on complexity

2020-11-02 Thread DanilZ 

DanilZ  added the comment:

FYI: I’ve tried all the three of the possible backends: ‘loky’ (default) / 
’threading’ / ‘multiprocessing’. None of them solved the problem.

> On 2 Nov 2020, at 17:34, Ken Jin  wrote:
> 
> A temporary workaround might be to reduce n_jobs OR even better: use 
> scikit-learn's multiprocessing parallel backend that's dedicated for that, 
> and should have the necessary protections in place against such behavior. 
> https://joblib.readthedocs.io/en/latest/parallel.html#joblib.parallel_backend 
> <https://joblib.readthedocs.io/en/latest/parallel.html#joblib.parallel_backend>

--

___
Python tracker 
<https://bugs.python.org/issue42245>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42245] concurrent.futures.ProcessPoolExecutor freezes depending on complexity

2020-11-02 Thread DanilZ 


DanilZ  added the comment:

Thank you so much for the input! I will study all the links you have sent:

Here is a screen recording of some additional experiments:
https://vimeo.com/user50681456/review/474733642/b712c12c2c 
<https://vimeo.com/user50681456/review/474733642/b712c12c2c>

--

___
Python tracker 
<https://bugs.python.org/issue42245>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42245] concurrent.futures.ProcessPoolExecutor freezes depending on complexity

2020-11-11 Thread DanilZ 


DanilZ  added the comment:

I have managed to solve the problem by inserting in the beginning of my program:

import multiprocessing
multiprocessing.set_start_method('forkserver')
as this is explained here: 
https://scikit-learn.org/stable/faq.html#why-do-i-sometime-get-a-crash-freeze-with-n-jobs-1-under-osx-or-linux
 
<https://scikit-learn.org/stable/faq.html#why-do-i-sometime-get-a-crash-freeze-with-n-jobs-1-under-osx-or-linux>
It works, but the shell looses some level of interactivity as the results 
intermediate results don't get printed as the program is executed.

> On 2 Nov 2020, at 19:03, Ken Jin  wrote:
> 
> 
> Ken Jin  added the comment:
> 
> Hmm apologies I'm stumped then. The only things I managed to surmise from 
> xgboost's and scikit-learn's GitHub issues is that this is a recurring issue 
> specifically when using GridSearchCV :
> 
> Threads with discussions on workarounds:
> https://github.com/scikit-learn/scikit-learn/issues/6627
> https://github.com/scikit-learn/scikit-learn/issues/5115
> 
> Issues reported:
> https://github.com/dmlc/xgboost/issues/2163
> https://github.com/scikit-learn/scikit-learn/issues/10533
> https://github.com/scikit-learn/scikit-learn/issues/10538 (this looks quite 
> similar to your issue)
> 
> Some quick workarounds I saw were:
> 1. Remove n_jobs argument from GridSearchCV
> 2. Use parallel_backend from sklearn.externals.joblib rather than 
> concurrent.futures so that the pools from both libraries don't have weird 
> interactions.
> 
> I recommend opening an issue on scikit-learn/XGBoost's GitHub. This seems 
> like a common problem that they face.
> 
> --
> 
> ___
> Python tracker 
> <https://bugs.python.org/issue42245>
> ___

--

___
Python tracker 
<https://bugs.python.org/issue42245>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42245] concurrent.futures.ProcessPoolExecutor freezes depending on complexity

2020-11-11 Thread DanilZ 


DanilZ  added the comment:

Hi Ken, 

Thanks for your comment.

Unfortunately at the time I can not upgrade to 3.8 to run this test. My whole 
system depends on 3.7 and some peculiarities of 3.8 need to be dealt with.

It would be great if someone with OSX and 3.8 could test this out, otherwise I 
will dig into this later creating a new environment.

> On 11 Nov 2020, at 18:12, Ken Jin  wrote:
> 
> 
> Ken Jin  added the comment:
> 
> Danil, thanks for finding the cause behind this. Could you check if the new 
> behavior in Python 3.8 and higher has the same problem on your machine 
> (without your fix)? multiprocessing on MacOS started using spawn in 3.8, and 
> I was wondering if it that fixed it.
> 
> What's new entry for 3.8 : 
> https://docs.python.org/3/whatsnew/3.8.html#multiprocessing
> 
> The bug tracked:
> https://bugs.python.org/issue33725
> 
> The PR for that
> https://github.com/python/cpython/pull/13603/files
> 
> --
> 
> ___
> Python tracker 
> <https://bugs.python.org/issue42245>
> ___

--

___
Python tracker 
<https://bugs.python.org/issue42245>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42245] concurrent.futures.ProcessPoolExecutor freezes depending on complexity

2020-11-19 Thread DanilZ 


DanilZ  added the comment:

Dear All,

Thanks for the great input. As described above it appears to be a MacOS problem.

--

___
Python tracker 
<https://bugs.python.org/issue42245>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com