[Rd] User interrupts parallel excution. Why it works or why not?
Hi all,
I just notice this interesting problem a few days before, but I cannot
find an answer for it. Say if you have a long-running job in a cluster
made by the parallel package and you decide to stop the execution by
pressing ctr + c in the terminal or the stop button in Rstudio for
some reason. After the interrupt, is the cluster still valid or not?
Below is a simple example code
library(parallel)
cl <- makeCluster(1)
## run and interrupt it
parLapply(cl, 1, function(i){Sys.sleep(10);Sys.getpid()})
## run another apply function to check the cluster status
parLapply(cl, 1, function(i)i)
>From my test result, the answer is yes. The worker is interrupted
immediately and the cluster is ready for the next command, but when I
create the worker manually, things seem different.
library(parallel)
cl <- makeCluster(1, manual = TRUE)
## run and interrupt it
parLapply(cl, 1, function(i){Sys.sleep(10);Sys.getpid()})
## run another apply function to check the cluster status
parLapply(cl, 1, function(i)i)
It seems like the worker does not know the manager has been
interrupted and still runs the current task. I have to wait for 10
seconds before I can get the result from the last line of the code and
the return value is the PID from the first apply function.
Both cases are reasonable, but it is surprising to see them at the
same time. I start to wonder how the user interrupt is handled, so I
looked at the code in the parallel package. However, it looks like
there is no related code, there is no try-catch statement in the
manager's code to handle the user interrupt, but the worker just
magically knows it should stop the current execution.
I can see this behavior in both Win and Ubuntu. It is kind of beyond
my knowledge, so I wonder if anyone can help me with it. Does the
cluster support the user interrupt? Why the above code works or not
works? Many thanks!
Best,
Jiefei
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] User interrupts parallel excution. Why it works or why not?
Hi Jiefei,
when you run the cluster "automatically" in your terminal and pres
Ctrl-C in Unix, both the master and the worker processes get the SIGINT
signal, because they belong to the same foreground process group. So you
are directly interrupting also the worker process.
When you run the cluster "manually", that is the master in one terminal
window and the worker in another, they are in different process groups
and if you pres Ctrl-C in the terminal running the master, only the
master will receive SIGINT signal, not the worker.
If you wanted to read the sources more, look for SIGINT handling in R,
the onintrEx() function, etc. A good source on signal handling is e.g.
http://www.linusakesson.net/programming/tty/
Best
Tomas
On 7/20/21 9:55 AM, Jiefei Wang wrote:
Hi all,
I just notice this interesting problem a few days before, but I cannot
find an answer for it. Say if you have a long-running job in a cluster
made by the parallel package and you decide to stop the execution by
pressing ctr + c in the terminal or the stop button in Rstudio for
some reason. After the interrupt, is the cluster still valid or not?
Below is a simple example code
library(parallel)
cl <- makeCluster(1)
## run and interrupt it
parLapply(cl, 1, function(i){Sys.sleep(10);Sys.getpid()})
## run another apply function to check the cluster status
parLapply(cl, 1, function(i)i)
From my test result, the answer is yes. The worker is interrupted
immediately and the cluster is ready for the next command, but when I
create the worker manually, things seem different.
library(parallel)
cl <- makeCluster(1, manual = TRUE)
## run and interrupt it
parLapply(cl, 1, function(i){Sys.sleep(10);Sys.getpid()})
## run another apply function to check the cluster status
parLapply(cl, 1, function(i)i)
It seems like the worker does not know the manager has been
interrupted and still runs the current task. I have to wait for 10
seconds before I can get the result from the last line of the code and
the return value is the PID from the first apply function.
Both cases are reasonable, but it is surprising to see them at the
same time. I start to wonder how the user interrupt is handled, so I
looked at the code in the parallel package. However, it looks like
there is no related code, there is no try-catch statement in the
manager's code to handle the user interrupt, but the worker just
magically knows it should stop the current execution.
I can see this behavior in both Win and Ubuntu. It is kind of beyond
my knowledge, so I wonder if anyone can help me with it. Does the
cluster support the user interrupt? Why the above code works or not
works? Many thanks!
Best,
Jiefei
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] User interrupts parallel excution. Why it works or why not?
Thanks for your explanation. This makes a lot of sense! SIGINT
handling is a blind spot to me, this introduction looks perfect!
Best,
Jiefei
On Tue, Jul 20, 2021 at 4:31 PM Tomas Kalibera wrote:
>
> Hi Jiefei,
>
> when you run the cluster "automatically" in your terminal and pres
> Ctrl-C in Unix, both the master and the worker processes get the SIGINT
> signal, because they belong to the same foreground process group. So you
> are directly interrupting also the worker process.
>
> When you run the cluster "manually", that is the master in one terminal
> window and the worker in another, they are in different process groups
> and if you pres Ctrl-C in the terminal running the master, only the
> master will receive SIGINT signal, not the worker.
>
> If you wanted to read the sources more, look for SIGINT handling in R,
> the onintrEx() function, etc. A good source on signal handling is e.g.
> http://www.linusakesson.net/programming/tty/
>
> Best
> Tomas
>
> On 7/20/21 9:55 AM, Jiefei Wang wrote:
> > Hi all,
> >
> > I just notice this interesting problem a few days before, but I cannot
> > find an answer for it. Say if you have a long-running job in a cluster
> > made by the parallel package and you decide to stop the execution by
> > pressing ctr + c in the terminal or the stop button in Rstudio for
> > some reason. After the interrupt, is the cluster still valid or not?
> > Below is a simple example code
> >
> > library(parallel)
> > cl <- makeCluster(1)
> > ## run and interrupt it
> > parLapply(cl, 1, function(i){Sys.sleep(10);Sys.getpid()})
> > ## run another apply function to check the cluster status
> > parLapply(cl, 1, function(i)i)
> >
> > From my test result, the answer is yes. The worker is interrupted
> > immediately and the cluster is ready for the next command, but when I
> > create the worker manually, things seem different.
> >
> > library(parallel)
> > cl <- makeCluster(1, manual = TRUE)
> > ## run and interrupt it
> > parLapply(cl, 1, function(i){Sys.sleep(10);Sys.getpid()})
> > ## run another apply function to check the cluster status
> > parLapply(cl, 1, function(i)i)
> >
> > It seems like the worker does not know the manager has been
> > interrupted and still runs the current task. I have to wait for 10
> > seconds before I can get the result from the last line of the code and
> > the return value is the PID from the first apply function.
> >
> > Both cases are reasonable, but it is surprising to see them at the
> > same time. I start to wonder how the user interrupt is handled, so I
> > looked at the code in the parallel package. However, it looks like
> > there is no related code, there is no try-catch statement in the
> > manager's code to handle the user interrupt, but the worker just
> > magically knows it should stop the current execution.
> >
> > I can see this behavior in both Win and Ubuntu. It is kind of beyond
> > my knowledge, so I wonder if anyone can help me with it. Does the
> > cluster support the user interrupt? Why the above code works or not
> > works? Many thanks!
> >
> > Best,
> > Jiefei
> >
> > __
> > R-devel@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] R-Forge http link redirection setup busted
Dear Dirk, Thank you for your feedback. There was a misconfigured redirection indeed. It should work well now. The top page does say "If you experience any problems or need help you can submit a support request to the R-Forge team or write an email to r-fo...@r-project.org. Thanks... and enjoy the site." But I`ll put it on the list of improvements anyway, to make the contact options more visible. Best, Gennadiy. On 19.07.21 22:03, Dirk Eddelbuettel wrote: (Sorry for posting here but the top-level r-forge page does not make it all that clear where to contact its admins.) When an old-style 'http' URL at r-forge is resolved / redirected to 'https', it is corrupted and the redirect breaks. That required a package re-upload for me a few days ago (as the CRAN url checker is unhappy about a borked URL) and you can see it live too e.g. via https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel and going to 'rcpp-devel archives' where the hover URL is (note 'http') http://lists.r-forge.r-project.org/pipermail/rcpp-devel/ which when followed 'naively' drops the / between .org and pipermail and gets a 404 https://lists.r-forge.r-project.orgpipermail/rcpp-devel/ Reinjecting the missing / helps as https://lists.r-forge.r-project.org/pipermail/rcpp-devel/ works fine. This is likely a typo / error in the redirection setup of the web server. I would be very grateful if someone could pass this on to whoever looks after r-forge these days. Best, Dirk __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] changes in some header files
We are working on rearranging some of our header files with the goal of making the installed headers correspond more closely to the C API available to packages. Packages that only use entry points and definitions that are part of the API as specified in Chapter 6 of Writing R Extensions should not be affected. I have committed an initial set of changes to R-devel in r80644. About 10 CRAN packages that use non-API features will fail under R-devel after these changes and their maintainers have been notified. If you are currently using non-API features in a package it would be a good idea to review what you are doing and to try to revise your code to work within the API. If you feel there are features missing in the API then you can suggest additions on this mailing list or bugzilla. Best, luke -- Luke Tierney Ralph E. Wareham Professor of Mathematical Sciences University of Iowa Phone: 319-335-3386 Department of Statistics andFax: 319-335-3017 Actuarial Science 241 Schaeffer Hall email: luke-tier...@uiowa.edu Iowa City, IA 52242 WWW: http://www.stat.uiowa.edu __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] R-Forge http link redirection setup busted
Hi Gennadiy, On 20 July 2021 at 10:25, Gennadiy Starostin wrote: | Thank you for your feedback. There was a misconfigured redirection | indeed. It should work well now. Fabulous, thanks as always. | The top page does say "If you experience any problems or need help you | can submit a support request to the R-Forge team or write an email to | r-fo...@r-project.org. | Thanks... and enjoy the site." Yes, thanks. I must have missed it. Marc also pointed me to it. | But I`ll put it on the list of improvements anyway, to make the contact | options more visible. That would be excellent. Thanks for looking after r-forge, even if my main use is down to the mailing list portion of it. Best, Dirk -- https://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
