[Rd] Parallel computing: how to transmit multiple parameters to a function in parLapply?
Hi R-developers
In the package Parallel, the function parLapply(cl, x, f) seems to allow
transmission of only one parameter (x) to the function f. Hence in order to
compute f(x, y) parallelly, I had to define f(x, y) as f(x) and tried to
access y within the function, whereas y was defined outside of f(x).
Script:
library(parallel)
f <- function(x) {
z <- 2 * x + .GlobalEnv$y # Try to access y in the global scope.
return(z)
}
np <- detectCores(logical = FALSE) # Two cores of my laptop
x <- seq(1, 10, by = 1)
y <- 0.5 # Y may be an array in reality.
cl <- makeCluster(np) # initiate the cluster
r <- parLapply(cl, x, f) # apply f to x for parallel computing
stopCluster(cl)
The r was a list with 10 empty elements which means f failed to access y.
Then I tested f without parallel computing:
z <- f(x)
print(z)
[1] 2.5 4.5 6.5 8.5 10.5 12.5 14.5 16.5 18.5 20.5
The results indicates that we can access y using .GlobalEnv$y in a function
without parLapply.
The question is, is there any method for me to transmit y to f, or access y
within f during parallel computing?
The version of my R is 3.0.1 and I am running it on a Win8-64x system.
Thanks,
Yu
--
View this message in context:
http://r.789695.n4.nabble.com/Parallel-computing-how-to-transmit-multiple-parameters-to-a-function-in-parLapply-tp4682667.html
Sent from the R devel mailing list archive at Nabble.com.
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Parallel computing: how to transmit multiple parameters to a function in parLapply?
This works:
clusterExport(cl, c("f","y"), envir=environment())
r <- parLapply(cl, x, function(x) f(x,y))
You need to export your function (“f”) and additional variables (“y”), and then
define that function inside parLapply ("f(x,y)”). If you were to also make use
of
additional libraries (or source some scripts) then you should also consult
“clusterEvalQ”.
The makeCluster command (at least in windows via socket) just initializes new R
processes which do not know about your functions or variables unless you
export those to them.
Perhaps a question best suited for R-help.
Alexios
On 24 Dec 2013, at 06:15, Yu Wan wrote:
> Hi R-developers
>
> In the package Parallel, the function parLapply(cl, x, f) seems to allow
> transmission of only one parameter (x) to the function f. Hence in order to
> compute f(x, y) parallelly, I had to define f(x, y) as f(x) and tried to
> access y within the function, whereas y was defined outside of f(x).
>
> Script:
>
> library(parallel)
>
> f <- function(x) {
> z <- 2 * x + .GlobalEnv$y # Try to access y in the global scope.
> return(z)
> }
>
> np <- detectCores(logical = FALSE) # Two cores of my laptop
> x <- seq(1, 10, by = 1)
> y <- 0.5 # Y may be an array in reality.
> cl <- makeCluster(np) # initiate the cluster
> r <- parLapply(cl, x, f) # apply f to x for parallel computing
> stopCluster(cl)
>
> The r was a list with 10 empty elements which means f failed to access y.
>
> Then I tested f without parallel computing:
> z <- f(x)
> print(z)
> [1] 2.5 4.5 6.5 8.5 10.5 12.5 14.5 16.5 18.5 20.5
>
> The results indicates that we can access y using .GlobalEnv$y in a function
> without parLapply.
>
> The question is, is there any method for me to transmit y to f, or access y
> within f during parallel computing?
>
> The version of my R is 3.0.1 and I am running it on a Win8-64x system.
>
> Thanks,
>
> Yu
>
>
>
> --
> View this message in context:
> http://r.789695.n4.nabble.com/Parallel-computing-how-to-transmit-multiple-parameters-to-a-function-in-parLapply-tp4682667.html
> Sent from the R devel mailing list archive at Nabble.com.
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Parallel computing: how to transmit multiple parameters to a function in parLapply?
You can put the function of interest and any global
variables it needs into a private environment, which gets sent
along with the function to the child processes. E.g.
library(parallel)
cl3 <- makeCluster(3)
y <- c(1,100,1)
addY <- function(x) x + y
withGlobals <- function(FUN, ...){
environment(FUN) <- list2env(list(...))
FUN
}
parLapply(cl3, 1:4, withGlobals(addY, y=y))
# [[1]]
# [1] 2 101 10001
#
# [[2]]
# [1] 3 102 10002
# ...
Bill Dunlap
Spotfire, TIBCO Software
wdunlap tibco.com
> -Original Message-
> From: r-devel-boun...@r-project.org [mailto:r-devel-boun...@r-project.org] On
> Behalf
> Of Yu Wan
> Sent: Monday, December 23, 2013 10:16 PM
> To: r-devel@r-project.org
> Subject: [Rd] Parallel computing: how to transmit multiple parameters to a
> function in
> parLapply?
>
> Hi R-developers
>
> In the package Parallel, the function parLapply(cl, x, f) seems to allow
> transmission of only one parameter (x) to the function f. Hence in order to
> compute f(x, y) parallelly, I had to define f(x, y) as f(x) and tried to
> access y within the function, whereas y was defined outside of f(x).
>
> Script:
>
> library(parallel)
>
> f <- function(x) {
> z <- 2 * x + .GlobalEnv$y # Try to access y in the global scope.
> return(z)
> }
>
> np <- detectCores(logical = FALSE) # Two cores of my laptop
> x <- seq(1, 10, by = 1)
> y <- 0.5 # Y may be an array in reality.
> cl <- makeCluster(np) # initiate the cluster
> r <- parLapply(cl, x, f) # apply f to x for parallel computing
> stopCluster(cl)
>
> The r was a list with 10 empty elements which means f failed to access y.
>
> Then I tested f without parallel computing:
> z <- f(x)
> print(z)
> [1] 2.5 4.5 6.5 8.5 10.5 12.5 14.5 16.5 18.5 20.5
>
> The results indicates that we can access y using .GlobalEnv$y in a function
> without parLapply.
>
> The question is, is there any method for me to transmit y to f, or access y
> within f during parallel computing?
>
> The version of my R is 3.0.1 and I am running it on a Win8-64x system.
>
> Thanks,
>
> Yu
>
>
>
> --
> View this message in context:
> http://r.789695.n4.nabble.com/Parallel-computing-how-
> to-transmit-multiple-parameters-to-a-function-in-parLapply-tp4682667.html
> Sent from the R devel mailing list archive at Nabble.com.
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] Object type name and class name
I came across the distinction between the name of an object and the name of the class that it belongs to in an oblique way again today, which made me question my acceptance that it would be natural for them to differ. I asked a question on SO here: http://stackoverflow.com/questions/20762559/why-is-the-name-of-an-object-type-different-from-the-name-of-the-class-it-belong I wonder if anyone on the R-Devel list has a better explanation for why the class name of an object and the object type name of an object should differ? Happy holidays, fg __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Object type name and class name
Fg, This is not really an r-devel question. It is more appropriate for r-help as far as I know. Please ask questions like it there in the future. Anyway, my understanding is that the type of an object has to do with how it is stored internally, whereas the class has to do with how it is dispatched on. For example, in the S3 system , it is entirely reasonable to do the following: > x = 1:10 > class(x) = "myspecialint" > x [1] 1 2 3 4 5 6 7 8 9 10 attr(,"class") [1] "myspecialint" > print.myspecialint = function(x, ...) print(mean(x)) > print(x) [1] 5.5 > typeof(x) [1] "integer" As you can see, changing the "class" of x did not change how it was stored internally. Another canonical example is the matrix class. Matrices in R are stored as vectors of the relevant type, with additional attributes indicating their dimension. So while there is a matrix class, there is no matrix type. > x = matrix(1:10, nrow=2) > x [,1] [,2] [,3] [,4] [,5] [1,]13579 [2,]2468 10 > typeof(x) [1] "integer" HTH, ~G On Tue, Dec 24, 2013 at 12:26 PM, Fg Nu wrote: > > > > I came across the distinction between the name of an object and the name > of the class that it belongs to in an oblique way again today, which made > me question my acceptance that it would be natural for them to differ. > > I asked a question on SO here: > > http://stackoverflow.com/questions/20762559/why-is-the-name-of-an-object-type-different-from-the-name-of-the-class-it-belong > > I wonder if anyone on the R-Devel list has a better explanation for why > the class name of an object and the object type name of an object should > differ? > > Happy holidays, > > fg > > __ > R-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel > -- Gabriel Becker Graduate Student Statistics Department University of California, Davis [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Object type name and class name
Gabriel, [I understand that this is not about R development, but it seemed that this was a question about R internals that would be better answered on the R-devel list.] Thanks for your answer. Yes, this is what the person who answered on SO said as well. But my question is about terminology really. Why is the internal storage type called object type, which clashes with my understanding of an object being an instance of a class, and hence should share the same name? This is a purely technical notion, which should have nothing to do with the physical reality of C storage and types. Thanks. On Wednesday, December 25, 2013 3:02 AM, Gabriel Becker wrote: Fg, > >This is not really an r-devel question. It is more appropriate for r-help as >far as I know. Please ask questions like it there in the future. > >Anyway, my understanding is that the type of an object has to do with how it >is stored internally, whereas the class has to do with how it is dispatched >on. For example, in the S3 system , it is entirely reasonable to do the >following: > > >> x = 1:10 >> class(x) = "myspecialint" >> x > [1] 1 2 3 4 5 6 7 8 9 10 >attr(,"class") >[1] "myspecialint" >> print.myspecialint = function(x, ...) print(mean(x)) >> print(x) >[1] 5.5 >> typeof(x) >[1] "integer" > >As you can see, changing the "class" of x did not change how it was stored >internally. > > >Another canonical example is the matrix class. Matrices in R are stored as >vectors of the relevant type, with additional attributes indicating their >dimension. So while there is a matrix class, there is no matrix type. > > >> x = matrix(1:10, nrow=2) >> x > [,1] [,2] [,3] [,4] [,5] >[1,] 1 3 5 7 9 >[2,] 2 4 6 8 10 >> typeof(x) >[1] "integer" > > >HTH, >~G > > > > >On Tue, Dec 24, 2013 at 12:26 PM, Fg Nu wrote: > > >> >> >>I came across the distinction between the name of an object and the name of >>the class that it belongs to in an oblique way again today, which made me >>question my acceptance that it would be natural for them to differ. >> >>I asked a question on SO here: >>http://stackoverflow.com/questions/20762559/why-is-the-name-of-an-object-type-different-from-the-name-of-the-class-it-belong >> >>I wonder if anyone on the R-Devel list has a better explanation for why the >>class name of an object and the object type name of an object should differ? >> >>Happy holidays, >> >>fg >> >>__ >>R-devel@r-project.org mailing list >>https://stat.ethz.ch/mailman/listinfo/r-devel >> > > >-- >Gabriel Becker >Graduate Student >Statistics Department >University of California, Davis > > > [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Object type name and class name
It sounds like you're attempting to apply a version of object orientation (objects being instances of classes) which does not apply to R (in the S3 world anyway). That simply isn't how S3 classes work (in many ways S3 classes aren't really "classes" at all in the way you seem to be using the word, they are simply dispatch instructions) and that fact is AFAIK at the core of the language. I'm not a core developer of R, and so I don't speak for them, but the likelihood of that changes seems vanishingly small at this juncture. Whether it "should" work the way you describe is debatable (I am not convinced myself), but regardless of whether it should it almost surely isn't going to. ~G On Tue, Dec 24, 2013 at 1:57 PM, Fg Nu wrote: > > Gabriel, > > [I understand that this is not about R development, but it seemed that > this was a question about R internals that would be better answered on the > R-devel list.] > > Thanks for your answer. Yes, this is what the person who answered on SO > said as well. > > But my question is about terminology really. Why is the internal storage > type called object type, which clashes with my understanding of an object > being an instance of a class, and hence should share the same name? This is > a purely technical notion, which should have nothing to do with the > physical reality of C storage and types. > > Thanks. > > > On Wednesday, December 25, 2013 3:02 AM, Gabriel Becker < > gmbec...@ucdavis.edu> wrote: > > Fg, > > This is not really an r-devel question. It is more appropriate for r-help > as far as I know. Please ask questions like it there in the future. > > Anyway, my understanding is that the type of an object has to do with how > it is stored internally, whereas the class has to do with how it is > dispatched on. For example, in the S3 system , it is entirely reasonable to > do the following: > > > x = 1:10 > > class(x) = "myspecialint" > > x > [1] 1 2 3 4 5 6 7 8 9 10 > attr(,"class") > [1] "myspecialint" > > print.myspecialint = function(x, ...) print(mean(x)) > > print(x) > [1] 5.5 > > typeof(x) > [1] "integer" > > As you can see, changing the "class" of x did not change how it was stored > internally. > > Another canonical example is the matrix class. Matrices in R are stored as > vectors of the relevant type, with additional attributes indicating their > dimension. So while there is a matrix class, there is no matrix type. > > > x = matrix(1:10, nrow=2) > > x > [,1] [,2] [,3] [,4] [,5] > [1,]13579 > [2,]2468 10 > > typeof(x) > [1] "integer" > > HTH, > ~G > > > On Tue, Dec 24, 2013 at 12:26 PM, Fg Nu wrote: > > > > > I came across the distinction between the name of an object and the name > of the class that it belongs to in an oblique way again today, which made > me question my acceptance that it would be natural for them to differ. > > I asked a question on SO here: > > http://stackoverflow.com/questions/20762559/why-is-the-name-of-an-object-type-different-from-the-name-of-the-class-it-belong > > I wonder if anyone on the R-Devel list has a better explanation for why > the class name of an object and the object type name of an object should > differ? > > Happy holidays, > > fg > > __ > R-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel > > > > > -- > Gabriel Becker > Graduate Student > Statistics Department > University of California, Davis > > > -- Gabriel Becker Graduate Student Statistics Department University of California, Davis [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Object type name and class name
Gabriel, Thanks. Not trying to influence any changes, just trying to understand better what is going on when the term "object type" is used. From what you and the SO answer seem to suggest, it has little to do with OOP/classes at all. This would be fine if John Chambers weren't explicitly juxtaposing the two in the quote given in the original post. Regards. On Wednesday, December 25, 2013 3:49 AM, Gabriel Becker wrote: It sounds like you're attempting to apply a version of object orientation (objects being instances of classes) which does not apply to R (in the S3 world anyway). > >That simply isn't how S3 classes work (in many ways S3 classes aren't really >"classes" at all in the way you seem to be using the word, they are simply >dispatch instructions) and that fact is AFAIK at the core of the language. > >I'm not a core developer of R, and so I don't speak for them, but the >likelihood of that changes seems vanishingly small at this juncture. Whether >it "should" work the way you describe is debatable (I am not convinced >myself), but regardless of whether it should it almost surely isn't going to. > >~G > > > > >On Tue, Dec 24, 2013 at 1:57 PM, Fg Nu wrote: > > >> >>Gabriel, >> >> >>[I understand that this is not about R development, but it seemed that this >>was a question about R internals that would be better answered on the R-devel >>list.] >> >> >>Thanks for your answer. Yes, this is what the person who answered on SO said >>as well. >> >> >>But my question is about terminology really. Why is the internal storage type >>called object type, which clashes with my understanding of an object being an >>instance of a class, and hence should share the same name? This is a purely >>technical notion, which should have nothing to do with the physical reality >>of C storage and types. >> >> >>Thanks. >> >> >> >>On Wednesday, December 25, 2013 3:02 AM, Gabriel Becker >> wrote: >> >>Fg, >>> >>>This is not really an r-devel question. It is more appropriate for r-help as >>>far as I know. Please ask questions like it there in the future. >>> >>>Anyway, my understanding is that the type of an object has to do with how it >>>is stored internally, whereas the class has to do with how it is dispatched >>>on. For example, in the S3 system , it is entirely reasonable to do the >>>following: >>> >>> x = 1:10 class(x) = "myspecialint" x >>> [1] 1 2 3 4 5 6 7 8 9 10 >>>attr(,"class") >>>[1] "myspecialint" print.myspecialint = function(x, ...) print(mean(x)) print(x) >>>[1] 5.5 typeof(x) >>>[1] "integer" >>> >>>As you can see, changing the "class" of x did not change how it was stored >>>internally. >>> >>> >>>Another canonical example is the matrix class. Matrices in R are stored as >>>vectors of the relevant type, with additional attributes indicating their >>>dimension. So while there is a matrix class, there is no matrix type. >>> >>> x = matrix(1:10, nrow=2) x >>> [,1] [,2] [,3] [,4] [,5] >>>[1,] 1 3 5 7 9 >>>[2,] 2 4 6 8 10 typeof(x) >>>[1] "integer" >>> >>> >>>HTH, >>>~G >>> >>> >>> >>> >>>On Tue, Dec 24, 2013 at 12:26 PM, Fg Nu wrote: >>> >>> I came across the distinction between the name of an object and the name of the class that it belongs to in an oblique way again today, which made me question my acceptance that it would be natural for them to differ. I asked a question on SO here: http://stackoverflow.com/questions/20762559/why-is-the-name-of-an-object-type-different-from-the-name-of-the-class-it-belong I wonder if anyone on the R-Devel list has a better explanation for why the class name of an object and the object type name of an object should differ? Happy holidays, fg __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel >>> >>> >>>-- >>>Gabriel Becker >>>Graduate Student >>>Statistics Department >>>University of California, Davis >>> >>> >>> > > >-- >Gabriel Becker >Graduate Student >Statistics Department >University of California, Davis > > > [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Object type name and class name
[Resending because HTML was scrubbed.] Gabriel, Thanks. Not trying to influence any changes, just trying to understand better what is going on when the term "object type" is used. From what you and the SO answer seem to suggest, it has little to do with OOP/classes at all. This would be fine if John Chambers weren't explicitly juxtaposing the two in the quote given in the original post. Regards. >On Wednesday, December 25, 2013 3:49 AM, Gabriel Becker >wrote: > >It sounds like you're attempting to apply a version of object orientation >(objects being instances of classes) which does not apply to R (in the S3 >world anyway). >> >>That simply isn't how S3 classes work (in many ways S3 classes aren't really >>"classes" at all in the way you seem to be using the word, they are simply >>dispatch instructions) and that fact is AFAIK at the core of the language. >> >>I'm not a core developer of R, and so I don't speak for them, but the >>likelihood of that changes seems vanishingly small at this juncture. Whether >>it "should" work the way you describe is debatable (I am not convinced >>myself), but regardless of whether it should it almost surely isn't going to. >> >>~G >> >> >> >> >>On Tue, Dec 24, 2013 at 1:57 PM, Fg Nu wrote: >> >> >>> >>>Gabriel, >>> >>> >>>[I understand that this is not about R development, but it seemed that this >>>was a question about R internals that would be better answered on the >>>R-devel list.] >>> >>> >>>Thanks for your answer. Yes, this is what the person who answered on SO said >>>as well. >>> >>> >>>But my question is about terminology really. Why is the internal storage >>>type called object type, which clashes with my understanding of an object >>>being an instance of a class, and hence should share the same name? This is >>>a purely technical notion, which should have nothing to do with the physical >>>reality of C storage and types. >>> >>> >>>Thanks. >>> >>> >>> >>>On Wednesday, December 25, 2013 3:02 AM, Gabriel Becker >>> wrote: >>> >>>Fg, This is not really an r-devel question. It is more appropriate for r-help as far as I know. Please ask questions like it there in the future. Anyway, my understanding is that the type of an object has to do with how it is stored internally, whereas the class has to do with how it is dispatched on. For example, in the S3 system , it is entirely reasonable to do the following: > x = 1:10 > class(x) = "myspecialint" > x [1] 1 2 3 4 5 6 7 8 9 10 attr(,"class") [1] "myspecialint" > print.myspecialint = function(x, ...) print(mean(x)) > print(x) [1] 5.5 > typeof(x) [1] "integer" As you can see, changing the "class" of x did not change how it was stored internally. Another canonical example is the matrix class. Matrices in R are stored as vectors of the relevant type, with additional attributes indicating their dimension. So while there is a matrix class, there is no matrix type. > x = matrix(1:10, nrow=2) > x [,1] [,2] [,3] [,4] [,5] [1,] 1 3 5 7 9 [2,] 2 4 6 8 10 > typeof(x) [1] "integer" HTH, ~G On Tue, Dec 24, 2013 at 12:26 PM, Fg Nu wrote: > > >I came across the distinction between the name of an object and the name >of the class that it belongs to in an oblique way again today, which made >me question my acceptance that it would be natural for them to differ. > >I asked a question on SO here: >http://stackoverflow.com/questions/20762559/why-is-the-name-of-an-object-type-different-from-the-name-of-the-class-it-belong > >I wonder if anyone on the R-Devel list has a better explanation for why >the class name of an object and the object type name of an object should >differ? > >Happy holidays, > >fg > >__ >R-devel@r-project.org mailing list >https://stat.ethz.ch/mailman/listinfo/r-devel > -- Gabriel Becker Graduate Student Statistics Department University of California, Davis >> >> >>-- >>Gabriel Becker >>Graduate Student >>Statistics Department >>University of California, Davis >> >> >> > > __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
