On Mon, 2 Nov 2020 at 02:22, Simon Urbanek <simon.urba...@r-project.org> wrote: > > It looks like R sockets on Linux could do with TCP_NODELAY -- without (status > quo):
How many network packets are generated with and without it? If there are many small writes and thus setting TCP_NODELAY causes many small packets to be sent, it might make more sense to set TCP_QUICKACK instead. Iñaki > Unit: microseconds > expr min lq mean median uq max > clusterEvalQ(cl, iris) 1449.997 43991.99 43975.21 43997.1 44001.91 48027.83 > neval > 1000 > > exactly the same machine + R but with TCP_NODELAY enabled in R_SockConnect(): > > Unit: microseconds > expr min lq mean median uq max neval > clusterEvalQ(cl, iris) 156.125 166.41 180.8806 170.247 174.298 5322.234 1000 > > Cheers, > Simon > > > > On 2/11/2020, at 3:39 AM, Jeff <j...@vtkellers.com> wrote: > > > > I'm exploring latency overhead of parallel PSOCK workers and noticed that > > serializing/unserializing data back to the main R session is significantly > > slower on Linux than it is on Windows/MacOS with similar hardware. Is there > > a reason for this difference and is there a way to avoid the apparent > > additional Linux overhead? > > > > I attempted to isolate the behavior with a test that simply returns an > > existing object from the worker back to the main R session. > > > > library(parallel) > > library(microbenchmark) > > gcinfo(TRUE) > > cl <- makeCluster(1) > > (x <- microbenchmark(clusterEvalQ(cl, iris), times = 1000, unit = "us")) > > plot(x$time, ylab = "microseconds") > > head(x$time, n = 10) > > > > On Windows/MacOS, the test runs in 300-500 microseconds depending on > > hardware. A few of the 1000 runs are an order of magnitude slower but this > > can probably be attributed to garbage collection on the worker. > > > > On Linux, the first 5 or so executions run at comparable speeds but all > > subsequent executions are two orders of magnitude slower (~40 milliseconds). > > > > I see this behavior across various platforms and hardware combinations: > > > > Ubuntu 18.04 (Intel Xeon Platinum 8259CL) > > Linux Mint 19.3 (AMD Ryzen 7 1800X) > > Linux Mint 20 (AMD Ryzen 7 3700X) > > Windows 10 (AMD Ryzen 7 4800H) > > MacOS 10.15.7 (Intel Core i7-8850H) > > > > ______________________________________________ > > R-devel@r-project.org mailing list > > https://stat.ethz.ch/mailman/listinfo/r-devel > > > > ______________________________________________ > R-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel -- Iñaki Úcar ______________________________________________ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
