Re: [Rd] Infrequent but steady NULL-pointer caused segfault in as.POSIXlt.POSIXct (R 3.4.4)
On 8/4/19 3:30 PM, Sun Yijiang wrote:
A reply from stackoverflow suggests I might have hit this bug:
https://sourceware.org/bugzilla/show_bug.cgi?id=14023
I can confirm that this glibc bug affects my system (latest CentOS 7).
However, as far as I know, R is not multithreaded in its core. Is it
possible that some library triggered this?
Yes, R is single threaded in its core, but that does not rule out
completely that this glibc bug has been triggered via say a bug in some
package that uses threads, either directly or indirectly via an external
library.
Best
Tomas
Regards,
Steve
Tomas Kalibera 于2019年8月2日周五 下午4:59写道：
In an optimized build, debug info is just an approximation. It might
help to debug in a build of R and packages without compiler
optimizations (-O0), where the debug information is accurate. However,
first I would try to modify the example to trigger more often, or try to
find external ways to make it trigger more often (e.g. via gctorture).
Then I would try to make the example smaller (not call gc() explicitly,
not call any external code - e.g. the jobs, etc) - any time the example
is reduced but still triggers the errors, the reasoning is made easier.
Once you have a repeatable situation in a build with reliable debug
symbols, debugging is easier too, e.g. sometimes a watchpoint helps to
find memory corruption. Please feel free to ask more when you have more
information/updates. If this ends up being a bug in R, please report
(and with a reproducible example, if it is not obvious from the source
code).
Best
Tomas
On 8/2/19 10:23 AM, Sun Yijiang wrote:
The R script I run daily for hours looks like this:
while (!finish) {
Sys.sleep(0.1)
time = as.integer(format(Sys.time(), "%H%M")) # always crash here
if (new.data.timestamp() <= time)
next
# ... do some jobs for about 2 minutes ...
gc()
}
Basically it waits for new data, which comes in every 10 minutes, and
do some jobs, then gc(), then loop again. It works great most of the
time, but crashes strangely once a month or so. Although infrequent,
it always crashes at the same place and gives the same error info,
like this:
*** caught segfault ***
address (nil), cause 'memory not mapped'
Traceback:
1: as.POSIXlt.POSIXct(x, tz)
2: as.POSIXlt(x, tz)
3: format.POSIXlt(as.POSIXlt(x, tz), format, usetz, ...)
4: structure(format.POSIXlt(as.POSIXlt(x, tz), format, usetz, ...),
names = names(x))
5: format.POSIXct(Sys.time(), format = "%H%M")
6: format(Sys.time(), format = "%H%M")
7: format(Sys.time(), format = "%H%M")
… …
I looked into the dumped core with gdb, and found something very strange:
gdb /usr/lib64/R/bin/exec/R ~/core.30387
(gdb) bt 5
#0 0x7f1dca844ff1 in __strlen_sse2_pminub () from /lib64/libc.so.6
#1 0x7f1dcb20e8f9 in Rf_mkChar (name=0x0) at envir.c:3725
#2 0x7f1dcb1dc225 in do_asPOSIXlt (call=,
op=, args=,
env=) at datetime.c:705
#3 0x7f1dcb22197f in bcEval (body=body@entry=0x4064b28,
rho=rho@entry=0xc449d38, useCache=useCache@entry=TRUE)
at eval.c:6473
#4 0x7f1dcb230370 in Rf_eval (e=0x4064b28,
rho=rho@entry=0xc449d38) at eval.c:624
(More stack frames follow…)
Tracing into src/main/datetime.c:705, it’s a simple string-making code:
SET_STRING_ELT(tzone, 1, mkChar(R_tzname[0]));
mkChar function is defined in envir.c:3725:
3723 SEXP mkChar(const char *name)
3724 {
3725 size_t len = strlen(name);
… …
gdb shows that the string pointer (name=0x0) mkChar received is NULL,
and subsequently strlen(NULL) caused the segfault. But quite
contradictorily, gdb shows the value passed to mkChar in the caller is
valid:
(gdb) frame 2
#2 0x7f1dcb1dc225 in do_asPOSIXlt (call=,
op=, args=,
env=) at datetime.c:705
705 datetime.c: No such file or directory.
(gdb) p tzname[0]
$1 = 0x4cf39c0 “CST”
R_tzname is an alias of tzname. (#define R_tzname tzname in the same file.)
At first, I suspect that some library may have messed up the memory
and accidentally zeroed tzname (a global variable). But with this gdb
trace, it shows that tzname is good, only that the pointer passed to
mkChar magically changed to zero. Like this:
mkChar(tzname[0]) // tzname[0] is “CST”, address 0x4cf39c
… …
SEXP mkChar(const char *name) // name should be 0x4cf39c, but gdb shows 0x0
{
size_t len = strlen(name); // segfault, as name is NULL
… …
The only theory I can think of so far is that, on calling mkChar, the
parameter passed on stack somehow got wiped out to zero by some buggy
code in R or library. At a higher level, what I see is this: If you
run format(Sys.time(), "%H%M”) a million times a day (together with
other codes of course), once in a month or so this simple line can
segfault.
I’m lost in this confusion, could someone please help me find the
right direction to further look into this problem?
Regards,
Steve
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/
Re: [Rd] MKL with latest Rs
Hi Tomas, That does the trick, thanks! In fact, I had those lines commented out in my Rmkl startup script but I can't remember why, since they're part of Intel's instructions. Strange that things still worked for somewhat older Rs. (Perhaps that's why I canceled them.) Dirk, if you're listening, it might help to add a note to your blog post. I guess it's not as simple as "update-alternatives and done" because environment variables still need to be set. Cheers, Bobby On Fri, 26 Jul 2019 at 06:22, Tomas Kalibera wrote: > Dear Robert, > > try setting these MKL environment variables > > export MKL_INTERFACE_LAYER=GNU,LP64 > export MKL_THREADING_LAYER=GNU > > the example should then report correct results (and regardless of the > version of R). There is no need to rebuild R from source, but when you > are building from source, this problem would be detected by checks in R > configure scripts. I can reproduce the problem on Ubuntu 19.04 and MKL > installed from binaries as described in Dirk's blog, both with R 3.5.2 > and R-devel, and the problem goes away on both when the variables are set. > > Best > Tomas > > On 7/17/19 3:18 PM, Robert B. Gramacy wrote: > > Dear R-devel team, > > > > I've encountered problems with recent Rs (>= 3.5.3) and MKL. I've > followed > > Dirk's (http://dirk.eddelbuettel.com/blog/2018/04/15/) and Intel's ( > > > https://software.intel.com/en-us/articles/quick-linking-intel-mkl-blas-lapack-to-r > ) > > instructions and many versions of MKL. All works fine in my Ubuntu 18 > > setup for R 3.5.2 and older. Carrying out the install and linking > > instructions works fine for later Rs, but calculations involving linear > > algebra become corrupted. E.g., following Dirk's very simple svd example > > from that link above ... > > > >> n <- 1e3 > >> X <- matrix(rnorm(n*n),n,n) > >> Xsvd <- svd(X) > >> Xsvd$d[1:5] > > [1] 63.09366 62.47868 62.26147 61.94530 61.84727 > > > > ... you get Eigenvalues that are bigger than 1e26 in R-devel, say, but > > (correct) values near 60 with R 3.5.2. > > > > I just wanted to throw that out there in case anyone else had come across > > something similar. I don't mind using a slightly older R/MKL for my > > stuff. I might speculate that the Microsoft R Open folks are having > > similar issues because they're still using older Rs. > > > > I haven't tried compiling and linking from source ( > > https://cran.r-project.org/doc/manuals/R-admin.html#MKL) because I > haven't > > had success with that in some time. > > > > -B > > > > [[alternative HTML version deleted]] > > > > __ > > R-devel@r-project.org mailing list > > https://stat.ethz.ch/mailman/listinfo/r-devel > > > [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] iconv: embedded nulls when converting to UTF-16
R-devel community: I have encountered some unexpected behavior using iconv, which may be the source of errors I am getting when connecting to a UTF-16 -encoded SQL Server database. A simple example is below. When researching this problem, I found r-devel reports of the same problem in threads from June 2010 and February, 2016, and that bug #16738 was posted to Bugzilla as a result. However, I have not been able to determine if the error is mine, if there is a known workaround, or it truly is a bug in R’s iconv implementation. Any additional help is appreciated. Thanks, Michael —— sessionInfo() #> R version 3.6.1 (2019-07-05). ## and replicated on R 3.4.1 on a cluster running CentOS Linux 7. #> Platform: x86_64-apple-darwin15.6.0 (64-bit) #> Running under: macOS Mojave 10.14.6 # #> locale: #> [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 #> attached base packages: #> [1] stats graphics grDevices utils datasets methods base #> loaded via a namespace (and not attached): #> [1] compiler_3.6.1 s <- "test" iconv(s, to="UTF-8”) #> [1] “test" iconv(s, to="UTF-16") #> Error in iconv(s, to = "UTF-16"): embedded nul in string: '\xfe\xff\0t\0e\0s\0t’ iconv(s, to="UTF-16BE") #> Error in iconv(s, to = "UTF-16BE"): embedded nul in string: '\0t\0e\0s\0t’ iconv(s, to="UTF-16LE") #> Error in iconv(s, to = "UTF-16LE"): embedded nul in string: 't\0e\0s\0t\0’ -- Michael Braun, Ph.D. Associate Professor of Marketing, and Corrigan Research Professor Cox School of Business Southern Methodist University Dallas, TX 75275 __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] iconv: embedded nulls when converting to UTF-16
On 03/08/2019 11:59 p.m., Braun, Michael wrote: R-devel community: I have encountered some unexpected behavior using iconv, which may be the source of errors I am getting when connecting to a UTF-16 -encoded SQL Server database. A simple example is below. When researching this problem, I found r-devel reports of the same problem in threads from June 2010 and February, 2016, and that bug #16738 was posted to Bugzilla as a result. However, I have not been able to determine if the error is mine, if there is a known workaround, or it truly is a bug in R’s iconv implementation. Any additional help is appreciated. R does not support embedded nulls in character strings, so it can't handle UTF-16 strings as character vectors. If you are using iconv(), you can set toRaw = TRUE, and you'll get a result containing the correct bytes. For example, > s <- "test" > iconv(s, to="UTF-16",toRaw=TRUE) [[1]] [1] fe ff 00 74 00 65 00 73 00 74 I don't know if SQL Server can handle raw vectors; I'd try to get it to accept UTF-8 input instead. Duncan Murdoch Thanks, Michael —— sessionInfo() #> R version 3.6.1 (2019-07-05). ## and replicated on R 3.4.1 on a cluster running CentOS Linux 7. #> Platform: x86_64-apple-darwin15.6.0 (64-bit) #> Running under: macOS Mojave 10.14.6 # #> locale: #> [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 #> attached base packages: #> [1] stats graphics grDevices utils datasets methods base #> loaded via a namespace (and not attached): #> [1] compiler_3.6.1 s <- "test" iconv(s, to="UTF-8”) #> [1] “test" iconv(s, to="UTF-16") #> Error in iconv(s, to = "UTF-16"): embedded nul in string: '\xfe\xff\0t\0e\0s\0t’ iconv(s, to="UTF-16BE") #> Error in iconv(s, to = "UTF-16BE"): embedded nul in string: '\0t\0e\0s\0t’ iconv(s, to="UTF-16LE") #> Error in iconv(s, to = "UTF-16LE"): embedded nul in string: 't\0e\0s\0t\0’ -- Michael Braun, Ph.D. Associate Professor of Marketing, and Corrigan Research Professor Cox School of Business Southern Methodist University Dallas, TX 75275 __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] CRAN incoming queue closed from Aug 09 to Aug 18
Dear package developers, the CRAN incoming queue will be closed from Aug 09, 2019 to Aug 18, 2019. Hence package submissions are only possible before and after that period. CRAN maintainance work and some work on a possibly forthcoming Windows toolchain will be pushed forward. Best, Uwe Ligges (for the CRAN team) __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] gfortran 9 quantreg bug
With extensive help from Dirk Eddelbuettel I have installed
docker on my mac mini from
https://hub.docker.com/editions/community/docker-ce-desktop-mac
which installs from a dmg in quite standard fashion. This has allowed
me to simulate running R in a Debian environment with gfortran-9 and
begin the process of debugging my ancient rqbr.f code.
Some further details:
0. After some initial testing, e.g.
docker --version
docker run hello-world
1. Download r-base and test os
docker pull r-base $ downloads r-base for us
docker run --rm -ti r-base R --version # to check we have the R we want
docker run --rm -ti r-base bash # now in shell, Ctrl-d to exit
2. Setup working directory -- tell Docker to run from the current directory
and access
cd projects/rq
docker run --rm -ti -v ${PWD}:/work -w /work r-base bash
This put the contents of projects/rq into the /work directory.
root@90521904fa86:/work# apt-get update
Get:1 http://cdn-fastly.deb.debian.org/debian sid InRelease [149 kB]
Get:2 http://cdn-fastly.deb.debian.org/debian testing InRelease [117 kB]
Get:3 http://cdn-fastly.deb.debian.org/debian sid/main amd64 Packages
[8,385 kB]
Get:4 http://cdn-fastly.deb.debian.org/debian testing/main amd64 Packages
[7,916 kB]
Fetched 16.6 MB in 4s (4,411 kB/s)
Reading package lists... Done
3. Get gcc-9 and gfortran-9
root@90521904fa86:/work# apt-get install gcc-9 gfortran-9
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
cpp-9 gcc-9-base libasan5 libatomic1 libcc1-0 libgcc-9-dev libgcc1
libgfortran-9-dev
libgfortran5 libgomp1 libitm1 liblsan0 libquadmath0 libstdc++6 libtsan0
libubsan1
Suggested packages:
gcc-9-locales gcc-9-multilib gcc-9-doc libgcc1-dbg libgomp1-dbg libitm1-dbg
libatomic1-dbg
libasan5-dbg liblsan0-dbg libtsan0-dbg libubsan1-dbg libquadmath0-dbg
gfortran-9-multilib
gfortran-9-doc libgfortran5-dbg libcoarrays-dev
The following NEW packages will be installed:
cpp-9 gcc-9 gfortran-9 libgcc-9-dev libgfortran-9-dev
The following packages will be upgraded:
gcc-9-base libasan5 libatomic1 libcc1-0 libgcc1 libgfortran5 libgomp1
libitm1 liblsan0
libquadmath0 libstdc++6 libtsan0 libubsan1
13 upgraded, 5 newly installed, 0 to remove and 71 not upgraded.
Need to get 35.6 MB of archives.
After this operation, 107 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://cdn-fastly.deb.debian.org/debian testing/main amd64 libasan5
amd64 9.1.0-10 [390 kB]
Get:2 http://cdn-fastly.deb.debian.org/debian testing/main amd64 libubsan1
amd64 9.1.0-10 [128 kB]
Get:3 http://cdn-fastly.deb.debian.org/debian testing/main amd64 libtsan0
amd64 9.1.0-10 [295 kB]
Get:4 http://cdn-fastly.deb.debian.org/debian testing/main amd64 gcc-9-base
amd64 9.1.0-10 [190 kB]
Get:5 http://cdn-fastly.deb.debian.org/debian testing/main amd64 libstdc++6
amd64 9.1.0-10 [500 kB]
Get:6 http://cdn-fastly.deb.debian.org/debian testing/main amd64
libquadmath0 amd64 9.1.0-10 [145 kB]
Get:7 http://cdn-fastly.deb.debian.org/debian testing/main amd64 liblsan0
amd64 9.1.0-10 [137 kB]
Get:8 http://cdn-fastly.deb.debian.org/debian testing/main amd64 libitm1
amd64 9.1.0-10 [27.6 kB]
Get:9 http://cdn-fastly.deb.debian.org/debian testing/main amd64 libgomp1
amd64 9.1.0-10 [88.1 kB]
Get:10 http://cdn-fastly.deb.debian.org/debian testing/main amd64
libgfortran5 amd64 9.1.0-10 [633 kB]
Get:11 http://cdn-fastly.deb.debian.org/debian testing/main amd64 libcc1-0
amd64 9.1.0-10 [47.7 kB]
Get:12 http://cdn-fastly.deb.debian.org/debian testing/main amd64
libatomic1 amd64 9.1.0-10 [9,012 B]
Get:13 http://cdn-fastly.deb.debian.org/debian testing/main amd64 libgcc1
amd64 1:9.1.0-10 [40.5 kB]
Get:14 http://cdn-fastly.deb.debian.org/debian testing/main amd64 cpp-9
amd64 9.1.0-10 [9,667 kB]
Get:15 http://cdn-fastly.deb.debian.org/debian testing/main amd64
libgcc-9-dev amd64 9.1.0-10 [2,346 kB]
Get:16 http://cdn-fastly.deb.debian.org/debian testing/main amd64 gcc-9
amd64 9.1.0-10 [9,945 kB]
Get:17 http://cdn-fastly.deb.debian.org/debian testing/main amd64
libgfortran-9-dev amd64 9.1.0-10 [676 kB]
Get:18 http://cdn-fastly.deb.debian.org/debian testing/main amd64
gfortran-9 amd64 9.1.0-10 [10.4 MB]
Fetched 35.6 MB in 6s (6,216 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 17787 files and directories currently installed.)
Preparing to unpack .../libasan5_9.1.0-10_amd64.deb ...
Unpacking libasan5:amd64 (9.1.0-10) over (9.1.0-8) ...
Preparing to unpack .../libubsan1_9.1.0-10_amd64.deb ...
Unpacking libubsan1:amd64 (9.1.0-10) over (9.1.0-8) ...
P
Re: [Rd] MKL with latest Rs
On 26 July 2019 at 08:48, Robert B. Gramacy wrote: | That does the trick, thanks! In fact, I had those lines commented out in | my Rmkl startup script but I can't remember why, since they're part of | Intel's instructions. Strange that things still worked for somewhat older | Rs. (Perhaps that's why I canceled them.) | | Dirk, if you're listening, it might help to add a note to your blog post. | I guess it's not as simple as "update-alternatives and done" because | environment variables still need to be set. Only if you want correct results :) Will do, if I remember once home. Thanks for the suggestion. Dirk -- http://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
