[R-pkg-devel] Issue with flang-new (segfault from C stack overflow)

[Jisca Huisman]

Hello,

My package sequoia contains Fortran code, and failed to pass the 
pre-test on Debian with the new flang-new compiler. I was able to 
reproduce the issue, but strongly suspect it is an issue with 
flang-new-17 rather than with my code. However, since in the past when I 
thought the problem was not with my code I usually eventually turned out 
to be wrong, I would be grateful for a 'second opinion' and/or advise on 
further checks to figure out what the issue might be.


The error in ..._Debian_00check.log during pretest was "Error: segfault 
from C stack overflow" when running one of the examples ( 
https://win-builder.r-project.org/incoming_pretest/sequoia_2.7.5_20231209_204908/Debian/ 
)


I was able to reproduce this error on Ubuntu 22.04 with R configured as 
follows:


LIBnn=lib64 \
       CC="clang-17" \
       CXX="clang++-17" \
       FC="flang-new-17" \
       FCFLAGS="-g -O2 -mtune=native" \
       CXXFLAGS="-g -O2 -mtune=native" \
       FFLAGS="-g -O2" \
       MAIN_LDFLAGS="-pthread" \
       ./configure -C --with-valgrind-instrumentation=2 \
       --with-system-valgrind-headers --with-x=no \
       --without-recommended-packages --enable-lto=yes


I isolated the problem in a minimal working example available here: 
https://github.com/JiscaH/flang_segfault_min_example . All that does is 
pass a vector of length N*N back and forth between R and Fortran. It 
works fine for very long vectors (tested up to length 5e8), but throws a 
segfault when I reshape a large array in Fortran to a vector to pass to 
R, both when using RESHAPE() and when using loops.


During installation of the minimal package there is a warning:

** building package indices
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
readelf: Warning: Unrecognized form: 0x22
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation 
path

* DONE (minWE)

and during installation of the sequoia package there are dozens of those 
warnings.


When running

R -d "valgrind --tool=memcheck --leak-check=full" --vanilla < test_example.R

where

test_example.R is:  minWE::test_fun(N=5e2); minWE::test_fun(N=5e3)

with Valgrind-3.22.0 I get no valgrind messages/warnings/errors when 
N=5e2, but a segfault and a sleuth of messages when N=5e3, see 
https://github.com/JiscaH/flang_segfault_min_example/blob/main/valgrind_output_v15.txt 
.


When configuring R with

LIBnn=lib64 \
       CC="gcc -std=gnu99" \
       CXX="g++ -fno-omit-frame-pointer" \
       FC="gfortran" \
       FCFLAGS="-g -O2 -mtune=native" \
       CXXFLAGS="-g -O2 -Wall -pedantic -mtune=native" \
       FFLAGS="-g -O2 -mtune=native" \
       MAIN_LDFLAGS="-pthread" \
       ./configure -C --with-valgrind-instrumentation=2 \
       --with-system-valgrind-headers --with-x=no \
       --without-recommended-packages --enable-lto=yes

It runs without issues.


If someone can confirm that I'm not overlooking anything, I will submit 
a bug report to https://github.com/llvm/llvm-project/issues , and hope 
that I can convince the CRAN team to accept my package despite failing 
the pre-test.



Thanks,

Jisca

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [R-pkg-devel] Issue with flang-new (segfault from C stack overflow)

[Jisca Huisman]

Hello Ivan & Tomas,

Thank you for your time and helpful suggestions!

The finer details of memory use and heap vs stack are still outside my 
comfort zone, but some trial and error shows that using an allocatable 
does indeed solve the issue. When using the largest value I expect users 
to use before running into Out Of Memory issues at other points in the 
code, I get


==3154== Warning: set address range perms: large range [0xd0ff3070, 
0x14834c470) (undefined)
==3154== Warning: set address range perms: large range [0x14834d040, 
0x1bf6a6440) (undefined)
==3154== Warning: set address range perms: large range [0x14834d028, 
0x1bf6a6458) (noaccess)


but no valgrind errors. So I'm happy with this fairly straightforward 
solution, thanks Ivan!





You might perhaps submit a bug report for flang-new, asking whether 
their heuristics for these cases are as intended, showing that they 
differ from gfortran.

Will do; I suspect gfortran may have some trick to make it work somehow.


You might get more help on mailing lists discussing Fortran language, 
specifically - this is not an R issue.


Since the original error was "segfault from C stack overflow" I was not 
convinced that this was a Fortran issue, but thanks for the suggestion - 
I will try to find those for future issues.




But in practice, yes, using "allocatable" should work much better for 
large arrays.


Good to know!




Best
Tomas 


Thanks,

Jisca



On 18-12-2023 16:06, Tomas Kalibera wrote:


On 12/18/23 15:09, Ivan Krylov wrote:

В Mon, 18 Dec 2023 11:06:16 +0100
Jisca Huisman  пишет:


I isolated the problem in a minimal working example available here:
https://github.com/JiscaH/flang_segfault_min_example . All that does
is pass a vector of length N*N back and forth between R and Fortran.
It works fine for very long vectors (tested up to length 5e8), but
throws a segfault when I reshape a large array in Fortran to a vector
to pass to R, both when using RESHAPE() and when using loops.

You've done an impressive amount of investigative work. Thank you for
reducing your problem to such a small example! My eyes are drawn to
these two lines:


  integer, intent(IN) :: N
  integer :: M(N,N)

If this was C, such a declaration would mean a variable-length array
that would have to be placed on the (limited-size) stack and eventually
overflow it. gfortran places the array on the heap, so the program
works:

   integer, intent(IN) :: N
   integer, intent(INOUT) :: V(N*N)
   integer :: M(N,N)
 1205:   48 63 db    movslq %ebx,%rbx
 1208:   b8 00 00 00 00  mov    $0x0,%eax
 120d:   48 85 db    test   %rbx,%rbx
 1210:   49 89 c4    mov    %rax,%r12
 1213:   4c 0f 49 e3 cmovns %rbx,%r12
 1217:   48 89 df    mov    %rbx,%rdi
 121a:   49 0f af fc imul   %r12,%rdi
 121e:   48 85 ff    test   %rdi,%rdi
 1221:   48 0f 48 f8 cmovs  %rax,%rdi
 1225:   48 c1 e7 02 shl    $0x2,%rdi
 1229:   b8 01 00 00 00  mov    $0x1,%eax
 122e:   48 0f 44 f8 cmove  %rax,%rdi
 1232:   e8 19 fe ff ff  callq  1050 
 1237:   48 89 c5    mov    %rax,%rbp
 123a:   4c 89 e7    mov    %r12,%rdi
 123d:   48 f7 d7    not    %rdi

(Looking at the address of M in GDB and comparing it with the output
of info proc mappings, I can confirm that it lives on the heap.)

flang-new makes M into a C-style VLA:

   integer, intent(IN) :: N
   integer, intent(INOUT) :: V(N*N)
   integer :: M(N,N)
 74ec:   48 63 17    movslq (%rdi),%rdx
 74ef:   89 d1   mov    %edx,%ecx
 74f1:   31 c0   xor    %eax,%eax
 74f3:   48 85 d2    test   %rdx,%rdx
 74f6:   48 0f 49 c2 cmovns %rdx,%rax
 74fa:   48 89 85 b0 fe ff ff    mov %rax,-0x150(%rbp)
 7501:   48 89 c2    mov    %rax,%rdx
 7504:   48 0f af d2 imul   %rdx,%rdx
 7508:   48 8d 34 95 0f 00 00    lea 0xf(,%rdx,4),%rsi
 750f:   00
 7510:   48 83 e6 f0 and $0xfff0,%rsi
 7514:   48 89 e2    mov    %rsp,%rdx
 7517:   48 29 f2    sub    %rsi,%rdx
 751a:   48 89 95 b8 fe ff ff    mov %rdx,-0x148(%rbp)
 7521:   48 89 d4    mov    %rdx,%rsp

(Looking at the value of the stack pointer in GDB after M(N,N) is
declared, I can see it way below the end of the stack and the loaded
shared libraries according to info proc mappings. GDB doesn't let me
see the address of M. The program crashes in `M = 42`, trying to
overwrite the code from the C standard library.)

Are Fortran processors allowed to place such "automatic data objects"
like integer :: M(N,

[R-pkg-devel] is Fortran write still strictly forbidden?

[Jisca Huisman]

Hello,

I like to use write() in Fortran code to combine text with some integers 
& doubles, to pass runtime information to R in a way that is prettier 
and more legible than with intpr() & dblepr(). In the past any calls to 
write() were strictly forbidden in Fortran code, as apparently it messed 
something up internally (I cannot recall the details). But from 'writing 
R extensions' it seems that there have been quite a few changes with 
respect to support for Fortran code, and it currently reads:


6.5.1 Printing from Fortran

On many systems Fortran|write|and|print|statements can be used, but the 
output may not interleave well with that of C, and may be invisible 
onGUIinterfaces. They are not portable and best avoided.


To be more specific, would the subroutine below be allowed? Is it needed 
to declare R >= 4.0 (?) in the package DESCRIPTION (& then use labelpr() 
instead of intpr() ?) Is there an alternative without write() to get the 
same result?


subroutine Rprint_pretty(iter, x)
     integer, intent(IN) :: iter
     double precision, intent(IN) :: x
     integer :: date_time_values(8), nchar, IntDummy(0)
     character(len=8) :: time_now
     character(len=200) :: msg_to_R

     call date_and_time(VALUES=date_time_values)
     write(time_now, '(i2.2,":",i2.2,":",i2.2)') date_time_values(5:7)
     write(msg_to_R, '(a8, " i: ", i5, "  value: ", f8.2)') time_now, 
iter, x

     nchar = len(trim(msg_to_R))
    call intpr(trim(msg_to_R), nchar, IntDummy, 0)

   end subroutine Rprint_pretty


Thanks!


[[alternative HTML version deleted]]

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [R-pkg-devel] is Fortran write still strictly forbidden?

[Jisca Huisman]

Dear Berwin & Erin,

Thanks for your replies. I should perhaps have clarified - this is for a 
package on CRAN.


In my experience anything with Fortran write will get a NOTE from CRAN's 
auto-check:


Check: compiled code, Result: NOTE
  File 'sequoia/libs/sequoia.so':
Found '_gfortran_st_write', possibly from 'write' (Fortran), 'print'
  (Fortran)
  Object: 'sequoia.o'
  
  Compiled code should not call entry points which might terminate R nor

  write to stdout/stderr instead of to the console, nor use Fortran I/O
  nor system RNGs.


In the past (2018) I've tried to `write` the iteration number to a char 
to then combine into the message send to R with `dplepr`, but when I 
tried to argue this was a false positive as it wasn't writing to a file 
I got the following reply from UL:




The manual says


In particular, any package that makes use of Fortran I/O will when 
compiled on Windows interfere with C I/O: when the Fortran I/O support 
code is initialized (typically when the package is loaded) the C 
stdout and stderr are switched to LF line endings.



and both of us read too quickly: when the Fortran I/O support code is 
initialized (typically when the package is loaded) the C stdout and 
stderr are switched to LF line endings.

This happens alway, even if you write to another variable.

You could of course write C interface code to do the conversion if you 
have to rely on write(). 


So what I'm wondering now - has the issue with the switch to LF line 
endings been fixed, and is it OK to use non-file `write`? Because I 
don't really want to make the time to figure out how to 'write C 
interface code to do the conversion' just for some prettier messages...



Thanks for your help!




On Thu, 09/05/2024 06:30, Berwin A Turlach wrote:

Hi Jisca,

On Wed, 8 May 2024 10:37:28 +0200
Jisca Huisman  wrote:


I like to use write() in Fortran code [...] But from 'writing R
extensions' it seems that there have been quite a few changes with
respect to support for Fortran code, and it currently reads:

6.5.1 Printing from Fortran

On many systems Fortran|write|and|print|statements can be used, but
the output may not interleave well with that of C, and may be
invisible onGUIinterfaces. They are not portable and best avoided.

I am not aware that there were any recent changes regarding printing
from Fortran recently, or that it was every strictly forbidden (perhaps
it is for packages that are submitted to CRAN?).  In fact, R-exts.texi
for R 1.0.0 states pretty much the same as what you quoted from the
current WRE manual:

   @subsection Printing from Fortran
   @cindex Printing from C

   In theory Fortran @code{write} and @code{print} statements can be
   used, but the output may not interleave well with that of C, and will
   be invisible on GUI interfaces.  They are best avoided.

   Three subroutines are provided to ease the output of information from
   Fortran code.

   @example
   subroutine dblepr(@var{label}, @var{nchar}, @var{data}, @var{ndata})
   subroutine realpr(@var{label}, @var{nchar}, @var{data}, @var{ndata})
   subroutine intpr (@var{label}, @var{nchar}, @var{data}, @var{ndata})
   @end example

Cheers,

Berwin


__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [R-pkg-devel] is Fortran write still strictly forbidden?

[Jisca Huisman]

Hello Berend,

Fantastic, thank you! I'm fairly sure I'll be able to figure something 
out with your code as a guide.


best,

Jisca


On Thu, 09/05/2024 13:08, Berend Hasselman wrote:

Hi,

Have a look at package nleqslv to see how you can avoid Fortran write 
completely.
Look at the file src/nwout.c and the calls of various functions therein in the 
other fortran files.

regards,

Berend Hasselman


On 08-05-2024, at 10:37, Jisca Huisman  wrote:

Hello,

I like to use write() in Fortran code to combine text with some integers
& doubles, to pass runtime information to R in a way that is prettier
and more legible than with intpr() & dblepr(). In the past any calls to
write() were strictly forbidden in Fortran code, as apparently it messed
something up internally (I cannot recall the details). But from 'writing
R extensions' it seems that there have been quite a few changes with
respect to support for Fortran code, and it currently reads:


6.5.1 Printing from Fortran

On many systems Fortran|write|and|print|statements can be used, but the
output may not interleave well with that of C, and may be invisible
onGUIinterfaces. They are not portable and best avoided.


To be more specific, would the subroutine below be allowed? Is it needed
to declare R >= 4.0 (?) in the package DESCRIPTION (& then use labelpr()
instead of intpr() ?) Is there an alternative without write() to get the
same result?


subroutine Rprint_pretty(iter, x)
 integer, intent(IN) :: iter
 double precision, intent(IN) :: x
 integer :: date_time_values(8), nchar, IntDummy(0)
 character(len=8) :: time_now
 character(len=200) :: msg_to_R

 call date_and_time(VALUES=date_time_values)
 write(time_now, '(i2.2,":",i2.2,":",i2.2)') date_time_values(5:7)
 write(msg_to_R, '(a8, " i: ", i5, "  value: ", f8.2)') time_now,
iter, x

 nchar = len(trim(msg_to_R))
call intpr(trim(msg_to_R), nchar, IntDummy, 0)

   end subroutine Rprint_pretty


Thanks!


[[alternative HTML version deleted]]

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel


__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [R-pkg-devel] Check errors for RMarkdown vignettes with error chunks

[Jisca huisman]

Hi Ivan,

In my experience, if you add purl=FALSE to those chunks, it will pass the
checks. This was the default but changed/ is the default on some platforms
not others when you have error=TRUE (don't recall details).

Jisca


On Fri, 7 Jun 2024, 16:05 Tim Taylor, 
wrote:

> Does this issue over in knitr shed some additional light ...
>
> https://github.com/yihui/knitr/issues/2338
>
> Tim
>
> On 07/06/2024 14:50, Ivan Krylov via R-package-devel wrote:
> > В Fri, 7 Jun 2024 09:10:03 +0200
> > "C.H."  пишет:
> >
> >> RMarkdown vignettes are sometimes used to demonstrate errors and one
> >> can write vignettes with `error` chunks.
> >>
> >> ```{r, error = TRUE}
> >> stop()
> >> ```
> >> But now CRAN also reports the same ERRORS for mac and linux `oldrel`
> >> (4.3.3).
> >>
> >> https://cran.r-project.org/web/checks/check_results_rio.html
> > It sounds like R Markdown, when tangling a vignette into an *.R file,
> > should provide additional wrapping for chunks with the error = TRUE
> > option set. Would a try({ chunk content }) suffice?
> >
> > R CMD check relies on the tangled script not to crash, and there is
> > currently no such wrapping in the generated output:
> > https://github.com/cran/rio/blob/master/inst/doc/remap.R
> >
>
> __
> R-package-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-package-devel
>

[[alternative HTML version deleted]]

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [R-pkg-devel] Build process generated non-portable files

[Jisca Huisman]

I'm not familar with fortran but the warning message is: "Found the following files with 
non-portbale usage of KIND:." In "Writing R Extensions/Writing portable packages/Portable 
Fortran code" it mentions the use of REAL(KIND=8)​ types are not portable since compilers can 
map values to different types (and these declarations are in the f90 files), but the compiler 
called the preprocessing of the files so it should be correct for this specific compiler.


This might actually be the main problem. As it says, REAL(KIND=8) does 
not have a consistent defintion, but the most common definition is 
double the precision of a default REAL (commonly KIND=8). So, replacing 
all 'REAL(KIND=8)' by 'DOUBLE PRECISION' in declarations should (might) 
solve the error message. Alternatively, you could use 
selected_real_kind(), see 
https://fortranwiki.org/fortran/show/Real+precision .


Best,

Jisca



On Tue, 13/08/2024 10:08, Ivan Krylov via R-package-devel wrote:

В Mon, 12 Aug 2024 18:24:30 +
David via R-package-devel  пишет:


in the intel environment (provided by rhub), the intel fortran
compiler generates intermediary files from *.f -> *__genmod.f90. The
R check then complains that the genmod files are not portable. I
include removal of the files in my cleanup file so the files do not
exist in the original package source or in the final source tarball
but it seems the portable files check is done after compilation but
before cleanup.

- Is there a way to get around this complaint?

Include the first 'all' target in your Makevars, make it depend on the
package shared library, and make it remove genmod files in the recipe:

all: $(SHLIB)
-rm -f arpack/*genmod.f90

A similar trick is mentioned in
.


- Should this complaint be here in the first place?

Perhaps not. If you manage to find an option for the ifx compiler that
disables creation of these files (a brief Web search says they are for
only the user to read, but most results are from early 2010's), post it
here. This may be a good argument to make this option recommended for R.


Shouldn't the portable files check only be performed on the shipped
source code?

False negatives are possible too, in case the installation stage
(configure and/or make) performs a lot of preprocessing, or unpacks
extra sources. You could be right; I don't have statistics to back
either option as less wasteful of effort.



__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [R-pkg-devel] Vignette build issue on Debian platform

[Jisca Huisman]

Hi Graeme,


There is only one function in my package that uses multiple cores:
bootSE(). However, this function is not called in any of my vignettes
because it is computationally expensive. The Rmd code has `eval = FALSE`
options for all of these cases. Therefore, I am at a loss as to why these
vignettes would be using more than 2 cores or what is causing this, or why
it only impacts the CRAN CMD check Debian system.


CRAN has changed its settings when compiling the vignette, and now 
*does* run everything in chunks with `eval = FALSE`.  The solution is to 
also add `purl=FALSE` to each chunk that should not be run.


If you dig through the mailing list archive you should be able to find 
further details on the why and how of this, or someone else might remember.


Best,

Jisca



On Tue, 14/01/2025 08:08, Graeme Hickey wrote:

Dear all,


I recently made some minor updates to the joineRML R package to squash some
CRAN CMD check NOTEs that have emerged over the past couple of years.  After
resubmitting to CRAN, a new NOTE appeared on the Debian platform only:



package joineRML_0.4.7.tar.gz does not pass the incoming checks

automatically, please see the following pre-tests (additional issue checks):


Windows: <

https://win-builder.r-project.org/incoming_pretest/joineRML_0.4.7_20250114_073418/Windows/00check.log

Status: OK
Debian: <

https://win-builder.r-project.org/incoming_pretest/joineRML_0.4.7_20250114_073418/Debian/00check.log

Status: 1 NOTE


The NOTE in particular is:



* checking re-building of vignette outputs ... [116s/30s] NOTE
Re-building vignettes had CPU time 3.9 times elapsed time


I emailed Uwe Ligges and got a reply:



We see
Flavor: r-devel-linux-x86_64-debian-gcc
Check: re-building of vignette outputs, Result: NOTE
Re-building vignettes had CPU time 3.9 times elapsed time



which suggests you are using more than 2 cores which is not permitted by

default.


There is only one function in my package that uses multiple cores:
bootSE(). However, this function is not called in any of my vignettes
because it is computationally expensive. The Rmd code has `eval = FALSE`
options for all of these cases. Therefore, I am at a loss as to why these
vignettes would be using more than 2 cores or what is causing this, or why
it only impacts the CRAN CMD check Debian system.


I noticed this has been a common reported issue (e.g.,
https://github.com/Rdatatable/data.table/issues/5658), but mainly for those
using data.table or openMP; I use neither. The same issue was showing for
some examples also, which I dealt with by wrapping in \dontrun{} — not
ideal, but seemed quickest way to avoid.


I have tried many things to fix this:

1. Modifying the vignettes to make faster
2. Adding various combinations of Sys.setenv("OMP_THREAD_LIMIT" = 1),
Sys.setenv("OMP_NUM_THREADS" = 1), options(Ncpus = 1), options(cores = 2)
to the Rmd vignettes
3. Checked the package builds using r-lib/actions (GitHub actions) and
r-hub Debian platform — I cannot reproduce this CRAN error


CRAN will not accept the update to my package until this NOTE is squashed.
If anyone is able to provide a recommendation on what I might do, I would
appreciate it.


Kind regards,


Graeme Hickey

[[alternative HTML version deleted]]

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel


__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel