Re: [R-pkg-devel] [UNCLASSIFIED] Remotes in description when submitting a package until a dependency is fixed

[Duncan Murdoch]

On 15/11/2022 11:59 p.m., Hugh Parsonage wrote:

I think you've misunderstood that excerpt.  By "temporary development
state", it means _between_ CRAN releases; packages in a development
state are not suitable for CRAN, as the policy states:


CRAN hosts packages of publication quality and is not a development platform.


You'll need to stop depending on that package until it's fixed and the
fix is on CRAN. That said, it looks like it might be relatively
straightforward to disentangle yourself from the package -- just
rewrite the offending example?


Another solution is to put a version of that package in your own drat 
repository, and use "Additional_repositories".  For example, at one 
point rgl used webshot2 before it was released, and I had


  Suggests:  webshot2, ...
  Additional_repositories:  https://dmurdoch.github.io/drat

with a copy of webshot2 in the drat repository.

The disadvantage of this approach is that you'll need to keep that 
repository up to date as the third party package evolves, and eventually 
remove the Additional_repositories: line from your DESCRIPTION, which 
requires your own package update.


See https://github.com/eddelbuettel/drat for instructions on setting up 
the drat repository.


Duncan Murdoch



On Wed, 16 Nov 2022 at 15:35, Bernd.Gruber  wrote:


Hi,

I have a package (dartR) that needs to be updated by CRAN (and got a time set 
until a certain date). It depends on a package that is currently showing errors 
in the CRAN results and therefore fails. The maintainer of that package is 
busily trying to rectify the error (as can be seen be repeated submissions in 
the last weeks), but was not able yet to fix it. As we are running out of time 
my approach would be to have a version of the package that fixes it and use 
Remotes: in the description. It runs fine without errors.

In the R-packages book I read the following:

"It's important to note that you should not submit your package to CRAN in the 
intermediate state, meaning with a Remotes field and with a dependency required at a 
version that's not available from CRAN or Bioconductor. For CRAN packages, this can only 
be a temporary development state, eventually resolved when the dependency updates on CRAN 
and you can bump your minimum version accordingly."

So is it okay to submit our package with a remote statement until the 
maintainer of the other package has fixed their issues?

Thanks in advance,
Bernd


==
Dr Bernd Gruber  )/_
  _.--..---"-,--c_
Professor Ecological Modelling  \|..'   ._O__)_
Tel: (02) 6206 3804 ,=._.+   _ \..--( /
Fax: (02) 6201 2328   \\.-''_.-' \ ( \_
Institute for Applied Ecology  `'''   `\__   /\
Faculty of Science and Technology  ')
University of Canberra   ACT 2601 AUSTRALIA
Email: bernd.gru...@canberra.edu.au
WWW: 
bernd-gruber
==

[UC Logo]

[Its time to control your Future. Apply now to study with Australia's fastest rising 
University. *QS, 2022]



The Ngunnawal people are the Traditional Custodians of the ACT where UC's Bruce 
Campus is situated and are an integral and celebrated part of UC's culture. We 
also acknowledge other First Nations Peoples.

Australian Government Higher Education Registered Provider (CRICOS) #00212K. 
TEQSA Provider ID: PRV12003 (Australian University)
Email 
Disclaimer

[UC Facebook][UC Twitter] 
 [UC Instagram]  [UC 
Linkedin]  [UC Youtube] 
 [University of Canberra] 

 [[alternative HTML version deleted]]

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel


__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel


__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [R-pkg-devel] Too many processes spawned on Windows and Debian, although only 2 cores should be used

[Riko Kelter]

Hi Ivan,

thanks for the info, I changed the check as you pointed out and it 
worked. R CMD build and R CMD check --as-cran run without errors or 
warnings on Linux + MacOS. However, I uploaded the package again at the 
WINBUILDER service and obtained the following weird error:

* checking re-building of vignette outputs ... ERROR
Check process probably crashed or hung up for 20 minutes ... killed
Most likely this happened in the example checks (?),
if not, ignore the following last lines of example output:

 End of example output (where/before crash/hang up occured ?) 

Strangely, there are no examples included in any .Rd file. Also, I 
checked whether a piece of code spawns new clusters. However, the 
critical lines are inside a function which is repeatedly called in the 
vignettes. The parallelized part looks as copied below. After the code 
is executed the cluster is stopped. I use registerDoSNOW(cl) because 
otherwise my progress bar does not work.


Code:

### CHECK CORES

chk <- tolower(Sys.getenv("_R_CHECK_LIMIT_CORES_", ""))
   if (nzchar(chk) && (chk != "false")){  # then limit the workers
     num_workers <- 2L
   } else {
     # use all cores
     num_workers <- parallel::detectCores()
   }

   chk <- Sys.getenv("_R_CHECK_LIMIT_CORES_", "")

   cl <- parallel::makeCluster(num_workers[1]-1) # not to overload your 
computer
   #doParallel::registerDoParallel(cl)
   doSNOW::registerDoSNOW(cl)

### SET UP PROGRESS BAR

pb <- progress_bar$new(
     format = "Iteration = :letter [:bar] :elapsed | expected time till 
finish: :eta",
     total = nsim,    # 100
     width = 120)

   progress_letter <- seq(1,nsim)  # token reported in progress bar

   # allowing progress bar to be used in foreach 
-
   progress <- function(n){
     pb$tick(tokens = list(letter = progress_letter[n]))
   }

   opts <- list(progress = progress)

### MAIN SIMULATION

if(method=="PP"){
     finalMatrix <- foreach::foreach(s=1:nsim, .combine=rbind, .packages 
= c("extraDistr", "fbst"), .options.snow = opts) %dopar% {
   tempMatrix = singleTrial_PP(s = s, n=nInit, responseMatrix = 
responseMatrix, nInit = nInit, Nmax = Nmax, batchsize = batchsize, a0 = 
a0, b0 = b0)

   tempMatrix #Equivalent to finalMatrix = cbind(finalMatrix, 
tempMatrix)
     }
   }

   if(method=="PPe"){
     refFunc = refFunc
     nu = nu
     shape1 = shape1
     shape2 = shape2
     if(refFunc == "flat"){
   finalMatrix <- foreach::foreach(s=1:nsim, .combine=rbind, 
.packages = c("extraDistr", "fbst"), .options.snow = opts) %dopar% {
     tempMatrix = singleTrial_PPe(s = s, n=nInit, responseMatrix = 
responseMatrix, nInit = nInit, Nmax = Nmax, batchsize = batchsize, a0 = 
a0, b0 = b0, refFunc = "flat")

     tempMatrix #Equivalent to finalMatrix = cbind(finalMatrix, 
tempMatrix)
   }
     }
     if(refFunc == "beta"){
   finalMatrix <- foreach::foreach(s=1:nsim, .combine=rbind, 
.packages = c("extraDistr", "fbst"), .options.snow = opts) %dopar% {
     tempMatrix = singleTrial_PPe(s = s, n=nInit, responseMatrix = 
responseMatrix, nInit = nInit, Nmax = Nmax, batchsize = batchsize, a0 = 
a0, b0 = b0, refFunc = "beta",
  shape1 = shape1, shape2 = shape2)

     tempMatrix #Equivalent to finalMatrix = cbind(finalMatrix, 
tempMatrix)
   }
     }
     if(refFunc == "binaryStep"){
   finalMatrix <- foreach::foreach(s=1:nsim, .combine=rbind, 
.packages = c("extraDistr", "fbst"), .options.snow = opts) %dopar% {
     tempMatrix = singleTrial_PPe(s = s, n=nInit, responseMatrix = 
responseMatrix, nInit = nInit, Nmax = Nmax, batchsize = batchsize, a0 = 
a0, b0 = b0, refFunc = "binaryStep",
  shape1 = shape1, shape2 = shape2, 
truncation = truncation)

     tempMatrix #Equivalent to finalMatrix = cbind(finalMatrix, 
tempMatrix)
   }
     }
     if(refFunc == "relu"){
   finalMatrix <- foreach::foreach(s=1:nsim, .combine=rbind, 
.packages = c("extraDistr", "fbst"), .options.snow = opts) %dopar% {
     tempMatrix = singleTrial_PPe(s = s, n=nInit, responseMatrix = 
responseMatrix, nInit = nInit, Nmax = Nmax, batchsize = batchsize, a0 = 
a0, b0 = b0, refFunc = "relu",
  shape1 = shape1, shape2 = shape2, 
truncation = truncation)

     tempMatrix #Equivalent to finalMatrix = cbind(finalMatrix, 
tempMatrix)
   }
     }
     if(refFunc == "palu"){
   finalMatrix <- foreach::foreach(s=1:nsim, .combine=rbind, 
.packages = c("extraDistr", "fbst"), .options.snow = opts) %dopar% {
     tempMatrix = singleTrial_PPe(s = s, n=nInit, responseMatrix = 
responseMatrix, nInit = nInit, Nmax = Nmax, batchsize = batchsize, a0 = 
a0, b0 = b0, refFunc = "palu",
  shape1 = shape1, shape2 = shape2, 
truncation = truncation)

     tempMatrix #Equi

Re: [R-pkg-devel] How ton print in the console from Rcpp

[Dirk Eddelbuettel]

On 14 November 2022 at 16:34, Elysée Aristide wrote:
| I was convinced that the issue is local. The warning message only appears
| on MAC.
| I did not try with another computer. I used CRAN macbuilder service and got
| the warning message. I guess this is also the computer CRAN uses.

Some further (and very patient) probing on GitHub in a related (older) issue
reveals that I was wrong here. My thanks for to Ivan for his patience, and
apologies to Elysée: it was indeed the combination of inclusing RcppArmadillo
prior to RcppNumerical and the combination of the former undefining NDEBUG
and the latter having assert() statement that caused your trouble.

And the `#undef NDEBUG` in RcppArmadillo, while not causing harm among the
1000+ packages on CRAN I can test against (and do test against each and every
time we make changes) simply does not do anything useful for RcppArmadillo or
Armadillo. But it leaves a possible side effect in packages using `assert()`
as RcppNumerical does.

So having tested a (conditional) removal (which one can override with a new
env.var. if one needs to) in another full reverse-dependencies run (which,
unsurprisingly, shows no ill effect as it showed no ill effect of having the
#undef) I will make that change on GitHub which will go into the next release
(which we aim to have less than monthly to comply with CRAN's desire of 'no
more than six releases in six months).

Thanks again to Ivan.

Cheers, Dirk

-- 
dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [R-pkg-devel] Too many processes spawned on Windows and Debian, although only 2 cores should be used

[Henrik Bengtsson]

Hello.

As already pointed out, the current R implementation treats any
non-empty value on _R_CHECK_LIMIT_CORES_ different from "false" as a
true value, e.g. "TRUE", "true", "T", "1", but also "donald duck".
Using '--as-cran' sets _R_CHECK_LIMIT_CORES_="TRUE", if unset.  If
already set, it'll not touch it.  So, it could be that a CRAN check
server already uses, say, _R_CHECK_LIMIT_CORES_="true".  We cannot
make assumptions about that.

To make your life, and an end-user's too, easier, I suggest just using

  num_workers <- 2L

without conditioning on running on CRAN or not.

Why? There are many problems with using parallel::detectCores().

First of all, it can return NA_integer_ on some systems, so you cannot
assume it gives a valid value (== error).  It can also return 1L,
which means your 'num_workers - 1' will give zero worker (== error).
You need to account for that if you rely on detectCores().

Second, detectCores() returns number of physical CPU cores. It's
getting more and more common to run in "cgroups" constrained
environments where your R process only gets access to a fraction of
these cores.  Such constrains are in place in many shared multi-user
HPC environments, and sometimes when using Linux containers (e.g.
Docker, Apptainer, and Podman).  A notable example of this is when
using the RStudio Cloud.  So, if you use detectCores() on those
systems, you'll actually over-parallelize, which slows things down and
you risk running out of memory. For example, you might launch 64
parallel workers when you only have access to four CPU cores.  Each
core will be clogged up by 16 workers.

Third, if you default to detectCores() and a user runs your code on a
machine shared by many users, the other users will not be happy.  Note
that the user will often not know they're overusing the machine.  So,
it's a loss-loss for everyone.

Fourth, detectCores() will return *all* physical CPU cores on the
current machine. These days we have machines with 128, 196, and more
cores.  Are you sure your software will actually run faster when using
that many cores?  The benefit from parallelization tends to decrease
as you add more workers until there is no longer an speed improvement.
If you keep adding more parallel workers you're going to see a
negative effect, i.e. you're penalized when parallelization too much.
So, be aware that when you test on 16 or 24 cores and things runs
really fast, that might not be the experience for other users, or
users in the future (who will have access to more CPU cores).

So, yes, I suggest not to use num_workers <- detectCores().  Pick a
fixed number instead, and the CRAN policy suggests using two.  You can
let the user control how many they want to use.  As a developer, it's
really really ... (read impossible) to know how many they want to use.

Cheers,

Henrik

PS. Note that detectCores() returns a single integer value (possible
NA_integer_).  Because of this, there is no need to subset with
num_workers[1]. I've seen this used in code; not sure where it comes
from but it looks like a cut'n'paste behavior.

On Wed, Nov 16, 2022 at 6:38 AM Riko Kelter  wrote:
>
> Hi Ivan,
>
> thanks for the info, I changed the check as you pointed out and it
> worked. R CMD build and R CMD check --as-cran run without errors or
> warnings on Linux + MacOS. However, I uploaded the package again at the
> WINBUILDER service and obtained the following weird error:
>
> * checking re-building of vignette outputs ... ERROR
> Check process probably crashed or hung up for 20 minutes ... killed
> Most likely this happened in the example checks (?),
> if not, ignore the following last lines of example output:
>
>  End of example output (where/before crash/hang up occured ?) 
>
> Strangely, there are no examples included in any .Rd file. Also, I
> checked whether a piece of code spawns new clusters. However, the
> critical lines are inside a function which is repeatedly called in the
> vignettes. The parallelized part looks as copied below. After the code
> is executed the cluster is stopped. I use registerDoSNOW(cl) because
> otherwise my progress bar does not work.
>
>
> Code:
>
> ### CHECK CORES
>
> chk <- tolower(Sys.getenv("_R_CHECK_LIMIT_CORES_", ""))
>if (nzchar(chk) && (chk != "false")){  # then limit the workers
>  num_workers <- 2L
>} else {
>  # use all cores
>  num_workers <- parallel::detectCores()
>}
>
>chk <- Sys.getenv("_R_CHECK_LIMIT_CORES_", "")
>
>cl <- parallel::makeCluster(num_workers[1]-1) # not to overload your
> computer
>#doParallel::registerDoParallel(cl)
>doSNOW::registerDoSNOW(cl)
>
> ### SET UP PROGRESS BAR
>
> pb <- progress_bar$new(
>  format = "Iteration = :letter [:bar] :elapsed | expected time till
> finish: :eta",
>  total = nsim,# 100
>  width = 120)
>
>progress_letter <- seq(1,nsim)  # token reported in progress bar
>
># allowing progress bar to b

Re: [R-pkg-devel] [UNCLASSIFIED] Remotes in description when submitting a package until a dependency is fixed

[Dirk Eddelbuettel]

On 16 November 2022 at 06:36, Duncan Murdoch wrote:
| On 15/11/2022 11:59 p.m., Hugh Parsonage wrote:
| > I think you've misunderstood that excerpt.  By "temporary development
| > state", it means _between_ CRAN releases; packages in a development
| > state are not suitable for CRAN, as the policy states:
| > 
| >> CRAN hosts packages of publication quality and is not a development 
platform.
| > 
| > You'll need to stop depending on that package until it's fixed and the
| > fix is on CRAN. That said, it looks like it might be relatively
| > straightforward to disentangle yourself from the package -- just
| > rewrite the offending example?
| 
| Another solution is to put a version of that package in your own drat 
| repository, and use "Additional_repositories".  For example, at one 
| point rgl used webshot2 before it was released, and I had
| 
|Suggests:  webshot2, ...
|Additional_repositories:  https://dmurdoch.github.io/drat
| 
| with a copy of webshot2 in the drat repository.
| 
| The disadvantage of this approach is that you'll need to keep that 
| repository up to date as the third party package evolves, and eventually 
| remove the Additional_repositories: line from your DESCRIPTION, which 
| requires your own package update.
| 
| See https://github.com/eddelbuettel/drat for instructions on setting up 
| the drat repository.

(Disclaimer: Author bias here.) Let me vouch for Additional_repositories:
along with packages in Suggests: (you cannot do this for Imports / Depends /
LinkingTo per CRAN Policy).  I have recently been relying on it a lot for a
package I (similarly) could not change that frequently at CRAN, and a
'semi-private / semi-public package' providing functionality I expect to be
provided by another package 'eventually'. With the caveat that you need the
Suggests: mechanism and hence _conditional_ use, it works just like CRAN.

In the meantime Additional_repositories: just works. It expands the set of
repos, so all operations relying on standard install.packages() calls (and
that includes using the remotes package to parse dependencies in CI, say)
just work, both at CRAN and in CI.

`drat` simply helps along by making it easier to provide compatible 
repositories.

Dirk

-- 
dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel

Re: [R-pkg-devel] [UNCLASSIFIED] Remotes in description when submitting a package until a dependency is fixed

[Henrik Bengtsson]

Not sure if it's already been said, but note that we can only use
'Additional_repositories' for *optional* dependencies, which are
listed under 'Suggests' (and less used 'Enhances').  We cannot use
them for *hard* dependencies, which are listed under 'Depends' or
'Imports'. From the CRAN Policies
(https://cran.r-project.org/web/packages/policies.html):

"Packages on which a CRAN package depends should be available from a
mainstream repository: if any mentioned in ‘Suggests’ or ‘Enhances’
fields are not from such a repository, where to obtain them at a
repository should be specified in an ‘Additional_repositories’ field
of the DESCRIPTION file (as a comma-separated list of repository URLs)
or for other means of access, described in the ‘Description’ field."

Currently, it's only CRAN and Bioconductor that are "mainstream" repositories.

/Henrik

On Wed, Nov 16, 2022 at 3:36 AM Duncan Murdoch  wrote:
>
> On 15/11/2022 11:59 p.m., Hugh Parsonage wrote:
> > I think you've misunderstood that excerpt.  By "temporary development
> > state", it means _between_ CRAN releases; packages in a development
> > state are not suitable for CRAN, as the policy states:
> >
> >> CRAN hosts packages of publication quality and is not a development 
> >> platform.
> >
> > You'll need to stop depending on that package until it's fixed and the
> > fix is on CRAN. That said, it looks like it might be relatively
> > straightforward to disentangle yourself from the package -- just
> > rewrite the offending example?
>
> Another solution is to put a version of that package in your own drat
> repository, and use "Additional_repositories".  For example, at one
> point rgl used webshot2 before it was released, and I had
>
>Suggests:  webshot2, ...
>Additional_repositories:  https://dmurdoch.github.io/drat
>
> with a copy of webshot2 in the drat repository.
>
> The disadvantage of this approach is that you'll need to keep that
> repository up to date as the third party package evolves, and eventually
> remove the Additional_repositories: line from your DESCRIPTION, which
> requires your own package update.
>
> See https://github.com/eddelbuettel/drat for instructions on setting up
> the drat repository.
>
> Duncan Murdoch
>
> >
> > On Wed, 16 Nov 2022 at 15:35, Bernd.Gruber  
> > wrote:
> >>
> >> Hi,
> >>
> >> I have a package (dartR) that needs to be updated by CRAN (and got a time 
> >> set until a certain date). It depends on a package that is currently 
> >> showing errors in the CRAN results and therefore fails. The maintainer of 
> >> that package is busily trying to rectify the error (as can be seen be 
> >> repeated submissions in the last weeks), but was not able yet to fix it. 
> >> As we are running out of time my approach would be to have a version of 
> >> the package that fixes it and use Remotes: in the description. It runs 
> >> fine without errors.
> >>
> >> In the R-packages book I read the following:
> >>
> >> "It's important to note that you should not submit your package to CRAN in 
> >> the intermediate state, meaning with a Remotes field and with a dependency 
> >> required at a version that's not available from CRAN or Bioconductor. For 
> >> CRAN packages, this can only be a temporary development state, eventually 
> >> resolved when the dependency updates on CRAN and you can bump your minimum 
> >> version accordingly."
> >>
> >> So is it okay to submit our package with a remote statement until the 
> >> maintainer of the other package has fixed their issues?
> >>
> >> Thanks in advance,
> >> Bernd
> >>
> >>
> >> ==
> >> Dr Bernd Gruber  )/_
> >>   _.--..---"-,--c_
> >> Professor Ecological Modelling  \|..'   ._O__)_
> >> Tel: (02) 6206 3804 ,=._.+   _ \..--( /
> >> Fax: (02) 6201 2328   \\.-''_.-' \ ( \_
> >> Institute for Applied Ecology  `'''   `\__   /\
> >> Faculty of Science and Technology  ')
> >> University of Canberra   ACT 2601 AUSTRALIA
> >> Email: bernd.gru...@canberra.edu.au
> >> WWW: 
> >> bernd-gruber
> >> ==
> >>
> >> [UC Logo]
> >>
> >> [Its time to control your Future. Apply now to study with Australia's 
> >> fastest rising University. *QS, 2022]
> >>
> >>
> >>
> >> The Ngunnawal people are the Traditional Custodians of the ACT where UC's 
> >> Bruce Campus is situated and are an integral and celebrated part of UC's 
> >> culture. We also acknowledge other First Nations Peoples.
> >>
> >> Australian Government Higher Education Registered Provider (CRICOS)