Re: [Rd] [External] Mitigating Stalls Caused by Call Deparse on Error

[Tierney, Luke]

This is probably best viewed in the context of other issue with
displaying calls, such as issues arising from calls constructed in
non-standard evaluation contexts. Might be good to move to a wishlist
item in bugzilla.

Best,

luke

On Sat, 13 Jul 2019, brodie gaslam via R-devel wrote:

> When large calls cause errors R may stall for extended periods.  This
> is particularly likely to happen with `do.call`, as in this example
> with a 24 second stall:
>
>     x <- runif(1e7)
>     system.time(do.call(paste0, list(abs, x)))  # intentional error
>     ## Error in (function (..., collapse = NULL)  :
>     ##   cannot coerce type 'builtin' to vector of type 'character'
>     ## Calls: system.time -> do.call -> 
>     ## Timing stopped at: 23.81 0.149 24.04
>
>     str(.Traceback)
>     ## Dotted pair list of 3
>     ##  $ : chr [1:2500488] "(function (..., collapse = NULL) " 
> ".Internal(paste0(list(...), collapse)))(.Primitive(\"abs\"), 
> c(0.718117154669017, " "0.494785501621664, 0.1453434410505, 
> 0.635028422810137, 0.0353180423844606, " "0.688418723642826, 
> 0.889682895969599, 0.728154224809259, 0.292572240810841, " ...
>     ##  $ : chr "do.call(paste0, list(abs, x))"
>     ##  $ : chr "system.time(do.call(paste0, list(abs, x)))"
>
> The first time I noticed this I thought my session had frozen/crashed
> as the standard interrupt ^C does not work during the deparse.  The
> stall happens when on error the call stack is deparsed prior to being
> saved to `.Traceback`.  The deparsing is done by `deparse1m` in native
> code, with the value of `getOption('deparse.max.lines')` which
> defaults to all lines.
>
> Since there is little value to seeing millions of lines of deparsed
> objects in `traceback()`, a simple work-around is to change the
> `deparse.max.lines` value:
>
>     options(deparse.max.lines=1)
>     system.time(do.call(paste0, list(abs, x)))
>     ## Error in (function (..., collapse = NULL)  :
>     ##   cannot coerce type 'builtin' to vector of type 'character'
>     ## Calls: system.time -> do.call -> 
>     ## Timing stopped at: 0 0 0
>
> Unfortunately this will affect all `deparse` calls, and it seems
> undesirable to pre-emptively enable it just for calls that might cause
> large deparses on error.
>
> An alternative is to store the actual calls instead of their deparsed
> character equivalents in `.Traceback`.  This defers the deparsing to
> when `traceback()` is used.  As per `?traceback`, it should be
> relatively safe to modify `.Traceback` in this way:
>
>> It is undocumented where .Traceback is stored nor that it is
>> visible, and this is subject to change.
>
> Deferring the deparsing to `traceback()` will give us the 
> opportunity to use a different `max.lines` setting as we do here
> with the patch applied:
>
>     system.time(do.call(paste0, list(abs, x)))
>     ## Error in (function (..., collapse = NULL)  :
>     ##   cannot coerce type 'builtin' to vector of type 'character'
>     ## Timing stopped at: 0.028 0 0.029
>
>     system.time(traceback(max.lines=3))
>     ## 3: (function (..., collapse = NULL)
>     ##    .Internal(paste0(list(...), collapse)))(.Primitive("abs"), 
> c(0.535468587651849,
>     ##    0.0540027911774814, 0.732930393889546, 0.565360915614292, 
> 0.544816034380347,
>     ## ...
>     ## 2: do.call(paste0, list(abs, x))
>     ## 1: system.time(do.call(paste0, list(abs, x)))
>     ##    user  system elapsed
>     ##   0.000   0.000   0.003
>
>
> More generally, it might be better to have a different smaller default
> value for the lines to deparse when calls  are _displayed_ as parts of
> lists, as is the case with `traceback()`, or in `print(sys.calls())` and
> similar.
>
> I attach a patch that does this.  I have run some basic tests
> and `make check-devel` passes. I can file an issue on bugzilla
> if that is a better place to have this conversation (assuming there
> is interest in it).
>
> Best,
>
> Brodie
>
> PS: for some reason my mail client is refusing to attach the patch so I paste 
> it
> starting on the next line.
> Index: src/gnuwin32/Rdll.hide
> ===
> --- src/gnuwin32/Rdll.hide    (revision 76827)
> +++ src/gnuwin32/Rdll.hide    (working copy)
> @@ -94,6 +94,7 @@
>   R_GetMaxNSize
>   R_GetMaxVSize
>   R_GetTraceback
> + R_GetTracebackParsed
>   R_GetVarLocSymbol
>   R_GetVarLocValue
>   R_HandlerStack
> Index: src/include/Defn.h
> ===
> --- src/include/Defn.h    (revision 76827)
> +++ src/include/Defn.h    (working copy)
> @@ -1296,6 +1296,7 @@
>  void NORET ErrorMessage(SEXP, int, ...);
>  void WarningMessage(SEXP, R_WARNING, ...);
>  SEXP R_GetTraceback(int);
> +SEXP R_GetTracebackParsed(int);
>  
>  R_size_t R_GetMaxVSize(void);
>  void R_SetMaxVSize(R_size_t);
> Index: src/library/base/R/traceback.R
> ===
> --- src/library/base/R/traceback.R   

[Rd] .travis.yml ... most likely included in error

[Spencer Graves]

Hello:


  Suggestions for whomever maintains "R CMD":


        1.  Can you change it so it doesn't complain about the 
presence of ".travis.yml", at least on GitHub?



        2.  What do you suggest people do to find error messages in 
the output?  I ask, because I'm getting "build failing" from 
travis-ci.org, but I can't see what failed in the following:



https://travis-ci.org/sbgraves237/Ecdat/builds/558528361?utm_medium=notification&utm_source=email


          https://api.travis-ci.org/v3/job/558528362/log.txt


  Or are thesejust Travis-CI problems?  If yes, what would you 
suggest they do?



  Thanks,
  Spencer Graves

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [External] Mitigating Stalls Caused by Call Deparse on Error

[brodie gaslam via R-devel]

Luke, thanks for considering the issue.  I would like to
try to separate the problem into two parts, as I _think_ 
your comments address primarily part 2 below:

1. How can we avoid significant and possibly crippling
   stalls on error with these non-standard calls.
2. What is the best way to view these non-standard calls.

I agree that issue 2. requires further thought and 
discussion under a wishlist issue ([on bugzilla now][1]).  
While I did raise issue 2., the patch itself makes no
attempt to resolve it.

The proposed patch resolves issue 1., which is a big
 usability problem.  Right now if you have the misfortune of
 using `do.call` with a big object and trigger an error, you 
have the choice of waiting a possibly long time for 
the deparse to complete, or killing your entire R session
 externally.

It seems a shame to allow a big usability issue for `do.call` 
to remain when there is a simple solution at hand, especially
since the complete deparse of large objects likely serves no
purpose in this case. Obviously, if storing the actual calls
 instead of their deparsed equivalents in .Traceback causes 
problems I'm not anticipating, then that's different.  
Is that the case?

Best,

Brodie.

[1]: https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17580

On Sunday, July 14, 2019, 8:52:45 AM EDT, Tierney, Luke 
 wrote: 





This is probably best viewed in the context of other issue with
displaying calls, such as issues arising from calls constructed in
non-standard evaluation contexts. Might be good to move to a wishlist
item in bugzilla.

Best,

luke

On Sat, 13 Jul 2019, brodie gaslam via R-devel wrote:

> When large calls cause errors R may stall for extended periods.  This
> is particularly likely to happen with `do.call`, as in this example
> with a 24 second stall:
>
>     x <- runif(1e7)
>     system.time(do.call(paste0, list(abs, x)))  # intentional error
>     ## Error in (function (..., collapse = NULL)  :
>     ##   cannot coerce type 'builtin' to vector of type 'character'
>     ## Calls: system.time -> do.call -> 
>     ## Timing stopped at: 23.81 0.149 24.04
>
>     str(.Traceback)
>     ## Dotted pair list of 3
>     ##  $ : chr [1:2500488] "(function (..., collapse = NULL) " 
> ".Internal(paste0(list(...), collapse)))(.Primitive(\"abs\"), 
> c(0.718117154669017, " "0.494785501621664, 0.1453434410505, 
> 0.635028422810137, 0.0353180423844606, " "0.688418723642826, 
> 0.889682895969599, 0.728154224809259, 0.292572240810841, " ...
>     ##  $ : chr "do.call(paste0, list(abs, x))"
>     ##  $ : chr "system.time(do.call(paste0, list(abs, x)))"
>
> The first time I noticed this I thought my session had frozen/crashed
> as the standard interrupt ^C does not work during the deparse.  The
> stall happens when on error the call stack is deparsed prior to being
> saved to `.Traceback`.  The deparsing is done by `deparse1m` in native
> code, with the value of `getOption('deparse.max.lines')` which
> defaults to all lines.
>
> Since there is little value to seeing millions of lines of deparsed
> objects in `traceback()`, a simple work-around is to change the
> `deparse.max.lines` value:
>
>     options(deparse.max.lines=1)
>     system.time(do.call(paste0, list(abs, x)))
>     ## Error in (function (..., collapse = NULL)  :
>     ##   cannot coerce type 'builtin' to vector of type 'character'
>     ## Calls: system.time -> do.call -> 
>     ## Timing stopped at: 0 0 0
>
> Unfortunately this will affect all `deparse` calls, and it seems
> undesirable to pre-emptively enable it just for calls that might cause
> large deparses on error.
>
> An alternative is to store the actual calls instead of their deparsed
> character equivalents in `.Traceback`.  This defers the deparsing to
> when `traceback()` is used.  As per `?traceback`, it should be
> relatively safe to modify `.Traceback` in this way:
>
>> It is undocumented where .Traceback is stored nor that it is
>> visible, and this is subject to change.
>
> Deferring the deparsing to `traceback()` will give us the 
> opportunity to use a different `max.lines` setting as we do here
> with the patch applied:
>
>     system.time(do.call(paste0, list(abs, x)))
>     ## Error in (function (..., collapse = NULL)  :
>     ##   cannot coerce type 'builtin' to vector of type 'character'
>     ## Timing stopped at: 0.028 0 0.029
>
>     system.time(traceback(max.lines=3))
>     ## 3: (function (..., collapse = NULL)
>     ##    .Internal(paste0(list(...), collapse)))(.Primitive("abs"), 
> c(0.535468587651849,
>     ##    0.0540027911774814, 0.732930393889546, 0.565360915614292, 
> 0.544816034380347,
>     ## ...
>     ## 2: do.call(paste0, list(abs, x))
>     ## 1: system.time(do.call(paste0, list(abs, x)))
>     ##    user  system elapsed
>     ##   0.000   0.000   0.003
>
>
> More generally, it might be better to have a different smaller default
> value for the lines to deparse when calls  are _displayed_ as parts of
> lists, as is the case with `traceback

Re: [Rd] .travis.yml ... most likely included in error

[Danny Smith]

Hi Spencer,

To get rid of the .travis.yml note add a .Rbuildignore file with this line:
^\.travis\.yml$
This will exclude the file from the build.

The build is failing because of a warning. As noted in the log, Travis is
treating a warning as an error:

Found warnings, treating as errors

It's a bit hard to find the warning in the logs because of all the pdfTeX
output but it's a warning about uncompressed datasets on line 3179:
https://travis-ci.org/sbgraves237/Ecdat/builds/558528361#L3179

You could try resaving your datasets, there are a couple of suggestions
here:
https://stackoverflow.com/questions/32605623/how-to-compress-saves-in-r-package-build/47074811

Cheers,
Danny


On Mon., 15 Jul. 2019, 00:32 Spencer Graves, 
wrote:

> Hello:
>
>
>Suggestions for whomever maintains "R CMD":
>
>
>  1.  Can you change it so it doesn't complain about the
> presence of ".travis.yml", at least on GitHub?
>
>
>  2.  What do you suggest people do to find error messages in
> the output?  I ask, because I'm getting "build failing" from
> travis-ci.org, but I can't see what failed in the following:
>
>
>
> https://travis-ci.org/sbgraves237/Ecdat/builds/558528361?utm_medium=notification&utm_source=email
>
>
>https://api.travis-ci.org/v3/job/558528362/log.txt
>
>
>Or are thesejust Travis-CI problems?  If yes, what would you
> suggest they do?
>
>
>Thanks,
>Spencer Graves
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] .travis.yml ... most likely included in error

[Spencer Graves]

Hi, Danny et al.:


   Thanks.  I found ".Rbuildignore" on my local copy that was, 
however, not tracked by git.  I added that, then committed and pushed 
the change.


   Obviously, I'm new to travis and am still learning how to read 
and understand what it's saying -- especially on how to find "warnings, 
treating as errors".


   Thanks again,
   Spencer Graves


On 2019-07-14 10:08, Danny Smith wrote:
> Hi Spencer,
>
> To get rid of the .travis.yml note add a .Rbuildignore file with this 
> line:
> ^\.travis\.yml$
> This will exclude the file from the build.
>
> The build is failing because of a warning. As noted in the log, Travis 
> is treating a warning as an error:
> Found warnings, treating as errors
> It's a bit hard to find the warning in the logs because of all the 
> pdfTeX output but it's a warning about uncompressed datasets on line 3179:
> https://travis-ci.org/sbgraves237/Ecdat/builds/558528361#L3179
>
> You could try resaving your datasets, there are a couple of 
> suggestions here:
> https://stackoverflow.com/questions/32605623/how-to-compress-saves-in-r-package-build/47074811
>
> Cheers,
> Danny
>
>
> On Mon., 15 Jul. 2019, 00:32 Spencer Graves, 
> mailto:spencer.gra...@prodsyse.com>> wrote:
>
> Hello:
>
>
>    Suggestions for whomever maintains "R CMD":
>
>
>      1.  Can you change it so it doesn't complain about the
> presence of ".travis.yml", at least on GitHub?
>
>
>      2.  What do you suggest people do to find error
> messages in
> the output?  I ask, because I'm getting "build failing" from
> travis-ci.org , but I can't see what failed
> in the following:
>
>
> 
> https://travis-ci.org/sbgraves237/Ecdat/builds/558528361?utm_medium=notification&utm_source=email
>
>
> https://api.travis-ci.org/v3/job/558528362/log.txt
>
>
>    Or are thesejust Travis-CI problems?  If yes, what would you
> suggest they do?
>
>
>    Thanks,
>    Spencer Graves
>
> __
> R-devel@r-project.org  mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>


[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] R-Forge > GitHub?

[Spencer Graves]

  Thanks to Ott and others, I now have separate GitHub 
repositories, one for each of the packages combined in the Ecdat R-Forge 
project.  In case it might help others in the future, I will summarize 
here key things I did to make this transition:



        1.  I first copied the "Ecfun" package into its own 
directory on my local computer and created a separate GitHub repository 
for that.  I lost the history in doing so, but I can live without that 
history.



        2.  I moved the contents of "~Ecdat/pkg/Ecdat" to "~Ecdat" 
and deleted the now-empty ""pkg/Ecdat" subdirectory.  I first tried to 
do this in RStudio, but wasn't sure it was done correctly.  So I used 
"“git reset --hard HEAD” to revert all that. Then I copied the material 
in Finder on my Mac, so I could see what I was doing.  Then I did "git 
add" of the individual files and folders in a Terminal plus "git rm -r 
pkg" to delete



        3.  Then I set up automatic checking for both packages 
using Travis CI as described by Hadley's "R Packages" 
(http://r-pkgs.had.co.nz/check.html).  [GitHub complained that 
".travis.yml" didn't belong there.  Danny Smith on Rd told me to add it 
to ".Rbuildignore".  I found it was already there, but ".Rbuildignore" 
was not part of the repository.  Now it is.]



        4.  Along the way, GitHub kept asking for my username and 
password, even though I had established SSH authentication.  I traced 
the problem to the ".git/config" that said, "url = 
https://github.com/sbgraves237/Ecdat".  I changed that line to read "url 
= https://sbgraves237:passw...@github.com/sbgraves237/Ecdat"; (where 
"password" is my GitHub password, which I had to change to make it work 
there, because it included "@" ;-)  oops.



        5.  I also had problems with "Warning: ‘inst/doc’ files":  
Those had not existed in the R-Forge versions but appeared somehow in 
migrating them to GitHub.  I deleted them in a Terminal with "git rm -r 
inst/doc".  After "git commit" and "git push", I found they had been 
deleted from the GitHub repository but not my local computer, so I 
deleted them locally -- without any apparent side effects.



  Thanks again,
  Spencer Graves


On 2019-07-03 23:30, Spencer Graves wrote:

    Thanks so much for your help.


    Now your "git push -u origin master" was "![rejected]", after
creating a new SSH and after your "git clone" and other "git remote
rename ..." commands seemed to work:


$ git clone...@github.com:joshuaulrich/tmp-ecfun.git  Ecdat
# Cloning into 'Ecdat'... done.

$ cd Ecdat/
$ git remote rename origin tmp
$ git remote add originhttps://github.com/sbgraves237/Ecdat
$ git push -u origin master
#[Username & password OK]
Tohttps://github.com/sbgraves237/Ecdat
   ! [rejected]    master -> master (fetch first)
error: failed to push some refs to 'https://github.com/sbgraves237/Ecdat'
hint: Updates were rejected because the remote contains work that you do
hint: not have locally. This is usually caused by another repository pushing
hint: to the same ref. You may want to first integrate the remote changes
hint: (e.g., 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
SpenceravessMBP:Ecdat sbgraves$


    Suggestions?
    Thanks again,
    Spencer Graves


On 2019-07-01 01:05, Ott Toomet wrote:

Apparently you created id_rsa key pair with a passphrase. Passphrase
is like an additional password protection layer on your ssh key.  I
don't know how did you create it.  But you can always create a new one
(you should delete the old one before you create a new one) using the
shell command 'ssh-keygen'.  It asks for a passphrase, just push enter
for an empty passphrase (twice).  You also have to update the ssh
public key (id_rsa.pub) on github by supplying the new public key
(id_rsa.pub).

There are some implications you should be aware of:
* if you delete id_rsa*, you cannot use any ssh authorization that
relies on this key any more (that's why you have to update on GH).
>From the what you write (... created 2 days ago) I guess you do not 
use these keys elsewhere but I may be wrong.

* if you supply empty passphrase, you bypass the optional extra
security layer.  I think this is OK for open source software
development on your personal computer but your preferences/situation
may differ.
* You cannot use the same keys with passphrase if they are created
without one.  This is likely not an issue, but if it turns out to be a
problem, you can either add passphrase to the default keys, or create
another set of keys, passphrase protected.

Cheers,
Ott


On Sun, Jun 30, 2019 at 9:51 PM Spencer Graves
mailto:spencer.gra...@prodsyse.com>> wrote:



 On 2019-06-30 06:58, Joshua Ulrich wrote:
 

 > I imported both packages into separate repositories:
 >https://github.com/joshuaulrich/tmp-ecdat
 >https://github.com/joshuaulrich/tmp-ecfun
 >
 > I changed your ema