[Rd] Getting an R bugzilla account

[Nathan Sosnovske via R-devel]

Hi All,

I have a fix to this bug ( 
https://bugs.r-project.org/bugzilla3/show_bug.cgi?id=16454) and would like to 
submit a patch to the bug report on Bugzilla. I'd also like to start going 
through some of the other Windows-specific issues and start fixing those. The 
bug submission instructions indicate that I should ask here for a Bugzilla 
account. Is that still the correct procedure?

Thanks!

Nathan


[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Getting an R bugzilla account

[Nathan Sosnovske via R-devel]

Thank you for creating this! I'll go ahead and start working on bugs as I have 
time. I have a patch for the bug I linked below so you should see that soon. :)

Nathan

-Original Message-
From: Martyn Plummer [mailto:plumm...@iarc.fr] 
Sent: Tuesday, May 23, 2017 11:16 PM
To: r-devel@r-project.org; Nathan Sosnovske 
Subject: Re: [Rd] Getting an R bugzilla account

Thanks for your help Nathan. I have added a bugzilla account for you.

Martyn

On Tue, 2017-05-23 at 21:02 +0000, Nathan Sosnovske via R-devel wrote:
> Hi All,
> 
> I have a fix to this bug ( 
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.
> r-project.org%2Fbugzilla3%2Fshow_&data=02%7C01%7Cnsosnov%40microsoft.c
> om%7Ce0541136ceb6429d9d0f08d4a26c1926%7C72f988bf86f141af91ab2d7cd011db
> 47%7C1%7C0%7C636312032542968195&sdata=oNadqT%2B%2Fm0pj86k8Z854lVSWWHUu
> t60YOaYmiaZfPZk%3D&reserved=0
> bug.cgi?id=16454) and would like to submit a patch to the bug report 
> on Bugzilla. I'd also like to start going through some of the other 
> Windows-specific issues and start fixing those. The bug submission 
> instructions indicate that I should ask here for a Bugzilla account.
> Is that still the correct procedure?
> 
> Thanks!
> 
> Nathan
> 
> 
>   [[alternative HTML version deleted]]
> 
> __
> R-devel@r-project.org mailing list
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-devel&data=02%7C01%7Cnsosnov%40microsoft.com%7Ce0541136ceb6429d9d0f08d4a26c1926%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636312032542968195&sdata=nDdVG5QFWYItWdoKPq9XvYOGiwqq51ODtd53wuES5JY%3D&reserved=0
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] suggestion to fix packageDescription() for Windows users

[Nathan Sosnovske via R-devel]

Hi Duncan,

I'm guessing I'll be able to look at this over the weekend/next week (probably 
closer to next week). It is on my list of things to do and I've just had a few 
other prior commitments that I have to finish first.

Sorry for the delay. I'll chime in with a status update next week.

Nathan

-Original Message-
From: R-devel [mailto:r-devel-boun...@r-project.org] On Behalf Of Duncan Murdoch
Sent: Friday, June 23, 2017 5:16 AM
To: Andrie de Vries 
Cc: r-devel@r-project.org; Ben Marwick 
Subject: Re: [Rd] suggestion to fix packageDescription() for Windows users

On 18/06/2017 5:57 AM, Andrie de Vries wrote:
> Hi, Duncan
>
> i have forwarded this thread to Nathan, who promised to look into it.

Any progress on this?

Duncan Murdoch

>
> Andrie
>
> On 17 Jun 2017 17:26, "Duncan Murdoch"  > wrote:
>
> On 17/06/2017 9:13 AM, Ben Marwick wrote:
>
> Hi Duncan,
>
> Thanks for your reply. Yes, it does seem to be specific to the CTYPE
> setting to Chinese on Windows. If I set it to English using
> Sys.setlocale() there is no problem, then back to Chinese and the
> authors disappear:
>
> Sys.setlocale("LC_ALL","English")
> citation("readr")
>
>
> Thanks, that makes the problem reproducible.  I'll submit it as a
> bug report.  Maybe someone from Microsoft will fix it.
>
> Duncan Murdoch
>
>
> #' To cite package ‘readr’ in publications use:
> #'
> #'   Hadley Wickham, Jim Hester and Romain Francois (2017).
> readr: Read
> #' Rectangular Text Data. R package version 1.1.1.
> #' 
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2FCRAN.R-project.org%2Fpackage%3Dreadr&data=02%7C01%7Cnsosnov%40microsoft.com%7Ccf07101e770643227da008d4ba31aa85%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636338169899157582&sdata=SnbAtaLB%2BUtjGEZkTXwTqnaJtiF3jQXUDbzD3E5EWRM%3D&reserved=0
> 
> 
> #'
> #' A BibTeX entry for LaTeX users is
> #'
> #' @Manual{,
> #'   title = {readr: Read Rectangular Text Data},
> #'   author = {Hadley Wickham and Jim Hester and Romain Francois},
> #'   year = {2017},
> #'   note = {R package version 1.1.1},
> #'   url = 
> {https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2FCRAN.R-project.org%2Fpackage%3Dreadr&data=02%7C01%7Cnsosnov%40microsoft.com%7Ccf07101e770643227da008d4ba31aa85%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636338169899157582&sdata=SnbAtaLB%2BUtjGEZkTXwTqnaJtiF3jQXUDbzD3E5EWRM%3D&reserved=0
> 
> },
> #' }
>
>
> Sys.setlocale("LC_CTYPE", "Chinese")
> citation("readr")
>
> #'
> #' To cite package ‘readr’ in publications use:
> #'
> #'   (2017). readr: Read Rectangular Text Data. R package
> version 1.1.1.
> #' 
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2FCRAN.R-project.org%2Fpackage%3Dreadr&data=02%7C01%7Cnsosnov%40microsoft.com%7Ccf07101e770643227da008d4ba31aa85%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636338169899157582&sdata=SnbAtaLB%2BUtjGEZkTXwTqnaJtiF3jQXUDbzD3E5EWRM%3D&reserved=0
> 
> 
> #'
> #' A BibTeX entry for LaTeX users is
> #'
> #' @Manual{,
> #'   title = {readr: Read Rectangular Text Data},
> #'   year = {2017},
> #'   note = {R package version 1.1.1},
> #'   url = 
> {https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2FCRAN.R-project.org%2Fpackage%3Dreadr&data=02%7C01%7Cnsosnov%40microsoft.com%7Ccf07101e770643227da008d4ba31aa85%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636338169899157582&sdata=SnbAtaLB%2BUtjGEZkTXwTqnaJtiF3jQXUDbzD3E5EWRM%3D&reserved=0
> 
> 

Re: [Rd] suggestion to fix packageDescription() for Windows users

[Nathan Sosnovske via R-devel]

I'd be curious to know what others think of Rich's patch. If it is acceptable, 
I can spend time that I was going to look at it this week on another bug.

-Original Message-
From: Rich Calaway 
Sent: Friday, June 23, 2017 6:34 PM
To: Nathan Sosnovske ; Duncan Murdoch 
; Andrie de Vries 
Cc: Ben Marwick ; R-devel Mailing List (r-devel@r-project.org) 

Subject: RE: [Rd] suggestion to fix packageDescription() for Windows users

The following patch is not the most elegant, but it restores the Authors when 
"LC_CTYPE" is set to either "Chinese" or "Arabic":

> Sys.setlocale("LC_CTYPE", "Chinese")
[1] "Chinese (Simplified)_China.936"
> citation("readr")

To cite package ‘readr’ in publications use:

  (2016). readr: Read Tabular Data. R package version 1.0.0.
  https://CRAN.R-project.org/package=readr

A BibTeX entry for LaTeX users is

  @Manual{,
title = {readr: Read Tabular Data},
year = {2016},
note = {R package version 1.0.0},
url = {https://CRAN.R-project.org/package=readr},
  }

ATTENTION: This citation information has been auto-generated from the package 
DESCRIPTION file and may need manual editing, see ‘help("citation")’.

> Sys.setlocale("LC_CTYPE", "Arabic")
[1] "Arabic_Saudi Arabia.1256"
> citation("readr")

To cite package ‘readr’ in publications use:

  (2016). readr: Read Tabular Data. R package version 1.0.0.
  https://CRAN.R-project.org/package=readr

A BibTeX entry for LaTeX users is

  @Manual{,
title = {readr: Read Tabular Data},
year = {2016},
note = {R package version 1.0.0},
url = {https://CRAN.R-project.org/package=readr},
  }

ATTENTION: This citation information has been auto-generated from the package 
DESCRIPTION file and may need manual editing, see ‘help("citation")’.

> citation <- newCitation
> citation("readr")

To cite package ‘readr’ in publications use:

  Hadley Wickham, Jim Hester and Romain Francois (2016). readr: Read
  Tabular Data. R package version 1.0.0.
  https://CRAN.R-project.org/package=readr

A BibTeX entry for LaTeX users is

  @Manual{,
title = {readr: Read Tabular Data},
author = {Hadley Wickham and Jim Hester and Romain Francois},
year = {2016},
note = {R package version 1.0.0},
url = {https://CRAN.R-project.org/package=readr},
  }



The patch is:

Index: citation.R
===
--- citation.R  (revision 72852)
+++ citation.R  (working copy)
@@ -1162,8 +1162,11 @@
 if(dir == "")
 stop(gettextf("package %s not found", sQuote(package)),
  domain = NA)
-meta <- packageDescription(pkg = package,
-   lib.loc = dirname(dir))
+   args <- list(pkg = package, lib.loc = dirname(dir))
+   if (!is.na(enc <- packageDescription(pkg = package, 
lib.loc=dirname(dir), field="Encoding")))
+   args$enc <- enc
+meta <- do.call("packageDescription", args=args)
+
 ## if(is.null(auto)): Use default auto-citation if no CITATION
 ## available.
 citfile <- file.path(dir, "CITATION")


Nathan says he can look into this further next week...

Cheers,

Rich Calaway
Microsoft R Product Team
24/1341
+1 (425) 4219919 X19919

-Original Message-
From: R-devel [mailto:r-devel-boun...@r-project.org] On Behalf Of Nathan 
Sosnovske via R-devel
Sent: Friday, June 23, 2017 7:36 AM
To: Duncan Murdoch ; Andrie de Vries 

Cc: r-devel@r-project.org; Ben Marwick 
Subject: Re: [Rd] suggestion to fix packageDescription() for Windows users

Hi Duncan,

I'm guessing I'll be able to look at this over the weekend/next week (probably 
closer to next week). It is on my list of things to do and I've just had a few 
other prior commitments that I have to finish first.

Sorry for the delay. I'll chime in with a status update next week.

Nathan

-Original Message-
From: R-devel [mailto:r-devel-boun...@r-project.org] On Behalf Of Duncan Murdoch
Sent: Friday, June 23, 2017 5:16 AM
To: Andrie de Vries 
Cc: r-devel@r-project.org; Ben Marwick 
Subject: Re: [Rd] suggestion to fix packageDescription() for Windows users

On 18/06/2017 5:57 AM, Andrie de Vries wrote:
> Hi, Duncan
>
> i have forwarded this thread to Nathan, who promised to look into it.

Any progress on this?

Duncan Murdoch

>
> Andrie
>
> On 17 Jun 2017 17:26, "Duncan Murdoch"  <mailto:murdoch.dun...@gmail.com>> wrote:
>
> On 17/06/2017 9:13 AM, Ben Marwick wrote:
>
> Hi Duncan,
>
> Thanks for your reply. Yes, it does seem to be specific to the CTYPE
> setting to Chinese on Windows. If I set it to English using
> Sys.s

Re: [Rd] suggestion to fix packageDescription() for Windows users

[Nathan Sosnovske via R-devel]

> *) Of course I'd hoped you'd find why the underlying
> packageDescription() function is not "getting the right thing" in this case 
> directly -- in Windows only in some locales -- and provide a Windows-only 
> patch for the underlying problem there, rather than the workaround patch in 
> citation().
> The patch does solve the problem at hand, alright, so thank you, Rich and 
> Nathan!

This makes sense. I was asking if we wanted to proceed with the 
workaround-esque patch Rich suggested because I was trying to understand what 
was more useful to the community: diving deeper into the issue to see what was 
going on, or spending that time on another issue that was symptomatic. Either 
way, I think it makes sense to fix the underlying issue at some point and I do 
have time set aside this week to do so.

> so ideally almost all of this follow up should have happened there.

Apologies about this. I'm still learning and appreciate the gentle correction. 😊

-Original Message-
From: Martin Maechler [mailto:maech...@stat.math.ethz.ch] 
Sent: Tuesday, June 27, 2017 3:34 AM
To: Nathan Sosnovske 
Cc: Rich Calaway ; Duncan Murdoch 
; Andrie de Vries ; R-devel 
Mailing List (r-devel@r-project.org) ; Ben Marwick 
; Martin Maechler 
Subject: Re: [Rd] suggestion to fix packageDescription() for Windows users

>>>>> Nathan Sosnovske via R-devel 
>>>>> on Mon, 26 Jun 2017 18:22:25 + writes:

> I'd be curious to know what others think of Rich's
> patch. If it is acceptable, I can spend time that I was
> going to look at it this week on another bug. 

It is a bit kludgy (*) of course, but I confirm it solves the problem in a 
"robust" way.

*) Of course I'd hoped you'd find why the underlying
packageDescription() function is not "getting the right thing" in this case 
directly -- in Windows only in some locales -- and provide a Windows-only patch 
for the underlying problem there, rather than the workaround patch in 
citation().
The patch does solve the problem at hand, alright, so thank you, Rich and 
Nathan!

Note that Duncan Murdoch did mention in this thread to file an official bug 
report and Ben Marwick gave the URL

> From: Ben Marwick ...
> Subject: Re: [Rd] suggestion to fix packageDescription() for Windows 
> users
> Date: Sun, 18 Jun 2017 08:34:56 +1000

> Thanks very much, I see your bug report here:
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.
> r-project.org%2Fbugzilla3%2Fshow_bug.cgi%3Fid%3D17291&data=02%7C01%7Cn
> sosnov%40microsoft.com%7Ce3ed6ea675b44c908c9208d4bd4817dc%7C72f988bf86
> f141af91ab2d7cd011db47%7C1%7C0%7C636341564724082662&sdata=AFG8tyP5Maqc
> iZwYFqBZ4wylbVJAoyWu4kASsxFZr%2F4%3D&reserved=0

so ideally almost all of this follow up should have happened there.
I have followed up there and also there attached a Windows-only
+ commented version of Rich's patch. As mentioned, I've tested
it and confirmed to work for the use case in the mean time, so plan to commit 
soon.

This will be too late for the release of R 3.4.1 tomorrow, of course [code 
freeze was on June 23].

Martin Maechler
ETH Zurich

> -Original Message-
> From: Rich Calaway 
> Sent: Friday, June 23, 2017 6:34 PM
> To: Nathan Sosnovske ; Duncan Murdoch 
; Andrie de Vries 
> Cc: Ben Marwick ; R-devel Mailing List 
(r-devel@r-project.org) 
> Subject: RE: [Rd] suggestion to fix packageDescription() for Windows users

> The following patch is not the most elegant, but it restores the Authors 
when "LC_CTYPE" is set to either "Chinese" or "Arabic":

>> Sys.setlocale("LC_CTYPE", "Chinese")
> [1] "Chinese (Simplified)_China.936"
>> citation("readr")

> To cite package ‘readr’ in publications use:

> (2016). readr: Read Tabular Data. R package version 1.0.0.
> 
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2FCRAN.R-project.org%2Fpackage%3Dreadr&data=02%7C01%7Cnsosnov%40microsoft.com%7Ce3ed6ea675b44c908c9208d4bd4817dc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636341564724082662&sdata=C91aZ4qu2CT56qCrqFFBshREabZzfkaLXAABgWQSvXg%3D&reserved=0

> A BibTeX entry for LaTeX users is

> @Manual{,
> title = {readr: Read Tabular Data},
> year = {2016},
> note = {R package version 1.0.0},
> url = 
{https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2FCRAN.R-project.org%2Fpackage%3Dreadr&data=02%7C01%7Cnsosnov%40microsoft.com%7Ce3ed6ea675b44c908c9208d4bd4817dc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636341564724082662&sdata=C91aZ4qu2CT56qCrqFFBshREabZzfkaLXAABgWQSvXg%3D&reserved=0},
> }

> ATTENTION: This citation information has b

Re: [Rd] write.csv

[Nathan Sosnovske via R-devel]

This doesn't really strike me as a bug. Lots of (most?) programming languages 
expect you to handle this as an error condition. If you tried the same thing in 
C you would get the same error.

-Original Message-
From: R-devel [mailto:r-devel-boun...@r-project.org] On Behalf Of Lipatz 
Jean-Luc
Sent: Tuesday, July 4, 2017 5:40 AM
To: Duncan Murdoch 
Cc: r-devel@r-project.org
Subject: Re: [Rd] write.csv

I would really like the bug fixed. At least this one, because I know people in 
my institute using this function.
I understand your arguments about open source, but I also saw in this mail list 
a proposal for a fix for this bug for which there were no answer from the 
people who are able to include it in the distribution. It looks like if there 
were interesting bugs and the other ones.
I don't understand the other arguments : the example was reproduced with a 
simple USB key and you cannot state that a disk will eternally be empty enough, 
specially when it has several users.

JLL


-Message d'origine-
De : Duncan Murdoch [mailto:murdoch.dun...@gmail.com] Envoyé : mardi 4 juillet 
2017 14:24 À : Lipatz Jean-Luc; r-devel@r-project.org Objet : Re: [Rd] write.csv

On 04/07/2017 5:40 AM, Lipatz Jean-Luc wrote:
> Hi all,
>
> I am currently studying how to generalize the usage of R in my statistical 
> institute and I encountered a problem that I cannot declare on bugzilla 
> (cannot understand why).

Bugzilla was badly abused by spammers last year, so you need to have your 
account created manually by one of the admins to post there.  Write to me 
privately if you'd like me to create an account for you.  (If you want it 
attached to a different email address, that's fine.)

Sorry for trying this mailing list but I am really worried about the problem 
itself and the possible implications in using R in a professionnal data 
production context.
> The issue about 'write.csv' is that it just doesn't check if there is enough 
> space on disk and doesn't report failure to write data.
>
> Example (R 3.4.0 windows 32 bits, but I reproduced the problem with 
> older versions and under Mac OS/X)
>
>> fwrite(as.list(1:100),"G:/Test")
> Error in fwrite(as.list(1:1e+06), "G:/Test") :
>   No space left on device: 'G:/Test'
>> write.csv(1:100,"G:/Test")
>>
>
> I have a big concern here, because it means that you could save some 
> important data at one point of time and discover a long time after that you 
> actually lost them.
 > I suppose that the fix is relatively straightforward, but how can we be sure 
 > that there is no another function with the same bad properties?

R is open source.  You could work out the patch for this bug, and in the 
process see the pattern of coding that leads to it.  Then you'll know if other 
functions use the same buggy pattern.

> Is the lesson that you should not use a R function, even from the core, 
> without having personnally tested it against extreme conditions?

I think the answer to that is yes.  Most people never write such big files that 
they fill their disk:  if they did, all sorts of things would go wrong on their 
systems.  So this kind of extreme condition isn't often tested.  It's not easy 
to test in a platform independent way:  R would need to be able to create a 
volume with a small capacity.  That's a very system-dependent thing to do.

> And wouldn't it be the work of the developpers to do such elementary tests?

Again, R is open source.  You can and should contribute code (and therefore 
become one of the developers) if you are working in unusual conditions.

R states quite clearly in the welcome message every time it starts: "R is free 
software and comes with ABSOLUTELY NO WARRANTY."  This is essentially the same 
lack of warranty that you get with commercial software, though it's stated a 
lot more clearly.

Duncan Murdoch

__
R-devel@r-project.org mailing list
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-devel&data=02%7C01%7Cnsosnov%40microsoft.com%7C92c3e87c4ca1482e32f908d4c2d9dd57%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636347688364867350&sdata=7z5OJqLZDZ1zIvx8pP7KhQzNaQ%2FBrhZFKdUHeiFfke4%3D&reserved=0

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] write.csv

[Nathan Sosnovske via R-devel]

Ah, I misread the example code (went straight to the line where the error was 
raised for fwrite). Apologies Jean-Luc.

-Original Message-
From: Duncan Murdoch [mailto:murdoch.dun...@gmail.com] 
Sent: Tuesday, July 4, 2017 6:39 AM
To: Nathan Sosnovske ; Lipatz Jean-Luc 

Cc: r-devel@r-project.org
Subject: Re: [Rd] write.csv

On 04/07/2017 8:46 AM, Nathan Sosnovske wrote:
> This doesn't really strike me as a bug. Lots of (most?) programming languages 
> expect you to handle this as an error condition. If you tried the same thing 
> in C you would get the same error.

The bug is that there is no error signalled.  It looks as though the write 
succeeded, when it didn't.

Duncan Murdoch

> -Original Message-
> From: R-devel [mailto:r-devel-boun...@r-project.org] On Behalf Of 
> Lipatz Jean-Luc
> Sent: Tuesday, July 4, 2017 5:40 AM
> To: Duncan Murdoch 
> Cc: r-devel@r-project.org
> Subject: Re: [Rd] write.csv
>
> I would really like the bug fixed. At least this one, because I know people 
> in my institute using this function.
> I understand your arguments about open source, but I also saw in this mail 
> list a proposal for a fix for this bug for which there were no answer from 
> the people who are able to include it in the distribution. It looks like if 
> there were interesting bugs and the other ones.
> I don't understand the other arguments : the example was reproduced with a 
> simple USB key and you cannot state that a disk will eternally be empty 
> enough, specially when it has several users.
>
> JLL
>
>
> -Message d'origine-
> De : Duncan Murdoch [mailto:murdoch.dun...@gmail.com] Envoyé : mardi 4 
> juillet 2017 14:24 À : Lipatz Jean-Luc; r-devel@r-project.org Objet : 
> Re: [Rd] write.csv
>
> On 04/07/2017 5:40 AM, Lipatz Jean-Luc wrote:
>> Hi all,
>>
>> I am currently studying how to generalize the usage of R in my statistical 
>> institute and I encountered a problem that I cannot declare on bugzilla 
>> (cannot understand why).
>
> Bugzilla was badly abused by spammers last year, so you need to have 
> your account created manually by one of the admins to post there.  
> Write to me privately if you'd like me to create an account for you.  
> (If you want it attached to a different email address, that's fine.)
>
> Sorry for trying this mailing list but I am really worried about the problem 
> itself and the possible implications in using R in a professionnal data 
> production context.
>> The issue about 'write.csv' is that it just doesn't check if there is enough 
>> space on disk and doesn't report failure to write data.
>>
>> Example (R 3.4.0 windows 32 bits, but I reproduced the problem with 
>> older versions and under Mac OS/X)
>>
>>> fwrite(as.list(1:100),"G:/Test")
>> Error in fwrite(as.list(1:1e+06), "G:/Test") :
>>   No space left on device: 'G:/Test'
>>> write.csv(1:100,"G:/Test")
>>>
>>
>> I have a big concern here, because it means that you could save some 
>> important data at one point of time and discover a long time after that you 
>> actually lost them.
>  > I suppose that the fix is relatively straightforward, but how can we be 
> sure that there is no another function with the same bad properties?
>
> R is open source.  You could work out the patch for this bug, and in the 
> process see the pattern of coding that leads to it.  Then you'll know if 
> other functions use the same buggy pattern.
>
>> Is the lesson that you should not use a R function, even from the core, 
>> without having personnally tested it against extreme conditions?
>
> I think the answer to that is yes.  Most people never write such big files 
> that they fill their disk:  if they did, all sorts of things would go wrong 
> on their systems.  So this kind of extreme condition isn't often tested.  
> It's not easy to test in a platform independent way:  R would need to be able 
> to create a volume with a small capacity.  That's a very system-dependent 
> thing to do.
>
>> And wouldn't it be the work of the developpers to do such elementary tests?
>
> Again, R is open source.  You can and should contribute code (and therefore 
> become one of the developers) if you are working in unusual conditions.
>
> R states quite clearly in the welcome message every time it starts: "R is 
> free software and comes with ABSOLUTELY NO WARRANTY."  This is essentially 
> the same lack of warranty that you get with commercial software, though it's 
> stated a lot more clearly.
>
> Duncan Murdoch
>
> __
> R-devel@r-project.org mailing list
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.
> ethz.ch%2Fmailman%2Flistinfo%2Fr-devel&data=02%7C01%7Cnsosnov%40micros
> oft.com%7C92c3e87c4ca1482e32f908d4c2d9dd57%7C72f988bf86f141af91ab2d7cd
> 011db47%7C1%7C0%7C636347688364867350&sdata=7z5OJqLZDZ1zIvx8pP7KhQzNaQ%
> 2FBrhZFKdUHeiFfke4%3D&reserved=0
>

__
R-devel@r-project.org mailing list
https

Re: [Rd] write.csv

[Nathan Sosnovske via R-devel]

The best way to test on Windows would probably be creating a small virtual hard 
disk (via CreateVirtualDisk), mounting it, and writing to the mounted location. 
I believe the drive could even be mounted to an arbitrary location on the 
filesystem (instead of a drive letter) so that drive letter conflicts don't 
come into play.

-Original Message-
From: R-devel [mailto:r-devel-boun...@r-project.org] On Behalf Of Duncan Murdoch
Sent: Tuesday, July 4, 2017 7:53 AM
To: Jim Hester 
Cc: r-devel@r-project.org; Lipatz Jean-Luc 
Subject: Re: [Rd] write.csv

On 04/07/2017 10:01 AM, Jim Hester wrote:
> On linux at least you can use `/dev/full` [1] to test writing to a full 
> device.
>
> > echo 'foo' > /dev/full
> bash: echo: write error: No space left on device

Unfortunately, I get a permission denied error if I try to write there from 
MacOS.  I don't know if Windows has an equivalent.

I've taken a look at the code.  Essentially it comes down to a call to the C 
function vfprintf, which is supposed to return the number of bytes written, or 
a negative value for an error. This return value is often not checked; in 
particular, write.table and friends don't check it.

I'll add code to signal an error if there's a negative value.

I don't think it's feasible to check the number of bytes (formatted text with 
possible translation to a different encoding could have any number of bytes) if 
it's positive.  So hopefully all of our file systems will correctly signal an 
error, and not just report how many bytes were successfully written.

>
> Although that won't be a perfect test for this case where part of the 
> file is written successfully.
>
> An alternative suggestion for testing this is to create and mount a 
> loop device [2] with a small file.
>
> [1]: 
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wi
> kipedia.org%2Fwiki%2F%2Fdev%2Ffull&data=02%7C01%7Cnsosnov%40microsoft.
> com%7Cb97a7371538b4dbe9a7308d4c2ec5aa0%7C72f988bf86f141af91ab2d7cd011d
> b47%7C1%7C0%7C636347767773809248&sdata=Cb2oduozc2IDCLvXZGG1C4i4hQA7FPs
> 5jHmnFYbk7zQ%3D&reserved=0
> [2]: 
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstack
> overflow.com%2Fa%2F16044420%2F2055486&data=02%7C01%7Cnsosnov%40microso
> ft.com%7Cb97a7371538b4dbe9a7308d4c2ec5aa0%7C72f988bf86f141af91ab2d7cd0
> 11db47%7C1%7C0%7C636347767773809248&sdata=%2BWPfqD0nUS%2F30DUNDqQU79lR
> EJh02ZX0yik9HXiY5kg%3D&reserved=0

Loop devices sound ideal, but seem to be Linux-only (at least with that recipe).

Duncan


>
> On Tue, Jul 4, 2017 at 3:38 PM, Duncan Murdoch  
> wrote:
>> On 04/07/2017 8:40 AM, Lipatz Jean-Luc wrote:
>>>
>>> I would really like the bug fixed. At least this one, because I know 
>>> people in my institute using this function.
>>> I understand your arguments about open source, but I also saw in 
>>> this mail list a proposal for a fix for this bug for which there 
>>> were no answer from the people who are able to include it in the 
>>> distribution. It looks like if there were interesting bugs and the other 
>>> ones.
>>
>>
>> Please post a link to that, and I'll look.  Bug reports should be 
>> posted to the bug list.  It's unfortunate that it is currently so 
>> difficult to do so, but if they are only posted here, they are often 
>> overlooked.
>>
>>> I don't understand the other arguments : the example was reproduced 
>>> with a simple USB key and you cannot state that a disk will 
>>> eternally be empty enough, specially when it has several users.
>>
>>
>> I am not denying that it's a bug, I'm just saying that it is a 
>> difficult one to test automatically (so we probably won't add a 
>> regression test once it's fixed), and it's not one that has been 
>> reported often.  I didn't know there were any reports before yours.
>>
>> Duncan Murdoch
>>
>>
>>> JLL
>>>
>>>
>>> -Message d'origine-
>>> De : Duncan Murdoch [mailto:murdoch.dun...@gmail.com] Envoyé : mardi 
>>> 4 juillet 2017 14:24 À : Lipatz Jean-Luc; r-devel@r-project.org 
>>> Objet : Re: [Rd] write.csv
>>>
>>> On 04/07/2017 5:40 AM, Lipatz Jean-Luc wrote:

 Hi all,

 I am currently studying how to generalize the usage of R in my 
 statistical institute and I encountered a problem that I cannot 
 declare on bugzilla (cannot understand why).
>>>
>>>
>>> Bugzilla was badly abused by spammers last year, so you need to have 
>>> your account created manually by one of the admins to post there.  
>>> Write to me privately if you'd like me to create an account for you.  
>>> (If you want it attached to a different email address, that's fine.)
>>>
>>> Sorry for trying this mailing list but I am really worried about the 
>>> problem itself and the possible implications in using R in a 
>>> professionnal data production context.

 The issue about 'write.csv' is that it just doesn't check if there 
 is enough space on disk and doesn't report failure to write data.

 Example (R 3.4.0 windows 32 bits, 

Re: [Rd] write.csv

[Nathan Sosnovske via R-devel]

As a follow up to this, Martin Maechler suggested that I write a small script 
that allows this to be tested on Windows from R. The attached script 
demonstrates creating and mounting a very small (4 MiB) VHD to a path on the 
system (in this case c:/smallmount) and then calling write.csv with a large 
dataframe to the newly mounted path.

The method I attached has two limitations. 

1) It will only work if run as administrator. This is a limitation of the OS. 
2) It will only work on Windows 8.1/Server 2012R2 or higher, as it uses 
powershell commands that only exist on those operating systems. I believe it 
could be made to work on Windows 7, but it would need to be written using the 
windows C apis at that point.

If a regression test is created for this issue I would be more than happy to 
integrate this method into that if there is interest and if it would work in 
the environment where automated builds/tests for windows are run.

-Original Message-
From: Nathan Sosnovske 
Sent: Tuesday, July 4, 2017 8:02 AM
To: 'Duncan Murdoch' ; Jim Hester 

Cc: r-devel@r-project.org; Lipatz Jean-Luc 
Subject: RE: [Rd] write.csv

The best way to test on Windows would probably be creating a small virtual hard 
disk (via CreateVirtualDisk), mounting it, and writing to the mounted location. 
I believe the drive could even be mounted to an arbitrary location on the 
filesystem (instead of a drive letter) so that drive letter conflicts don't 
come into play.

-Original Message-
From: R-devel [mailto:r-devel-boun...@r-project.org] On Behalf Of Duncan Murdoch
Sent: Tuesday, July 4, 2017 7:53 AM
To: Jim Hester 
Cc: r-devel@r-project.org; Lipatz Jean-Luc 
Subject: Re: [Rd] write.csv

On 04/07/2017 10:01 AM, Jim Hester wrote:
> On linux at least you can use `/dev/full` [1] to test writing to a full 
> device.
>
> > echo 'foo' > /dev/full
> bash: echo: write error: No space left on device

Unfortunately, I get a permission denied error if I try to write there from 
MacOS.  I don't know if Windows has an equivalent.

I've taken a look at the code.  Essentially it comes down to a call to the C 
function vfprintf, which is supposed to return the number of bytes written, or 
a negative value for an error. This return value is often not checked; in 
particular, write.table and friends don't check it.

I'll add code to signal an error if there's a negative value.

I don't think it's feasible to check the number of bytes (formatted text with 
possible translation to a different encoding could have any number of bytes) if 
it's positive.  So hopefully all of our file systems will correctly signal an 
error, and not just report how many bytes were successfully written.

>
> Although that won't be a perfect test for this case where part of the 
> file is written successfully.
>
> An alternative suggestion for testing this is to create and mount a 
> loop device [2] with a small file.
>
> [1]: 
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wi
> kipedia.org%2Fwiki%2F%2Fdev%2Ffull&data=02%7C01%7Cnsosnov%40microsoft.
> com%7Cb97a7371538b4dbe9a7308d4c2ec5aa0%7C72f988bf86f141af91ab2d7cd011d
> b47%7C1%7C0%7C636347767773809248&sdata=Cb2oduozc2IDCLvXZGG1C4i4hQA7FPs
> 5jHmnFYbk7zQ%3D&reserved=0
> [2]: 
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstack
> overflow.com%2Fa%2F16044420%2F2055486&data=02%7C01%7Cnsosnov%40microso
> ft.com%7Cb97a7371538b4dbe9a7308d4c2ec5aa0%7C72f988bf86f141af91ab2d7cd0
> 11db47%7C1%7C0%7C636347767773809248&sdata=%2BWPfqD0nUS%2F30DUNDqQU79lR
> EJh02ZX0yik9HXiY5kg%3D&reserved=0

Loop devices sound ideal, but seem to be Linux-only (at least with that recipe).

Duncan


>
> On Tue, Jul 4, 2017 at 3:38 PM, Duncan Murdoch  
> wrote:
>> On 04/07/2017 8:40 AM, Lipatz Jean-Luc wrote:
>>>
>>> I would really like the bug fixed. At least this one, because I know 
>>> people in my institute using this function.
>>> I understand your arguments about open source, but I also saw in 
>>> this mail list a proposal for a fix for this bug for which there 
>>> were no answer from the people who are able to include it in the 
>>> distribution. It looks like if there were interesting bugs and the other 
>>> ones.
>>
>>
>> Please post a link to that, and I'll look.  Bug reports should be 
>> posted to the bug list.  It's unfortunate that it is currently so 
>> difficult to do so, but if they are only posted here, they are often 
>> overlooked.
>>
>>> I don't understand the other arguments : the example was reproduced 
>>> with a simple USB key and you cannot state that a disk will 
>>> eternally be empty enough, specially when it has several users.
>>
>>
>> I am not denying that it's a bug, I'm just saying that it is a 
>> difficult one to test automatically (so we probably won't add a 
>> regression test once it's fixed), and it's not one that has been 
>> reported often.  I didn't know there were any reports before yours.
>>
>> Duncan Murdoch
>>
>>
>>> JLL
>>>
>>>
>>> 

Re: [Rd] write.csv

[Nathan Sosnovske via R-devel]

Two thoughts:

1) Will the new server that you are setting up with Server 2016 eventually host 
build and test? If so, this could at least run on that.
2) CreateVirtualDisk and OpenVirtualDisk are C functions that are available in 
Windows 7 and Server 2008 R2. So these are options that we could use, but it 
would require creating a small program to drive creation/mounting/deletion of 
the disk and compiling it at build time.

Nathan

-Original Message-
From: Uwe Ligges [mailto:lig...@statistik.tu-dortmund.de] 
Sent: Tuesday, July 11, 2017 5:09 AM
To: Nathan Sosnovske ; Duncan Murdoch 
; Jim Hester 
Cc: r-devel@r-project.org; Lipatz Jean-Luc 
Subject: Re: [Rd] write.csv

This is a bit difficult:

R binaries (both for R base and R packages) are still built with Windows Server 
2008 and my desktop machine is Windows 7, hence at least currently such a check 
would not get executed on the machines R core / CRAN use ...

Best,
Uwe

On 11.07.2017 00:59, Nathan Sosnovske via R-devel wrote:
> As a follow up to this, Martin Maechler suggested that I write a small script 
> that allows this to be tested on Windows from R. The attached script 
> demonstrates creating and mounting a very small (4 MiB) VHD to a path on the 
> system (in this case c:/smallmount) and then calling write.csv with a large 
> dataframe to the newly mounted path.
> 
> The method I attached has two limitations.
> 
> 1) It will only work if run as administrator. This is a limitation of the OS.
> 2) It will only work on Windows 8.1/Server 2012R2 or higher, as it uses 
> powershell commands that only exist on those operating systems. I believe it 
> could be made to work on Windows 7, but it would need to be written using the 
> windows C apis at that point.
> 
> If a regression test is created for this issue I would be more than happy to 
> integrate this method into that if there is interest and if it would work in 
> the environment where automated builds/tests for windows are run.
> 
> -Original Message-
> From: Nathan Sosnovske
> Sent: Tuesday, July 4, 2017 8:02 AM
> To: 'Duncan Murdoch' ; Jim Hester 
> 
> Cc: r-devel@r-project.org; Lipatz Jean-Luc 
> Subject: RE: [Rd] write.csv
> 
> The best way to test on Windows would probably be creating a small virtual 
> hard disk (via CreateVirtualDisk), mounting it, and writing to the mounted 
> location. I believe the drive could even be mounted to an arbitrary location 
> on the filesystem (instead of a drive letter) so that drive letter conflicts 
> don't come into play.
> 
> -Original Message-
> From: R-devel [mailto:r-devel-boun...@r-project.org] On Behalf Of 
> Duncan Murdoch
> Sent: Tuesday, July 4, 2017 7:53 AM
> To: Jim Hester 
> Cc: r-devel@r-project.org; Lipatz Jean-Luc 
> Subject: Re: [Rd] write.csv
> 
> On 04/07/2017 10:01 AM, Jim Hester wrote:
>> On linux at least you can use `/dev/full` [1] to test writing to a full 
>> device.
>>
>>  > echo 'foo' > /dev/full
>>  bash: echo: write error: No space left on device
> 
> Unfortunately, I get a permission denied error if I try to write there from 
> MacOS.  I don't know if Windows has an equivalent.
> 
> I've taken a look at the code.  Essentially it comes down to a call to the C 
> function vfprintf, which is supposed to return the number of bytes written, 
> or a negative value for an error. This return value is often not checked; in 
> particular, write.table and friends don't check it.
> 
> I'll add code to signal an error if there's a negative value.
> 
> I don't think it's feasible to check the number of bytes (formatted text with 
> possible translation to a different encoding could have any number of bytes) 
> if it's positive.  So hopefully all of our file systems will correctly signal 
> an error, and not just report how many bytes were successfully written.
> 
>>
>> Although that won't be a perfect test for this case where part of the 
>> file is written successfully.
>>
>> An alternative suggestion for testing this is to create and mount a 
>> loop device [2] with a small file.
>>
>> [1]:
>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.w
>> i 
>> kipedia.org%2Fwiki%2F%2Fdev%2Ffull&data=02%7C01%7Cnsosnov%40microsoft.
>> com%7Cb97a7371538b4dbe9a7308d4c2ec5aa0%7C72f988bf86f141af91ab2d7cd011
>> d 
>> b47%7C1%7C0%7C636347767773809248&sdata=Cb2oduozc2IDCLvXZGG1C4i4hQA7FP
>> s
>> 5jHmnFYbk7zQ%3D&reserved=0
>> [2]:
>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstac
>> k 
>> overflow.com%2Fa%2F16044420%2F2055486&data=02%7C01%7Cnsosnov%40micros
>&g