On 2018-04-18 17:38, Steven McKinney wrote:
Hi Colton,
You could divide your write task into chunks that do not violate the 2^31-1
limit.
write.table has an append argument (default FALSE).
Figure out a row chunk size nri < nr such that nri * nc is under 2^31-1 and use
write.table() to write that out.
Then use
write.table( append = TRUE, )
for the next chunk of rows, looping over chunks until done. Two chunks will
get your 2.8 billion entries done.
Magnificent: Is that something that could be implemented inside
write.table?
Spencer
Best
Steve
Steven McKinney, Ph.D.
Statistician
Molecular Oncology and Breast Cancer Program
British Columbia Cancer Research Centre
-----Original Message-----
From: R-devel [mailto:r-devel-boun...@r-project.org] On Behalf Of Tousey,
Colton
Sent: April-18-18 2:08 PM
To: r-c...@r-project.org; simon.urba...@r-project.org; R-devel@r-
project.org
Subject: [Rd] R Bug: write.table for matrix of more than 2, 147, 483, 648
elements
Hello,
I want to report a bug in R that is limiting my capabilities to export a
matrix with write.csv or write.table with over 2,147,483,648 elements (C's
int limit). I found this bug already reported about before: https://bugs.r-
project.org/bugzilla/show_bug.cgi?id=17182. However, there appears to be no
solution or fixes in upcoming R version releases.
The error message is coming from the writetable part of the utils package
in the io.c source code(https://svn.r-
project.org/R/trunk/src/library/utils/src/io.c):
/* quick integrity check */
if(XLENGTH(x) != (R_len_t)nr * nc)
error(_("corrupt matrix -- dims not not match
length"));
The issue is that nr*nc is an integer and the size of my matrix, 2.8
billion elements, exceeds C's limit, so the check forces the code to fail.
My version:
R.Version()
$platform
[1] "x86_64-w64-mingw32"
$arch
[1] "x86_64"
$os
[1] "mingw32"
$system
[1] "x86_64, mingw32"
$status
[1] ""
$major
[1] "3"
$minor
[1] "4.3"
$year
[1] "2017"
$month
[1] "11"
$day
[1] "30"
$`svn rev`
[1] "73796"
$language
[1] "R"
$version.string
[1] "R version 3.4.3 (2017-11-30)"
$nickname
[1] "Kite-Eating Tree"
Thank you,
Colton
Colton Tousey
Research Associate II
P: 816.585.0300 E: colton.tou...@kc.frb.org
FEDERAL RESERVE BANK OF KANSAS CITY
1 Memorial Drive * Kansas City, Missouri 64198 *
www.kansascityfed.org
[[alternative HTML version deleted]]
______________________________________________
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
