Hello,
This is far from a complete answer.
A quicky one: no loops.
mc_list2 <- grep(srchStr1, lines)
tmp_list2 <- grep(srchStr2, lines)
identical(mc_list, mc_list2) # [1] TRUE
identical(tmp_list, tmp_list2) # [1] TRUE
Another one: don't extend lists or vectors inside loops, reserve memory
beforehand.
wc <- vector("list", length = length(mc_list))
tmp <- vector("list", length = length(tmp_list))
are much better than your
wc <- list()
tmp <- list()
Maybe I will find ways to save time with the really slow instructions.
Hope this helps,
Rui Barradas
Às 19:54 de 24/07/19, Morway, Eric via R-help escreveu:
The small reproducible example below works, but is way too slow on the real
problem. The real problem is attempting to extract ~2920 repeated arrays
from a 60 Mb file and takes ~80 minutes. I'm wondering how I might
re-engineer the script to avoid opening and closing the file 2920 times as
is the case now. That is, is there a way to keep the file open and peel
out the arrays and stuff them into a list of data.tables, as is done in the
small reproducible example below, but in a significantly faster way?
wha <- " INITIAL PRESSURE HEAD
INITIAL TEMPERATURE SET TO 4.000E+00 DEGREES C
VS2DH - MedSand for TL test
TOTAL ELAPSED TIME = 0.000000E+00 sec
TIME STEP 0
MOISTURE CONTENT
Z, IN
m X OR R DISTANCE, IN m
0.500
0.075 0.1475
0.225 0.1475
0.375 0.1475
0.525 0.1475
0.675 0.1475
blah
blah
blah
TEMPERATURE, IN DECREES C
Z, IN
m X OR R DISTANCE, IN m
0.500
0.075 1.1475
0.225 2.1475
0.375 3.1475
0.525 4.1475
0.675 5.1475
blah
blah
blah
TOTAL ELAPSED TIME = 8.6400E+04 sec
TIME STEP 0
MOISTURE CONTENT
Z, IN
m X OR R DISTANCE, IN m
0.500
0.075 0.1875
0.225 0.1775
0.375 0.1575
0.525 0.1675
0.675 0.1475
blah
blah
blah TEMPERATURE, IN DECREES C
Z, IN
m X OR R DISTANCE, IN m
0.500
0.075 1.1475
0.225 2.1475
0.375 3.1475
0.525 4.1475
0.675 5.1475
blah
blah
blah"
example_content <- textConnection(wha)
srchStr1 <- ' MOISTURE CONTENT'
srchStr2 <- 'TEMPERATURE, IN DECREES C'
lines <- readLines(example_content)
mc_list <- NULL
for (i in 1:length(lines)){
# Look for start of water content
if(grepl(srchStr1, lines[i])){
mc_list <- c(mc_list, i)
}
}
tmp_list <- NULL
for (i in 1:length(lines)){
# Look for start of temperature data
if(grepl(srchStr2, lines[i])){
tmp_list <- c(tmp_list, i)
}
}
# Store the water content arrays
wc <- list()
# Read all the moisture content profiles
for(i in 1:length(mc_list)){
lineNum <- mc_list[i] + 3
mct <- read.table(text = wha, skip=lineNum, nrows=5,
col.names=c('depth','wc'))
wc[[i]] <- mct
}
# Store the water temperature arrays
tmp <- list()
# Read all the temperature profiles
for(i in 1:length(tmp_list)){
lineNum <- tmp_list[i] + 3
tmpt <- read.table(text = wha, skip=lineNum, nrows=5,
col.names=c('depth','tmp'))
tmp[[i]] <- tmpt
}
# quick inspection
length(wc)
wc[[1]]
# Looks like what I'm after, but too slow in real world problem
[[alternative HTML version deleted]]
______________________________________________
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
______________________________________________
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
