perfmon under windows will give you the amount of I/O that a process
is doing. You can see if the number if I/O Reads and I/O Writes is
increasing for your process. I don't know what the internals of the
functions you are calling are doing w.r.t. how the files are accessed.
On Thu, Oct 7, 2010 at 1:18 PM, Bastien Ferland-Raymond
<bastien.ferland-raymon...@ulaval.ca> wrote:
> Actually, I rechecked the paging and thought about it more, and realized the
> windows was French and therefore, the "," was the decimal separator and not
> the "thousand" separator. So the paging was staying around 6 (not 6000) most
> of the time and hitting the 100 other times. It looks a little bit more
> normal.
>
> About the I/O. I'm really not familiar with this concept. I really want to
> answer your questions and test all of that, but I don't know how. Can you
> check the I/O in R, or are you suppose to do it in Windows? What
> function/program should I use?
>
>
> Thanks
> Bastien
>
>
> ________________________________________
> De : jim holtman [jholt...@gmail.com]
> Date d'envoi : 6 octobre 2010 20:09
> À : Bastien Ferland-Raymond
> Cc : r-help@R-project.org
> Objet : Re: RE : [R] R getting slower until it breaks...
>
> 6000 pages/sec sounds very high, but if your CPU utilization is
> decreasing over time, this is one of the causes. The system is paging
> memory out and having to wait for I/O to complete and therefore is not
> using the CPU. What other I/O is your system doing? When you are
> partitioning the image, are you doing a lot of I/O to write it out?
> So I would assume that you are somehow blocking on I/O and that is why
> your utilization (and throughput) is decreasing. You might want to
> look at what your application is doing in this area. Paging is also
> contributed to reading in some data.
>
> When are are breaking apart the data, are you starting the read from
> the front of the file each time, or picking up where you left off? So
> you might want to take a closer look around how you are doing your
> I/O.
>
> On Wed, Oct 6, 2010 at 3:06 PM, Bastien Ferland-Raymond
> <bastien.ferland-raymon...@ulaval.ca> wrote:
>> Thanks a lot for your quick answer. Here is my answer to your questions:
>>
>> Have you looked to see how fast your memory might be growing?
>> BFR- Yes I did, it's not to bad, it's starts around 60 000ko, rise up to 120
>> 000 at the most, so not too scary.
>>
>> Are you leaving around any large objects that should be removed?
>> BFR- I was carefull making sure the function doesn't create anything that
>> would be visible with objects(). Could it be creating other type (hidden)
>> objects? Maybe, but I'm not very familliar with that stuff.
>>
>> Have you looked to see if you are paging?
>> BFR- I just red the wiki about paging, didn't know that term before. If I
>> look at perfmon, its looks like keeping steady at 6000 pages/s with rare
>> peaks as high as 900 000. Does it sounds normal? How can it affect R?
>>
>> Is it your CPU time that is increasing, or your wall clock time?
>> BFR- If I go to the task manager - performance. R is initially using around
>> 40% of the processor (so around 80% of 1 core) but with (real) time passing,
>> it gets lower and lower to get as low as 6% (12% of one core). I was
>> surprized to see that as usually my simulation in R use one whole core.
>>
>> It sounds like there might be some memory leak that might be causing your
>> process size to grow and possibly causing paging. You will need to gather
>> some of the performance data that perfmon can provide and look at the memory
>> usage, CPU time and I/O rates over time to see if there are any changes.
>> BFR- The term "Memory leak" feels right with my problem. Is there ways I
>> can control/detect/prevent this kind of problem in R. Also, how can I check
>> the I/O, i never looked at that before.
>>
>> Thanks again
>>
>> Bastien
>>
>>
>>
>> On Wed, Oct 6, 2010 at 2:11 PM, Bastien Ferland-Raymond
>> <bastien.ferland-raymon...@ulaval.ca> wrote:
>>> Hello R-users,
>>>
>>> I'm currently facing a pretty hard problem which I'm hopping you'll be able
>>> to help me with. I'm using R to create images. That alone is not the
>>> problem, the problem is that I'm using R to create 168 000 images... My
>>> code (which is given below) use different package (raster and rgdal) to
>>> import a image (size 20gig) and divide it into 168 000 pictures that are
>>> 100 pixel x 100 pixel. The code works fine for making the images, but if I
>>> ask it to run all 168 000, it always breaks around 15 000.
>>>
>>> It starts with the code being able to make around 2 pictures per second,
>>> but then it slows down and after around 2000 pictures it's only 1 picture
>>> per second. Later on it's getting closer to 1 pictures every 3 seconds
>>> etc. until it bugs. I have no error message, only Windows that tells me
>>> that "R encounter a problem and most be close..." Initially I though it
>>> was a Windows problem, that I couldn't put too many file into a folder and
>>> it was slowering it down. Then I divided my batch process into smaller
>>> (5000 files) folder but it didn't help, still breaks at 15 000. I also try
>>> to do gc() after each 5000 pictures to save memory but it didn't help
>>> either. I removed every loops from the code because I thought it was the
>>> problem, but it was just faster at bugging... After the bug, I need to
>>> restart the computer if I want to go back to the initial speed.
>>>
>>> I'm pretty much running out of options. It's there limitation in R as the
>>> number of files it can create in one session? Is it a windows problem? Is
>>> there better way to clear the memory than gc()? Any thought on that?
>>>
>>> I'm using R 2.11.1, win XP, my hard drive is NTSF, computer: intel core2
>>> duo E6750 32 bit with 2 gig of Ram.
>>>
>>> Here is my code, but I doubt it would help much with my problem:
>>>
>>> ########
>>> # It made of 4 functions (sorry, it's french):
>>>
>>> ##########################################################################
>>> ##########################################################################
>>> ### Ensemble des fonctions pour faire les images NDVI rouge et verte ###
>>> ##########################################################################
>>> ###### Bastien Ferland-Raymond, 5 oct 2010 #############################
>>> ##########################################################################
>>>
>>> ########
>>> ## Simplement rouler le script au complet
>>> ########
>>> ### Library nécessaire:
>>> library(raster)
>>> library(rgdal)
>>> library(shapefiles)
>>>
>>> #############################################################################
>>> ## Fonction 1 - NDVI a partir de coordonnee Pixel et largeur #####
>>> calculate_NDVI<- function(Type, object, VALUE) {
>>> redorgreen <- ifelse(Type=="red",2,3)
>>> list1 <- unstack(object)
>>> rast1 <- list1[[1]]
>>> rast2 <- list1[[redorgreen]]
>>> NAvalue(rast1)<- -99999
>>> NAvalue(rast2)<- -99999
>>> cells1 <-
>>> getValuesBlock(rast1,row=VALUE[[2]],nrow=VALUE[[3]],col=VALUE[[1]],ncol=VALUE[[3]])
>>> cells2 <-
>>> getValuesBlock(rast2,row=VALUE[[2]],nrow=VALUE[[3]],col=VALUE[[1]],ncol=VALUE[[3]])
>>> cells1[is.na(cells1)]<-0;
>>> cells2[is.na(cells2)]<-0;
>>> calculNDVI <-(cells1 - cells2) / (cells1 + cells2)
>>> NDVImatrix <- matrix(calculNDVI,nrow=VALUE[[3]],ncol=VALUE[[3]],
>>> byrow=TRUE)
>>> NDVImatrix <- NDVImatrix + 1
>>> NDVImatrix <- NDVImatrix * (255/2)
>>> return(NDVImatrix)
>>> }
>>> #################################################################################
>>> ## Fonction 1b - Faire le tiff
>>> make.tiff<- function(NV=newValues,TT=Type,img=imgRaster,nom){
>>> pixelNDVIMatrix <- calculate_NDVI(TT,img,NV[c(1,2,3)])
>>> newRaster <- raster(pixelNDVIMatrix)
>>> NAvalue(newRaster)<-999999
>>> nnom<-nom[NV[4]]
>>> writeRaster(newRaster,
>>> filename=nnom,datatype="INT1U",format="GTiff",overwrite=FALSE)
>>> aaa<-2
>>> }
>>>
>>> #################################################################################
>>> ## Fonction 2 - Creation de fonction convertissant les coordonnee
>>> metrique en coordonnee pixels #####
>>> latlong_to_pixels<- function(Coord, facteur, meterWidth=NULL) { #Coord
>>> doit être c(x,y)
>>> newX <- Coord[1] / facteur
>>> newY <- Coord[2] / facteur
>>> if(!is.null(meterWidth)){
>>> newWidth <- meterWidth / facteur
>>> return(c(newX,newY,newWidth))
>>> }
>>> return(c(newX,newY))
>>> }
>>>
>>> #############################################################################
>>> #### Fonction 3 - Fonction principal #####
>>> make.NDVI.photo<-
>>> function(tableDesPlacettes,Type,newImagesDirectory,textAndImgDirectory="U:\\kNN_Valcartier\\Photo
>>> aerienne"){
>>> lastWD<- getwd()
>>> setwd(textAndImgDirectory)
>>> imgRaster<- stack(imageAssociee)
>>> x1<- tableDesPlacettes[,2] - xmin(imgRaster) - (tailleFenetres/2) #
>>> The image origin for calculation is in the top left corner
>>> y1<- ymax(imgRaster) - tableDesPlacettes[,3] - (tailleFenetres/2)
>>> coo <- cbind(x1,y1)
>>> newValues<-
>>> t(apply(coo,1,latlong_to_pixels,facteurMetreParPixel,tailleFenetres))
>>> newImgName<-
>>> paste(newImagesDirectory,substr(Type,1,3),"_","GC",tableDesPlacettes[,1],".tif",sep="")
>>> apply(cbind(newValues,1:length(newImgName)),1,make.tiff,Type,imgRaster,nom=newImgName)
>>> setwd(lastWD)
>>> }
>>>
>>>
>>> ###########################
>>> ## Executing fonctions:
>>> #############################
>>>
>>>
>>> ### loader les données brutes de fenêtres
>>> fichier.fenetre.brute<-read.dbf("U:\\kNN_Valcartier\\Fenêtres 30x30 1
>>> octobre\\168700_30m_centroid.dbf", header=T)
>>> ### Sélectionner les fenêtres complètes
>>> fenetre.complete<-round(fichier.fenetre.brute[[1]][,2],1)==900
>>> ### Sortir les centroides pour extraction
>>> centro.tout.900<-fichier.fenetre.brute[[1]][fenetre.complete,c(1,5,6)]
>>> #rm(fichier.fenetre.brute) ; gc()
>>>
>>> ## données nécessaires pour la fonction
>>> imageAssociee<- "mosaique_all_v1.img" # nom de l'image
>>> facteurMetreParPixel<- 0.3 # combien de metre vaut un pixel
>>> tailleFenetres= 30 # en metre
>>>
>>> start.time<-Sys.time();start.time
>>> make.NDVI.photo(centro.tout.900[19137:24136,],"red","BFR\\NDVI_red_fenetre\\batch
>>> 3\\")
>>> stop.time<-Sys.time()
>>> time.run<-stop.time-start.time
>>> alarm()
>>> time.run
>>> gc()
>>> start.time<-Sys.time();start.time
>>> make.NDVI.photo(centro.tout.900[24137:29136,],"red","BFR\\NDVI_red_fenetre\\batch
>>> 4\\")
>>> stop.time<-Sys.time()
>>> time.run<-stop.time-start.time
>>> alarm()
>>> time.run
>>> gc()
>>>
>>> #############
>>>
>>>
>>> Voilà,
>>>
>>> Thanks!
>>>
>>> Bastien Ferland-Raymond
>>> ______________________________________________
>>> R-help@r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>>> and provide commented, minimal, self-contained, reproducible code.
>>>
>>
>>
>>
>> --
>> Jim Holtman
>> Cincinnati, OH
>> +1 513 646 9390
>>
>> What is the problem that you are trying to solve?
>
>
>
> --
> Jim Holtman
> Cincinnati, OH
> +1 513 646 9390
>
> What is the problem that you are trying to solve?
>
--
Jim Holtman
Cincinnati, OH
+1 513 646 9390
What is the problem that you are trying to solve?
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
