Re: [gdal-dev] does gdal support multiple simultaneous writers to raster

2013-01-14 Thread Tim Keitt
On Mon, Jan 14, 2013 at 2:47 AM, Jukka Rahkonen wrote: > Kennedy, Paul fugro.com.au> writes: > > > > > > > Hi > > Simultaneous writers would be a better long term solution as we often > improve > the raster following initial creation. This improvement may well be a > filter run > on a sub region

Re: [gdal-dev] does gdal support multiple simultaneous writers to raster

2013-01-14 Thread Jan Hartmann
Thanks Frank, that's what I have been doing until now. I was hoping that GDAL could do some of the bookkeeping for me. Bu the multithreaded issue remains important. Anything that could speed up the warping process, for whatever configuration, would be welcome and I would be happy to assist.

Re: [gdal-dev] does gdal support multiple simultaneous writers to raster

2013-01-14 Thread Frank Warmerdam
Jan, While it is desirable to resolve the multithreaded write issue in gdal I don't think it is critical for your use case. I think your case would be handled more appropriately by multiple *processes*. You would still run into issues with write consistency but of a different nature. The easies

Re: [gdal-dev] does gdal support multiple simultaneous writers to raster

2013-01-14 Thread Even Rouault
Selon "N. Farah" : > Thanks Even for your response. That clarifies more the current capabilities > of GDAL on parallel processing.I think that as parallel processing gain more > momentum, GDAL and multiple GDAL based applications would benefit for > adopting it.In Windows world, parallel processin

Re: [gdal-dev] does gdal support multiple simultaneous writers to raster

2013-01-14 Thread N. Farah
(and may be reading), I can help at least by testing. ThanksNoureddine Farah > From: even.roua...@mines-paris.org > To: gdal-dev@lists.osgeo.org > Subject: Re: [gdal-dev] does gdal support multiple simultaneous writers to > raster > Date: Sat, 12 Jan 2013 16:35:28 +0100 > CC:

Re: [gdal-dev] does gdal support multiple simultaneous writers to raster

2013-01-14 Thread Jukka Rahkonen
Kennedy, Paul fugro.com.au> writes: > > > Hi > Simultaneous writers would be a better long term solution as we often improve the raster following initial creation. This improvement may well be a filter run on a sub region ( eg a despeckle) or updating a piece of the dtm with better information

Re: [gdal-dev] does gdal support multiple simultaneous writers to raster

2013-01-12 Thread Kennedy, Paul
Hi Simultaneous writers would be a better long term solution as we often improve the raster following initial creation. This improvement may well be a filter run on a sub region ( eg a despeckle) or updating a piece of the dtm with better information or even some manual edits as a last resort.

Re: [gdal-dev] does gdal support multiple simultaneous writers to raster

2013-01-12 Thread Jan Hartmann
ot;Even Rouault" mailto:even.roua...@mines-paris.org>> wrote: Re: [gdal-dev] does gdal support multiple simultaneous writers to raster Hi, This is an intersting topic, with many "intersecting" issues to deal with at different levels. First, are you confident that in the use cas

Re: [gdal-dev] does gdal support multiple simultaneous writers to raster

2013-01-12 Thread Jan Hartmann
I would be interested in an implementation. I'm preparing a proposal to georeference the complete cadastral map of the Netherlands in 1832 at a 10 cm/pixel scale with Cloud facilities. Gdalwarp is the central piece of software, and distributed processing capabilities would be very important. Co

Re: [gdal-dev] does gdal support multiple simultaneous writers to raster

2013-01-12 Thread Even Rouault
> ex. convert > multiple datasets to different output datasets in a parallel way. As Frank underlined, there's currently an issue with the global block cache regarding write support. Imagine that you have 2 threads A and B. Thread A deal with dataset A, and thread B deal with dataset . Thread A

Re: [gdal-dev] does gdal support multiple simultaneous writers to raster

2013-01-12 Thread Even Rouault
Le samedi 12 janvier 2013 02:38:55, Kennedy, Paul a écrit : > Hi, > Yes, we are pretty sure we will see a significant benefit. The processing > algorithms are CPU bound not io bound. Our digital terrain model > interpolations often run for many hours ( we do them overnight) but the > underlying fi

Re: [gdal-dev] does gdal support multiple simultaneous writers to raster

2013-01-12 Thread Even Rouault
; > > > I believe slower cores but more of them in pcs are the future. My pc > > has 8 but they rarely get used to their potential. > > > > I am certain there are some challenges here, that's why it is > > interesting;) > > > > Regards >

Re: [gdal-dev] does gdal support multiple simultaneous writers to raster

2013-01-12 Thread Paul Meems
To add my 2cnts. With MapWindow GIS we use TauDEM binaries to perform watershed delineations. http://hydrology.usu.edu/taudem/taudem5.0/index.html These TauDEM binaries are optimized to use MPI, but also work if you don't have MPI installed. I don't know in detail how it works but in general you

Re: [gdal-dev] does gdal support multiple simultaneous writers to raster

2013-01-12 Thread Jan Hartmann
some challenges here, that's why it is interesting;) Regards pk On 11/01/2013, at 6:54 PM, "Even Rouault" mailto:even.roua...@mines-paris.org>> wrote: Re: [gdal-dev] does gdal support multiple simultaneous writers to raster Hi, This is an intersting topic, with many &qu

Re: [gdal-dev] does gdal support multiple simultaneous writers to raster

2013-01-11 Thread Joaquim Luis
On 12-01-2013 01:38, Kennedy, Paul wrote: Hi, Yes, we are pretty sure we will see a significant benefit. The processing algorithms are CPU bound not io bound. Our digital terrain model interpolations often run for many hours ( we do them overnight) but the underlying file is only a few gigaby

Re: [gdal-dev] does gdal support multiple simultaneous writers to raster

2013-01-11 Thread Kennedy, Paul
Hi, Yes, we are pretty sure we will see a significant benefit. The processing algorithms are CPU bound not io bound. Our digital terrain model interpolations often run for many hours ( we do them overnight) but the underlying file is only a few gigabytes. If we split them into multiple files of

[gdal-dev] does gdal support multiple simultaneous writers to raster

2013-01-11 Thread N. Farah
Simultaneous writers to different dataset or the same dataset ?In related topic, can GDAL writte to different dataset in a parallel way ?ex. convert multiple datasets to different output datasets in a parallel way.Would those parallel operations not be affected by GDAL caching for bot read and

Re: [gdal-dev] does gdal support multiple simultaneous writers to raster

2013-01-11 Thread Jan Hartmann
I would like to make a further suggestion: what about an option for gdalwarp to create *external* tiles, instead of one big output file? That's what I am doing now: just create one big georeferenced raster, and split it up in tiles, mostly 2000*2000. Accessing those tiles with a tileindex is re

Re: [gdal-dev] does gdal support multiple simultaneous writers to raster

2013-01-11 Thread Even Rouault
Hi, This is an intersting topic, with many "intersecting" issues to deal with at different levels. First, are you confident that in the use cases you imagine that I/O access won't be the limiting factor, in which case serialization of I/O could be acceptable and this would just require an API wit

Re: [gdal-dev] does gdal support multiple simultaneous writers to raster

2013-01-11 Thread Kennedy, Paul
dev Subject: Re: [gdal-dev] does gdal support multiple simultaneous writers to raster I'v tried this in a Cloud environment with multiple VMs writing to the same shared file location. I didn't use tiles, so the results where negative. It would be a great asset for Cloud environments, h

Re: [gdal-dev] does gdal support multiple simultaneous writers to raster

2013-01-11 Thread Jan Hartmann
I'v tried this in a Cloud environment with multiple VMs writing to the same shared file location. I didn't use tiles, so the results where negative. It would be a great asset for Cloud environments, however. Is it in principle possible to build such a facility into GDAL, and if yes, what kind o

Re: [gdal-dev] does gdal support multiple simultaneous writers to raster

2013-01-10 Thread Kennedy, Paul
.@gmail.com] On Behalf Of Frank Warmerdam Sent: Friday, 11 January 2013 9:33 AM To: Kennedy, Paul Cc: gdal dev Subject: Re: [gdal-dev] does gdal support multiple simultaneous writers to raster Paul, Generally speaking - no - GDAL does not now support multi-threaded writing. Part of this is due t

Re: [gdal-dev] does gdal support multiple simultaneous writers to raster

2013-01-10 Thread Frank Warmerdam
Paul, Generally speaking - no - GDAL does not now support multi-threaded writing. Part of this is due to the way stuff gets write-flushed out of the block cache and partly it is just that none of GDAL supports multiple threading on a single dataset. This statement is based on the assumption it i

[gdal-dev] does gdal support multiple simultaneous writers to raster

2013-01-10 Thread Kennedy, Paul
I am looking to parallelize a long duration gridding process, and was wondering if GDAL can support multiple simultaneous writers from processess? I am not sure if any of the underlying raster formats support this? thanks in advance. regards pk __