On Mon, Jan 14, 2013 at 2:47 AM, Jukka Rahkonen
wrote:
> Kennedy, Paul fugro.com.au> writes:
>
> >
> >
> > Hi
> > Simultaneous writers would be a better long term solution as we often
> improve
> the raster following initial creation. This improvement may well be a
> filter run
> on a sub region
Thanks Frank, that's what I have been doing until now. I was hoping that
GDAL could do some of the bookkeeping for me.
Bu the multithreaded issue remains important. Anything that could speed
up the warping process, for whatever configuration, would be welcome and
I would be happy to assist.
Jan,
While it is desirable to resolve the multithreaded write issue in gdal I
don't think it is critical for your use case.
I think your case would be handled more appropriately by multiple
*processes*. You would still run into issues with write consistency but of
a different nature. The easies
Selon "N. Farah" :
> Thanks Even for your response. That clarifies more the current capabilities
> of GDAL on parallel processing.I think that as parallel processing gain more
> momentum, GDAL and multiple GDAL based applications would benefit for
> adopting it.In Windows world, parallel processin
(and may be reading), I can
help at least by testing.
ThanksNoureddine Farah
> From: even.roua...@mines-paris.org
> To: gdal-dev@lists.osgeo.org
> Subject: Re: [gdal-dev] does gdal support multiple simultaneous writers to
> raster
> Date: Sat, 12 Jan 2013 16:35:28 +0100
> CC:
Kennedy, Paul fugro.com.au> writes:
>
>
> Hi
> Simultaneous writers would be a better long term solution as we often improve
the raster following initial creation. This improvement may well be a filter run
on a sub region ( eg a despeckle) or updating a piece of the dtm with better
information
Hi
Simultaneous writers would be a better long term solution as we often improve
the raster following initial creation. This improvement may well be a filter
run on a sub region ( eg a despeckle) or updating a piece of the dtm with
better information or even some manual edits as a last resort.
ot;Even Rouault"
mailto:even.roua...@mines-paris.org>>
wrote:
Re: [gdal-dev] does gdal support multiple simultaneous writers to raster
Hi,
This is an intersting topic, with many "intersecting" issues to deal
with at
different levels.
First, are you confident that in the use cas
I would be interested in an implementation. I'm preparing a proposal to
georeference the complete cadastral map of the Netherlands in 1832 at a
10 cm/pixel scale with Cloud facilities. Gdalwarp is the central piece
of software, and distributed processing capabilities would be very
important. Co
> ex. convert
> multiple datasets to different output datasets in a parallel way.
As Frank underlined, there's currently an issue with the global block cache
regarding write support.
Imagine that you have 2 threads A and B.
Thread A deal with dataset A, and thread B deal with dataset .
Thread A
Le samedi 12 janvier 2013 02:38:55, Kennedy, Paul a écrit :
> Hi,
> Yes, we are pretty sure we will see a significant benefit. The processing
> algorithms are CPU bound not io bound. Our digital terrain model
> interpolations often run for many hours ( we do them overnight) but the
> underlying fi
; >
> > I believe slower cores but more of them in pcs are the future. My pc
> > has 8 but they rarely get used to their potential.
> >
> > I am certain there are some challenges here, that's why it is
> > interesting;)
> >
> > Regards
>
To add my 2cnts.
With MapWindow GIS we use TauDEM binaries to perform watershed delineations.
http://hydrology.usu.edu/taudem/taudem5.0/index.html
These TauDEM binaries are optimized to use MPI, but also work if you don't
have MPI installed.
I don't know in detail how it works but in general you
some challenges here, that's why it is
interesting;)
Regards
pk
On 11/01/2013, at 6:54 PM, "Even Rouault"
mailto:even.roua...@mines-paris.org>>
wrote:
Re: [gdal-dev] does gdal support multiple simultaneous writers to raster
Hi,
This is an intersting topic, with many &qu
On 12-01-2013 01:38, Kennedy, Paul wrote:
Hi,
Yes, we are pretty sure we will see a significant benefit. The
processing algorithms are CPU bound not io bound. Our digital terrain
model interpolations often run for many hours ( we do them overnight)
but the underlying file is only a few gigaby
Hi,
Yes, we are pretty sure we will see a significant benefit. The processing
algorithms are CPU bound not io bound. Our digital terrain model interpolations
often run for many hours ( we do them overnight) but the underlying file is
only a few gigabytes. If we split them into multiple files of
Simultaneous writers to different dataset or the same dataset ?In related
topic, can GDAL writte to different dataset in a parallel way ?ex. convert
multiple datasets to different output datasets in a parallel way.Would those
parallel operations not be affected by GDAL caching for bot read and
I would like to make a further suggestion: what about an option for
gdalwarp to create *external* tiles, instead of one big output file?
That's what I am doing now: just create one big georeferenced raster,
and split it up in tiles, mostly 2000*2000. Accessing those tiles with a
tileindex is re
Hi,
This is an intersting topic, with many "intersecting" issues to deal with at
different levels.
First, are you confident that in the use cases you imagine that I/O access won't
be the limiting factor, in which case serialization of I/O could be acceptable
and this would just require an API wit
dev
Subject: Re: [gdal-dev] does gdal support multiple simultaneous writers
to raster
I'v tried this in a Cloud environment with multiple VMs writing to the
same shared file location. I didn't use tiles, so the results where
negative. It would be a great asset for Cloud environments, h
I'v tried this in a Cloud environment with multiple VMs writing to the
same shared file location. I didn't use tiles, so the results where
negative. It would be a great asset for Cloud environments, however. Is
it in principle possible to build such a facility into GDAL, and if yes,
what kind o
.@gmail.com] On Behalf Of
Frank Warmerdam
Sent: Friday, 11 January 2013 9:33 AM
To: Kennedy, Paul
Cc: gdal dev
Subject: Re: [gdal-dev] does gdal support multiple simultaneous writers
to raster
Paul,
Generally speaking - no - GDAL does not now support multi-threaded
writing. Part of this is due t
Paul,
Generally speaking - no - GDAL does not now support multi-threaded
writing. Part of this is due to the way stuff gets write-flushed out
of the block cache and partly it is just that none of GDAL supports
multiple threading on a single dataset.
This statement is based on the assumption it i
I am looking to parallelize a long duration gridding process, and was
wondering if GDAL can support multiple simultaneous writers from
processess?
I am not sure if any of the underlying raster formats support this?
thanks in advance.
regards
pk
__
24 matches
Mail list logo
