November 2014 6:00 PM
To: 'Blake Thompson'; Even Rouault
Cc: gdal-dev@lists.osgeo.org
Subject: RE: [gdal-dev] RFC 47 and Threading
Blake,
I tested your proposed change. It is better but sleeps are still visible. BTW:
isn’t it an example of double checked locking?
Even,
I am using I
u see the same problem with hCOAMutex?
> > It would be good if i could say: give me per dataset cache, open this
> dataset
> > with no locking. Unless the global cache provides its operations with
> less
> > sleeping.
> >
> > Regards.
> > Jacek Tomaka
> >
; sleeping.
>
> Regards.
> Jacek Tomaka
>
> Od: gdal-dev-boun...@lists.osgeo.org [gdal-dev-boun...@lists.osgeo.org] w
> imieniu Blake Thompson [flippm...@gmail.com]
> Wys³ano: 29 sierpnia 2014 02:35
> Do: Even Rouault
> DW: gdal-dev@lists.
imieniu Blake Thompson [flippm...@gmail.com]
Wysłano: 29 sierpnia 2014 02:35
Do: Even Rouault
DW: gdal-dev@lists.osgeo.org
Temat: Re: [gdal-dev] RFC 47 and Threading
Even and Andre,
> I want to start off by saying a big thanks to Blake for taking his time
> to tackle what can only be
Even and Andre,
> I want to start off by saying a big thanks to Blake for taking his time
> > to tackle what can only be a very difficult problem.
> > From what I can observe, the current discussion seems to be around the
> > boundary of who should be responsible for ensuring thread safety around
Even,
> Yeah, that why was I suggested the reverse ;-) But an automatic renaming
> should be doable.
> Another danger of keeping IReadBlock and making it now responsible of
> ensuring
> its thread safety is for out-of-tree drivers that wouldn't realize they
> need
> to change something as code wou
Selon Andre Vautour :
> I want to start off by saying a big thanks to Blake for taking his time
> to tackle what can only be a very difficult problem.
> From what I can observe, the current discussion seems to be around the
> boundary of who should be responsible for ensuring thread safety around
I want to start off by saying a big thanks to Blake for taking his time
to tackle what can only be a very difficult problem.
From what I can observe, the current discussion seems to be around the
boundary of who should be responsible for ensuring thread safety around
the block cache. The core
Le vendredi 22 août 2014 22:17:37, Blake Thompson a écrit :
> Even,
>
> The naming is perhaps not well choosen, but the documentation of the
>
> > contract
> > of this API should make it clear on what an implementation should do and
> > what
> > it should not do.
>
> Perhaps it should be reverse
Even,
The naming is perhaps not well choosen, but the documentation of the
> contract
> of this API should make it clear on what an implementation should do and
> what
> it should not do.
>
Perhaps it should be reversed? IReadBlock and IReadBlock_not_thread_safe?
(would obviously require lots of
Le vendredi 22 août 2014 21:50:56, Blake Thompson a écrit :
> Even,
>
> On Fri, Aug 22, 2014 at 1:33 PM, Even Rouault
>
> wrote:
> > Note: after re-reading, I realize that I misread your above sentence as
> > "it will work to translate multiple datasets in a parallel way with a
> > *per- dataset
Le vendredi 22 août 2014 21:34:49, Blake Thompson a écrit :
> Even,
>
> > This might be a problem in practice since the amount of cache might grow
> > quickly. By default a VRT will open simultaneously up to
> > GDAL_MAX_DATASET_POOL_SIZE (whose default is 100, but can be changed at
> > runtime wi
Even,
On Fri, Aug 22, 2014 at 1:33 PM, Even Rouault
wrote:
>
>
> Note: after re-reading, I realize that I misread your above sentence as "it
> will work to translate multiple datasets in a parallel way with a *per-
> dataset* cache").
>
> So, even if you didn't write it, I'm afraid that people wi
Even,
> This might be a problem in practice since the amount of cache might grow
> quickly. By default a VRT will open simultaneously up to
> GDAL_MAX_DATASET_POOL_SIZE (whose default is 100, but can be changed at
> runtime with the config option) source datasets.
> If you do a gdal_translate of
Le vendredi 22 août 2014 17:53:50, Blake Thompson a écrit :
> Jeff,
>
> Thanks Blake for the detailed response. I did not realize that I did not do
>
> > a reply all in my previous email I sent.
>
> Not an issue, glad you guys are interested in my changes.
>
> > --> I thought that this was not
> >
> > Quick question - presumably for VRT datasets any source images currently
> > share the global cache and are treated from this proposals' POV as their
> > own "datasets"? As well as the VRT being a separate dataset? If so, seems
> > like this could be quite a major win for users with VRTs f
Kurt,
>
> With a 2 week old baby, my brain has been soggy.
>
Congrats!
>
> I do see the 3 threads here:
> http://lists.osgeo.org/pipermail/gdal-dev/2014-August/thread.html
>
> The asan/msan/tsan tools are
>
> asan - address sanitizer - https://code.google.com/p/address-sanitizer/
> msan - memor
Blake,
With a 2 week old baby, my brain has been soggy.
I do see the 3 threads here:
http://lists.osgeo.org/pipermail/gdal-dev/2014-August/thread.html
The asan/msan/tsan tools are
asan - address sanitizer - https://code.google.com/p/address-sanitizer/
msan - memory sanitizer - https://code.goog
Kurt,
On Fri, Aug 22, 2014 at 9:07 AM, Kurt Schwehr wrote:
> I've got threading issues on my list of things to investigate with GDAL.
> I've been running MSAN on GDAL lately and want to get TSAN going too. I'm
> out on leave this month, but hope to get more into this stuff (testing,
> threadin
Jeff,
Thanks Blake for the detailed response. I did not realize that I did not do
> a reply all in my previous email I sent.
>
Not an issue, glad you guys are interested in my changes.
>
> --> I thought that this was not possible using current trunk GDAL because
> of the global cache. At least t
Robert,
> I agree - from reading it seems like the major improvement is shifting
> away from a global, locking cache to a per-dataset cache. (ah. has been
> edited since you read it?)
>
Yes, I updated the RFC some yesterday after the email from Jeff Lacoste. He
forgot to hit reply all so it ende
I've got threading issues on my list of things to investigate with GDAL.
I've been running MSAN on GDAL lately and want to get TSAN going too. I'm
out on leave this month, but hope to get more into this stuff (testing,
threading, caching, etc) come October.
On Fri, Aug 22, 2014 at 6:11 AM, Jeff
Thanks Blake for the detailed response. I did not realize that I did not do
a reply all in my previous email I sent.
On Thu, Aug 21, 2014 at 3:59 PM, Blake Thompson wrote:
>
>
>
> On Thu, Aug 21, 2014 at 11:36 AM, Jeff Lacoste
> wrote:
>
>> Hi,
>>
>> Improving the thread safety of GDAL is a bi
On Thu, Aug 21, 2014 at 8:26 AM, Even Rouault
wrote:
>
> I guess the silence in thread is due to people being impressed by the
> austerity of the topic...
>
Something like that :)
>
> On the other side, I would be very pleased with having "just" the
> preliminary
> step of Blake's work, i.e.
On Thu, Aug 21, 2014 at 11:36 AM, Jeff Lacoste
wrote:
> Hi,
>
> Improving the thread safety of GDAL is a big improvement. I know this
> proposal is not claiming to 'fix' gdal thread safety but adress at least
> the cache safety. This is said, may be to help
> clarify more the proposal, we can sta
Even,
Thank you very much for response to this, your helping me understand all of
this stuff has been very valuable.
I'm not sure the ratio gain/effort to make dataset methods "a bit more"
> thread
> safe is high enough. Very few drivers can be made fully thread-safe, and
> fully
> parallelized (
Hi,
I guess the silence in thread is due to people being impressed by the
austerity of the topic...
For what is worth, I've had a few opportunities to discuss directly with Blake
about his work. I'm very impressed with his energy and enthousiasm to hack in
that difficult core area of GDAL, and
I wanted to call to everyone's attention the work I have been doing in an
attempt to make it possible for more portions of GDAL to be thread safe and
improve speed in multi-threaded environments. I have put a RFC up here:
http://trac.osgeo.org/gdal/wiki/rfc47_dataset_caching
Originally my work fo
28 matches
Mail list logo
