Re: [RFC PATCH 2/3] run-commands: add an async queue processor

2015-08-24 Thread Junio C Hamano
Stefan Beller writes: > diff --git a/builtin/index-pack.c b/builtin/index-pack.c > index 3f10840..159ee36 100644 > --- a/builtin/index-pack.c > +++ b/builtin/index-pack.c > @@ -11,6 +11,7 @@ > #include "exec_cmd.h" > #include "streaming.h" > #include "thread-utils.h" > +#include "run-command.h

Re: [RFC PATCH 2/3] run-commands: add an async queue processor

2015-08-21 Thread Stefan Beller
On Fri, Aug 21, 2015 at 12:44 PM, Jeff King wrote: > On Fri, Aug 21, 2015 at 12:05:13PM -0700, Junio C Hamano wrote: > >> The primary reason I suspect is because you sent to a wrong set of >> people. Submodule folks have largely been working in the scripted >> ones, and may not necessarily be the

Re: [RFC PATCH 2/3] run-commands: add an async queue processor

2015-08-21 Thread Stefan Beller
On Fri, Aug 21, 2015 at 1:47 PM, Junio C Hamano wrote: > > I do not think we are on the same wavelength. What I meant was to > do this: > > aq = xmalloc(...); > set up _everything_ in aq and make it a consistent state; > /* aq->first and aq->last are part of _everything_ i

Re: [RFC PATCH 2/3] run-commands: add an async queue processor

2015-08-21 Thread Junio C Hamano
Stefan Beller writes: >>> +struct task_queue *create_task_queue(unsigned max_threads) >>> +{ >>> + struct task_queue *aq = xmalloc(sizeof(*aq)); >>> + >>> +#ifndef NO_PTHREADS >>> + int i; >>> + if (!max_threads) >>> + aq->max_threads = online_cpus(); >>> + else >>> +

Re: [RFC PATCH 2/3] run-commands: add an async queue processor

2015-08-21 Thread Junio C Hamano
Jeff King writes: > On Fri, Aug 21, 2015 at 12:05:13PM -0700, Junio C Hamano wrote: > >> The primary reason I suspect is because you sent to a wrong set of >> people. Submodule folks have largely been working in the scripted >> ones, and may not necessarily be the ones who are most familiar with

Re: [RFC PATCH 2/3] run-commands: add an async queue processor

2015-08-21 Thread Stefan Beller
On Fri, Aug 21, 2015 at 12:51 PM, Jeff King wrote: > On Fri, Aug 21, 2015 at 12:48:23PM -0700, Stefan Beller wrote: > >> > Before even looking at the implementation, my first question would be >> > whether this pattern is applicable in several places in git (i.e., is it >> > worth the extra comple

Re: [RFC PATCH 2/3] run-commands: add an async queue processor

2015-08-21 Thread Jeff King
On Fri, Aug 21, 2015 at 12:48:23PM -0700, Stefan Beller wrote: > > Before even looking at the implementation, my first question would be > > whether this pattern is applicable in several places in git (i.e., is it > > worth the extra complexity of abstracting out in the first place). I > > think t

Re: [RFC PATCH 2/3] run-commands: add an async queue processor

2015-08-21 Thread Stefan Beller
On Fri, Aug 21, 2015 at 12:44 PM, Jeff King wrote: > On Fri, Aug 21, 2015 at 12:05:13PM -0700, Junio C Hamano wrote: > >> The primary reason I suspect is because you sent to a wrong set of >> people. Submodule folks have largely been working in the scripted >> ones, and may not necessarily be the

Re: [RFC PATCH 2/3] run-commands: add an async queue processor

2015-08-21 Thread Stefan Beller
On Fri, Aug 21, 2015 at 12:05 PM, Junio C Hamano wrote: > Stefan Beller writes: > >> This adds functionality to do work in parallel. >> >> The whole life cycle of such a thread pool would look like >> >> struct task_queue * tq = create_task_queue(32); // no of threads >> for (...) >>

Re: [RFC PATCH 2/3] run-commands: add an async queue processor

2015-08-21 Thread Jeff King
On Fri, Aug 21, 2015 at 12:05:13PM -0700, Junio C Hamano wrote: > The primary reason I suspect is because you sent to a wrong set of > people. Submodule folks have largely been working in the scripted > ones, and may not necessarily be the ones who are most familiar with > the run-command infrast

Re: [RFC PATCH 2/3] run-commands: add an async queue processor

2015-08-21 Thread Junio C Hamano
Stefan Beller writes: > This adds functionality to do work in parallel. > > The whole life cycle of such a thread pool would look like > > struct task_queue * tq = create_task_queue(32); // no of threads > for (...) > add_task(tq, process_one_item_function, item); // non blocking

[RFC PATCH 2/3] run-commands: add an async queue processor

2015-08-20 Thread Stefan Beller
This adds functionality to do work in parallel. The whole life cycle of such a thread pool would look like struct task_queue * tq = create_task_queue(32); // no of threads for (...) add_task(tq, process_one_item_function, item); // non blocking ... int ret = finish_task_qu