The parallelization machinery in CircleCI is coarse-grained and intended for orthogonal builds I think, I don't know how well you could a fork/join model with it, but it may be possible. I'd suggest aggressive caching first if you haven't exhausted that.
As you probably know, we abandoned CircleCI for greener (faster) pastures: http://bitemyapp.com/posts/2016-03-28-speeding-up-builds.html those numbers I quote in the article are for a quad-core dedi I think, not the 10 core I mentioned earlier. On Fri, Jul 8, 2016 at 2:42 PM, <[email protected]> wrote: > On a semi-related note, if folks have figured out a sane way of > parallelizing large Haskell builds and test suites in CircleCI, I'd be all > ears. > > On Fri, Jul 8, 2016 at 12:35 pm, <[email protected]> Christopher Allen > wrote: > > I've tried setting Stack's package build parallelism to j20 or j30 > instead of the j10 it infers fro my core count but it just leads to > more sleepy threads. > > On Fri, Jul 8, 2016 at 2:34 PM, Michael Sloan <[email protected]> > wrote: > > > > Oh, and it's also worth mentioning that stack also takes "-j" to specify > > building more packages concurrently. It defaults to your processor count. > > It can be beneficial to use a higher value. > > > > To pass the -j to the cabal builds, use > > > > --ghc-options "-j2" > > > > On Fri, Jul 8, 2016 at 12:31 PM, Michael Sloan <[email protected]> > wrote: > >> > >> On Fri, Jul 8, 2016 at 11:53 AM, Erik Rantapaa <[email protected]> > >> wrote: > >>> > >>> Hi all, > >>> > >>> I've been building a large number of packages with a single `stack > >>> install` command, e.g. something like: > >>> > >>> stack install --resolver ... --keep-going $(cat all-packages) > >>> > >>> On a 8-core / 16 GB box I've noticed that the worker threads are not > >>> always kept busy. The overall %-idle time is at least 20%. > >> > >> > >>> > >>> > >>> So I'm wondering... > >>> > >>> - Is there anything I can do on the command line to improve CPU > >>> utilization? > >> > >> > >> > >> You can definitely increase CPU utilization by specifying "-jN" options > >> for GHC (e.g. "-j2"). Unfortunately, in my experience it usually doesn't > >> help as much as I'd hope with overall build time. One issue is that it's > >> possible for it to interact poorly with package-level parallelism. For > >> example, if stack is building 5 packages with "-j5", suddenly there's > >> potentially 25 processes each trying to use 100% of a core. > >> > >>> > >>> > >>> - Where can I find the code that does the job scheduling? I would like > to > >>> modify how package builds are prioritized to improve throughput. > >> > >> > >> Scheduling is done here: > >> > https://github.com/commercialhaskell/stack/blob/master/src/Control/Concurrent/Execute.hs > >> > >> Converting the build tasks to actions for scheduling is here: > >> > https://github.com/commercialhaskell/stack/blob/4304711d91f41a137931502ac4b1d00b3a4bc842/src/Stack/Build/Execute.hs#L573… > <https://github.com/commercialhaskell/stack/blob/4304711d91f41a137931502ac4b1d00b3a4bc842/src/Stack/Build/Execute.hs#L573> > >> > >> > >>> Thanks, > >>> Erik > >>> > >>> -- > >>> You received this message because you are subscribed to the Google > Groups > >>> "haskell-stack" group. > >>> To unsubscribe from this group and stop receiving emails from it, send > an > >>> email to [email protected]. > >>> To post to this group, send email to [email protected]. > >>> To view this discussion on the web visit > >>> > https://groups.google.com/d/msgid/haskell-stack/329a3117-b2ff-411c-954d-156001b622d6%40googlegroups.com > . > >>> For more options, visit https://groups.google.com/d/optout. > >> > >> > > > > -- > > You received this message because you are subscribed to the Google Groups > > "haskell-stack" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [email protected]. > > To post to this group, send email to [email protected]. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/haskell-stack/CAEYHaY6jQkuKUpa5E0pqNGgLOv47%2Bpqtb49wPat18%2Br1rBE9Jw%40mail.gmail.com… > <https://groups.google.com/d/msgid/haskell-stack/CAEYHaY6jQkuKUpa5E0pqNGgLOv47%2Bpqtb49wPat18%2Br1rBE9Jw%40mail.gmail.com> > . > > > > For more options, visit https://groups.google.com/d/optout. > > -- > Chris Allen > Currently working on http://haskellbook.com > > -- > You received this message because you are subscribed to the Google Groups > "haskell-stack" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/haskell-stack/CADnndOr-e8tL85zbjy-5wesOH1g%3DxDezJG_aP%3Dzv0i_M%2BD1iOw%40mail.gmail.c… > <https://groups.google.com/d/msgid/haskell-stack/CADnndOr-e8tL85zbjy-5wesOH1g%3DxDezJG_aP%3Dzv0i_M%2BD1iOw%40mail.gmail.com> > . > For more options, visit https://groups.google.com/d/optout. > > -- Chris Allen Currently working on http://haskellbook.com -- You received this message because you are subscribed to the Google Groups "haskell-stack" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/haskell-stack/CADnndOpGW6LbSvOiKJFg%2B-8J0CbjB4nRzmPpsYztK5TMUCoT4Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
